Read from Multiple Text Files in Java

This post explains a way to read from multiple text files. It builds on this post that explains how to iterate over large files line by line without reading them into memory. It also builds on this post about using wildcards to search for files in a directory.

It also allows you to specify whether the files have a header row and/or whether you want to skip any rows at the top of each file (maybe because they contain metadata that you want to ignore).

import java.io.*;
import java.util.*;
 
public class MultiFileReader implements Iterable<String>
{
    private ArrayList<File> _files;
    private boolean _filesHaveHeaders;
    private int _numSkipLines;
 
    public MultiFileReader(String dir, String filePattern, boolean filesHaveHeaders, int numSkipLines) throws Exception
    {
	_files = Files.GetFilesInDirRecursively(dir, filePattern);
	_filesHaveHeaders = filesHaveHeaders;
	_numSkipLines = numSkipLines;
    }
 
    public MultiFileIterator iterator()
    {
	return new MultiFileIterator();
    }
 
    public class MultiFileIterator implements Iterator<String>
    {
	private ArrayList<File> _iteratorFiles;
	private Iterator<String> _currentFileIterator;
 
	public String HeaderLine = "";
 
	public MultiFileIterator()
	{
	    _iteratorFiles = new ArrayList<File>(_files);
	}
 
	private boolean IterateFile() throws Exception
	{
	    while (_iteratorFiles.size() > 0)
	    {
		_currentFileIterator = new BigFile(_iteratorFiles.remove(0)).iterator();
 
		if (_currentFileIterator.hasNext())
		{
		    if (_filesHaveHeaders)
			HeaderLine = _currentFileIterator.next();
 
		    for (int i=0; i<_numSkipLines; i++)
			if (_currentFileIterator.hasNext())
			    _currentFileIterator.next();
 
		    if (_currentFileIterator.hasNext())
			return true;
		}
	    }
 
	    return false;
	}
 
	public boolean hasNext()
	{
		try
		{
		    if (_currentFileIterator == null)
			return IterateFile();
 
		    if (_currentFileIterator.hasNext())
			return true;
		    else
			return IterateFile();
		}
		catch (Exception ex)
		{
		    ex.printStackTrace();
		    return false;
		}
	    }
 
	public String next()
	{
	    return _currentFileIterator.next();
	}
 
	public void remove()
	{
	    _currentFileIterator.remove();
	}
    }
}

You would invoke this class in a way such as the following:

MultiFileReader reader = new MultiFileReader("C:\\Temp\\", "*.txt", false, 0);
 
for (String line : reader)
    System.out.println(line);

2 Responses to “Read from Multiple Text Files in Java”

  1. BigFileReader?? this show me an error, Do you now why? what is this class?.

    Thanks a lot

  2. Sorry, might be a little confusing. At the top of this post, I reference another post that contains the code for BigFileReader. See http://code.hammerpig.com/how-to-read-really-large-files-in-java.html

Leave a Reply