How to Read Really Large Files in Java
Here’s a class that makes this really easy. The entire file is never read into memory, so it should be able to handle files of any size (that your operating system can handle).
import java.util.*; import java.io.*; public class BigFile implements Iterable<String> { private BufferedReader _reader; public BigFile(String filePath) throws Exception { _reader = new BufferedReader(new FileReader(filePath)); } public void Close() { try { _reader.close(); } catch (Exception ex) {} } public Iterator<String> iterator() { return new FileIterator(); } private class FileIterator implements Iterator<String> { private String _currentLine; public boolean hasNext() { try { _currentLine = _reader.readLine(); } catch (Exception ex) { _currentLine = null; ex.printStackTrace(); } return _currentLine != null; } public String next() { return _currentLine; } public void remove() { } } }
Here’s how you might use it:
BigFile file = new BigFile("C:\Temp\BigFile.txt"); for (String line : file) System.out.println(line);
If you liked this, please tell someone else about it and subscribe to this blog. If not, please leave me a suggestion below.
Hi,
Thanks for sharing the code. I really liked the way you have made file reading very easy.
Thanks a lot.
Satish
Hi,
Have you tried this on files larger than 4GB? I have used a similar method but it can’t cope with anything >4GB
Dave
My apologies – my method did not check properly for EOF. I am sure yours will be OK for files > 4GB.
Dave
I haven’t tried anything larger than 4 GB that I can remember. What really matters is whether you’re trying to save the file contents into memory as you read it. If you do, then you will run into a memory limit based on your computer hardware. If you don’t (i.e., process it a line at a time), then there should be no limit other than hard drive space.
Hi, im a Java newbie. Where can I place the code
BigFile file = new BigFile(“C:\Temp\BigFile.txt”);
for (String line : file)
System.out.println(line);
??
Can you send me all the code?
Thanks
Hi David,
You should be able to put the code into a Main.java class that would look something like this.
public class Main
{
public static void main(String[] args)
{
BigFile file = new BigFile(“C:\Temp\BigFile.txt”);
for (String line : file)
System.out.println(line);
}
}
If that doesn’t help you, maybe you can post what you have, and we can try from there.
Thanks for the help. I’ve created that class you suggested but it gives me the following error regarding the file variable: Type mismatch: cannot convert from element type Object to String. Any idea how to fix this? Thanks in advance.
Hi David,
I think I found the problem. Part of the code got chopped out by the blogging platform I’m using because it though they were HTML tags. I’ve updated the code above. Basically all I changes was anywhere it said Iterator to Iterator<String>. See above. This way when it iterates through the file it knows to look for String objects rather than Object objects. Hope that helps.
-Steve
Thanks for the help. That was exactly what I needed.
HiAll,
I want to read one large text file, then i have to split this file at runtime then i need to place this 2 files into one folder, Please give me the solution,it’s very urgent.
Please help me ya
Sreenivas
Sreenivas,
I’m afraid your description of what you’re trying to do is not adequate for us to offer a solution. Please provide more details, even pseudocode. Also I’m curious what you need it for?
i have 2000000 numbers of txt file upload to two servers . using java . for each server upload to 1000000 number using mysql db.
That clearly works, but if perfomance is important, you should use a MappedByteBuffer of the NIO package instead, like in this example.
I have a large log file, which I have to read and have to do the following:
In the log file I would have an entry saying “processing 1234″ and somewhere down the line in the file I would have corresponding line
saying “acknowledged 1234″. So based on this number I have to say whether or not it was acknowledged. If we don’t recieve ack, we would not have the second line. I am trying for a good optimal solution. Any help appreciated.
Girish, do you know what you’re looking for in the file from the beginning? Or is the value (for example, “1234″) different depending on the file? If the latter, then you’ll probably need to use a technique called “regular expressions.”
Could you implement delete functionality ?
Probably. Can you please give me more detail about what you want to do?
I want to replace/delete string occurrence in a very big log file without writing it to another file.
If you are working with a very large file, the problem you often run into is a lack of memory. If you didn’t have to worry about memory capacity, you could read the file contents into a StringBuilder (or array) object, modify it, and then spit it back out to the same file. But since we are talking about ways to deal with very large files, the only solution I can think of would be to read it from the source file (using the technique shown in this post), modify each line as you read it, and then spit it out to a new file. Then you could delete the original file and rename the new file to the name of the original file if you wanted. Hope that helps.
I tried to read a very big file and write to another it works fine but if I try to call this class from another one (like main but from another class) I get out of memory error any idea why?
Not sure. Can you post a code sample of where it’s working and where it’s not?
The BigFile class with
public void main(String[] args)
{
BigFile file = new BigFile(“C:\Temp\BigFile.txt”);
for (String line : file)
System.out.println(line);
}
works fine
but using another class that has function that do the same as above I get out of memory on the line
for (String line : file)