Comments on: How to Read Really Large Files in Java

By: Devang Bhalgama

Devang Bhalgama — Tue, 08 Nov 2011 05:57:50 +0000

Thank you Admin for your help. As we have decided to change our code as per your code and we’ll do the performance testing. After getting result i’ll definetely post our statistics on this forum.

I’ll also recommend this forum to my friends and my colleague.
Again thank you, thank you so much for your time and help.

By: admin

admin — Tue, 08 Nov 2011 05:18:43 +0000

THe code in this post will help you parse through large files, but only if you don’t read the contents into memory. You are putting a lot of data into the memory list, so you will eventually run out of memory. I suggest you look into using a micro database like HSQLDB. It will probably give you much better results. Here’s a post on that. http://code.hammerpig.com/what-is-an-in-memory-database-and-what-is-it-good-for.html

By: Devang Bhalgama

Devang Bhalgama — Fri, 04 Nov 2011 16:19:03 +0000

I am doing Data Profiling which is part Data Quality(Master Data Management). So for data profiling what i am doing is i’ll select some attributes for example SSN,DOB,State,Country_Code after selecting delimited file. Now after selecting these attributes i’ll try to find unique values and its occurences for that perticular attribute(means will select perticular columns). I’ll also try to find how many patterns or how many different different formats SSN number contains.We’ll also predict datatype for perticular attribute. So for these functionalities we have to check all the domain value and have to process them. So it is necessary to keep them in memory so then we get our result very fast. We have also applied another option, to process file using chunks of data but it is time killer.

You are right that ArrayList will become very huge but it is necessary and we have dedicated server also to process them. We dont have any other application deployed on that server.

As i have said in our current code we are using bufferdreader and Scanner in traditional way and i feel that your code is looking optimized but not sure will it give good performance to us or not thats why i am asking .

By: admin

admin — Fri, 04 Nov 2011 13:48:42 +0000

Hi Bevang,

I think the problem is that the ArrayList object is getting too large. If you have a huge file, that will probably cause a problem. Do you have to load everything into an ArrayList? What do you want to do after the data values are in the ArrayList?

By: Devang Bhalgama

Devang Bhalgama — Fri, 04 Nov 2011 12:19:50 +0000

Hi Admin,

I read all the post and my problem is also somewhat similar to others.

I want to read pipe delimited file which has more then 100 million records.
My server RAM is : 12 GB

I have set Heap memory to 8 GB. OS is Windows server 2008 64 bit.

I am using Apache tomcat 7.0.22 64 bit.

I am using buffered reader and Scanner to read and extract the file and i am storing these records in the arraylist.

But it is taking two much time.
For reading 3 lakhs records it is taking 2 mins. So should i use scanner in the for loop like below code is this right approach to extract data from one line and storing them in the arraylist as i have 56 columns.

BigFile file = new BigFile(“C:\Temp\BigFile.txt”);

for (String line : file)
{
// System.out.println(line);
Scanner part and generate arraylist

}

So will it give any performance impact to me. As our code is in production environment. Can i read and process 3 lakhs records within minute.

Please help me

By: Vahid

Vahid — Wed, 28 Sep 2011 04:54:47 +0000

Thanks. Nice piece of code. I learnt something and saved me hours!

By: raymundo

raymundo — Thu, 18 Aug 2011 21:43:05 +0000

Thanks! I will chech the link.

By: admin

admin — Wed, 17 Aug 2011 17:52:53 +0000

Raymundo,

Sounds like you have a very slow machine if it is taking that long, or else that you have a small amount of memory. One suggestion is to try another zip program that doesn’t try to store the whole file in memory. 7Zip might work for this. If you want to do it in Java, you could try it. There are a few explanations on the web on how to do this, such as here: http://java.sun.com/developer/technicalArticles/Programming/compression/

By: raymundo

raymundo — Wed, 17 Aug 2011 17:46:22 +0000

Thanks for answer.

I unzzip the file (or I try) using winzip tool to do it. It takes a lot of time and I have to pause it and search for another way to extract the txt file.

By: admin

admin — Wed, 17 Aug 2011 16:42:15 +0000

Raymundo,

How are you unzipping the file? In Java? Or at the command line?