Large File reading with LV 8

[Chris Davis]

I've recently started a new project in LabView 8 and I'm trying to read large binary files. I'm having trouble reading a 4 Gb file. I know this was a problem with LV 7.1.1 and below, but I thought with LV 8 having Int64 file location this would work ok. Anyway I'm getting a error 6 Generic File i/o Error in my error cluster, and I don't understand why. Perhaps someone could try to reproduce this problem, to see if it is unique to my install?

Thanks.

[crelf]

I've recently started a new project in LabView 8 and I'm trying to read large binary files.

LabVIEW does indeed have issue with opening large files - what you need it the "Large File" package from OpenG - download and install the Commander and you'll be able to get access to all of the packages...

[Chris Davis]

LabVIEW does indeed have issue with opening large files - what you need it the "Large File" package from OpenG - download and install the Commander and you'll be able to get access to all of the packages...

I know that LV7 had this problem and I've used the Large File package from OpenG in all of my projects that dealt with LV7 and 7.1.x. But I was hoping that they had given us the ability to read and write files larger than 2 Gb when they redesigned the file i/o primitives (they now use Int64s as their file pointer after all). Is the 2Gb limitation still there in LV8?

Chris

[i2dx]

aehem ... does your machine have enough memory to load the file completely?

if you dont have at least 6 GB Ram, it will be swapped anyway?!

I think you have to work with the file pointers and pick out that data, you need in that moment.

[Chris Davis]

I thought I would try another experiment today. I lowered my reading size from 1.3 megapixels to 1 megapixel and things work. Has anyone else had this problem or can anyone else reproduce the problem. I've included the VI so you won't have to make it yourself. Remember to try 1024x1024 and 1280 x 1024 as well. You'll have to create a large file yourself, LAVA won't let me post one that big!

BTW my machine specs are

2 x AMD Opteron 242

4 x 512 MB RAM

1 x 80 GB HD

1 x 120 GB HD

Windows XP SP2

LV 8.0.1

Download File:post-2547-1143035164.vi

[Chris Davis]

I've submitted a request to NI support to check this out to see if they can replicate the problem. I've tried it on three different machines here and they all exhibit the same problem. I've attached a VI that will create a 2.1 Gb file on your hard drive and then try to read the first 1024x1280 U8's out of it. If you can run this on your machine please post with your results. I think this may be a bug in 8.0.1. If anyone is still running 8.0 it would be great to see if they also have the same problem.

Thanks,

Chris

Download File:post-2547-1143139932.vi

[i2dx]

I've submitted a request to NI support to check this out to see if they can replicate the problem. I've tried it on three different machines here and they all exhibit the same problem. I've attached a VI that will create a 2.1 Gb file on your hard drive and then try to read the first 1024x1280 U8's out of it. If you can run this on your machine please post with your results. I think this may be a bug in 8.0.1. If anyone is still running 8.0 it would be great to see if they also have the same problem.

Thanks,

Chris

I can only guess, but for me that sounds like a file size variable overflow. If they use an I32 as (internal) file size variable, they can store file sizes up to 2147483647 Bytes in that variable. If the file is bigger, the I32 will roll over and you will get weird results ...

which would explain, that you have no problems with smaller files, but you get into trouble, when using files with a size above 2 GB ... Maybe you can check this out, by testing your SW with 2 files: one with a size of 2GB -1 Btye and the second with 2GB + 1 Byte

[Chris Davis]

I can only guess, but for me that sounds like a file size variable overflow. If they use an I32 as (internal) file size variable, they can store file sizes up to 2147483647 Bytes in that variable. If the file is bigger, the I32 will roll over and you will get weird results ...

which would explain, that you have no problems with smaller files, but you get into trouble, when using files with a size above 2 GB ... Maybe you can check this out, by testing your SW with 2 files: one with a size of 2GB -1 Btye and the second with 2GB + 1 Byte

Actually the file size is only part of the problem. As you can see from one of my earlier posts, you can read an array of 1024x1024 from any size file. When you request more array elements than that from a file that is bigger than 2 gb you get an error 6. I've checked this with a file that is 2.0 gb and one that is 2.1 gb in size. I've submitted this to NI and they admit that it may be a bug, and the R&D team is looking at it now. I'll report back thier workaround and "bug" number. I don't know how many other people this will effect, but I ran into it because I use LabView to play back video I've captured using machine vision cameras. One of the cameras I use all the time has an array size of 1024x1280, and I can record several Gb worth of data from it.

Thanks for the thoughts,

Chris

[Chris Davis]

Here is a VI that will demonstrate the problem on someone else's computer, don't worry the large file gets deleted after it is created. :headbang:

This is an official bug, CAR # 3VM9NQ3A. I've only tested it with U8's but I imagine it will produce the same error with other data types. I've added this to the LAVA LV8 buglist.

Chris

Download File:post-2547-1143659538.vi

[Chris Davis]

Here is a VI that will demonstrate the problem on someone else's computer, don't worry the large file gets deleted after it is created. :headbang:

This is an official bug, CAR # 3VM9NQ3A. I've only tested it with U8's but I imagine it will produce the same error with other data types. I've added this to the LAVA LV8 buglist.

Chris

BTW, this bug has been fixed in LV 8.2.

[LAVA 1.0 Content]

This is an official bug, CAR # 3VM9NQ3A. I've only tested it with U8's but I imagine it will produce the same error with other data types. I've added this to the LAVA LV8 buglist.

Just looked at this thread today after seeing your update. I've been writing to large binary files recently with 8.0.1; In one test I created 38.9 GBytes of data over 66 hours. The data was 128 byte long strings, with a U32 counter as the first 4 bytes. This was received using a TCP/IP connection, passed to my logging function via a queue, then flushed and written to file once a second.

The one thing that I noticed early on was that the Prepend Array or String Size boolean didn't behave as I expected it to. I was using Flush Queue and receiving an array of TCP strings. I wired the array into the Data input and a False into Prepend Array or String Size and still got 4 additional bytes in front of each of my 128 byte strings (0x0080). I had to put Write To Binary File inside an auto-indexed loop to get the "true raw data" to disk.

I verified the integrity of the data transferred by simply reading in the logged file 128 bytes at a time and monitoring the counter. I actually had a roll-over of the U32 back to zero.

I've been looking at your example and was able to modify it and "make it work". It might not be fast or pretty, but it can write a file > 2 GB and read it back... You mentioned that you hadn't tested with other data types, so I thought that I would point out that with 8.0.1 and Windows XP you CAN write to LARGE files using simple strings.

Download File:post-949-1158953469.vi

Now, if I didn't see SO many bugs (like with tables) I might consider actually working in 8.20

[Chris Davis]

Just looked at this thread today after seeing your update. I've been writing to large binary files recently with 8.0.1; In one test I created 38.9 GBytes of data over 66 hours. The data was 128 byte long strings, with a U32 counter as the first 4 bytes. This was received using a TCP/IP connection, passed to my logging function via a queue, then flushed and written to file once a second.

The one thing that I noticed early on was that the Prepend Array or String Size boolean didn't behave as I expected it to. I was using Flush Queue and receiving an array of TCP strings. I wired the array into the Data input and a False into Prepend Array or String Size and still got 4 additional bytes in front of each of my 128 byte strings (0x0080). I had to put Write To Binary File inside an auto-indexed loop to get the "true raw data" to disk.

I verified the integrity of the data transferred by simply reading in the logged file 128 bytes at a time and monitoring the counter. I actually had a roll-over of the U32 back to zero.

I've been looking at your example and was able to modify it and "make it work". It might not be fast or pretty, but it can write a file > 2 GB and read it back... You mentioned that you hadn't tested with other data types, so I thought that I would point out that with 8.0.1 and Windows XP you CAN write to LARGE files using simple strings.

Download File:post-949-1158953469.vi

Now, if I didn't see SO many bugs (like with tables) I might consider actually working in 8.20

Actually writing the file is not a problem, its reading back large chuncks of it. Try reading more than 8192 of your 128 byte strings at a time and you should get an error 6.

BTW, my large files come from a vision camera where 35 gb can be made in one minute of acquisition.

I had trouble with the Prepend Array or String size switch as well, but my problem was that I didn't know it was on be default. I made a subroutine to provide a progress bar to be used when copying big files. The VI reads 4 megabytes of the original file at a time as a string, then copies it to a new file. Finally figured out that the Prepend switch should be hard wired to false for my routine to work.

[crelf]

Now, if I didn't see SO many bugs... I might consider actually working in 8.20

Sounds like a rant to me

[Yair]

BTW, my large files come from a vision camera where 35 gb can be made in one minute of acquisition.

As in several hundred MB of data every second?

What kind of images and camera\image specs is this?

[Chris Davis]

As in several hundred MB of data every second?

What kind of images and camera\image specs is this?

Basler 504k or 504kc captured at 400 fps straight to disk for 1 minute. 1280x1024 (imager size) x 400 fps x70 seconds = 35 gb. I use the NI 1429 with an 8 drive SATA RAID. And before you ask, no I don't use LabView to write data that quick, I use CVI (actually I use Windows dlls), I use LabView and IMAQ to be the GUI for my imaging application. If you look at the OpenG Large File routines, you can begin to find out how I write data that quick to disk.