Advanced TDMS

[0_o]

Hi,

I need to work with 5000*5000 arrays.

Since LabVIEW (and Windows 32bit) can't handle it I had to write the arrays into a file.

The operations need to be as fast as possible and thus, I turned to the advanced TDMS block that allow pre-allocation of both the buffers and the TDMS asynchronously (without creating the index file) while allowing an asynchronous read from an index inside the TDMS.

However, this is the first time I am using those advance blocks and even though it seems to work, I'm not sure how optimized my code is and if all the parameters are correctly entered.

For example, here are some questions:

1. Is it ok to use both asynchronous writing and buffering?

2. What should be the buffer size (wrong number corrupts the data)?

3. How long should I wait between the write and the read (asynchronous write means that I know when I sent the write request and I can't be sure if it is ready to read already)?

4. For some reason TDMS uses a lot of cpu, thus, if I don't have enough free cpu to spare the fast TDMS becomes suddenly very very slow. Am I right?

5. Did I enter the buffer and the allocation sizes correctly?

6. Can I read and write asynchronously at the same time? Should I read while writing?

7. Is there a better way to handle 5000*5000 arrays without getting out of memory?

I attached two vis that demonstrate what I'm doing. The only difference between it and my actual code is that in my write I add 1 value to each channel in a loop (might be able to improve it by collecting more values and writing to the TDMS more rarely) and I take the index of that loop to my read subvi in order to read the file while writing it (which might result in a read that comes before the actual write from the buffers).

Thanks in advance,

DD.

P.S - I'm working on a dual core machine with Win7 32 bit + LV11

write TDMS (SubVI).vi

read TDMS (SubVI).vi

Edited January 25, 2012 by 0_o

[tnt]

Hi,

a while back I had some similar issues.

I just did a small test om my dualcore XP 32-bit LV8.5.1 with 4 GB RAM and had no problems allocating and using a 5000x5000 SGL (4-bytes)-array,

when switching the datatype to DBL I got "memory full" / "not enough memory to complete this operation".

Other solutions that helped in my particular case was using the XP /3GB switch in boot.ini which allows for larger memoryblocks

and/or creating the 5000*5000 array (If possible in your case) as an array of 5000 clusters with a 5000-elements 1D-array

which does not require a big area of continuous free memory (in my case a 8000 pixels linscan camera) but 5000 times 5000 elements.

Also be aware of memcopies of your data when displaying in indicators...

Good luck...

[0_o]

Thanks tnt!

I heard something about queues of arrays to prevent a need for a continous memory, I'll try it and report along with the boot.ini after I present the current demo with TDMS to the company's board.

As for memory copies, yep, I'm aware of it. The old code used the display as database to save all the outputs while in my new code the db is on the HD.

Until I have the time to test your alternative solution, do you have any input regarding the advanced TDMS questions?

DD.