Jeff Plotzke Posted January 4, 2007 Report Share Posted January 4, 2007 I need to stream a lot of data to disk quickly with a LVRT system running on PXI. The data is being acquired at up to 5kHz from my DAQ card of about 350 channels. (So, if all the data was SGL's, it's up to 7 MB/sec) Right now, I have my data aquisition running in a timed loop and writing the data to RT FIFOs. The FIFOs are then read by a consumer loop which will write the data to files. My question is -- What's the best way to write this data to file to make sure that the rest of my application doesn't die? I can write in pure binary, but I know that I need to wait for 512-byte chunks. What about TDMS? Since the 'S' stands for streaming, is this optimized for streaming data to disk? Will it automatically do the 512-byte chunks for me? Has anyone had luck with TDMS and fast data? Can you share any experiences? Thanks for all your help! Quote Link to comment
crelf Posted January 4, 2007 Report Share Posted January 4, 2007 Here's some ideas on the TDMS format. Quote Link to comment
Olivier Jourdan Posted January 5, 2007 Report Share Posted January 5, 2007 Can you share any experiences? About writting file on HD my experience is to use firewire HD (you need a firewire PXI Board) in place to write on the PXI HD. I use this with LV7.1 on PXI-8187. CPU load dicrease significantly... hope this help you... Quote Link to comment
konroh Posted January 5, 2007 Report Share Posted January 5, 2007 About writting file on HD my experience is to use firewire HD (you need a firewire PXI Board) in place to write on the PXI HD. I use this with LV7.1 on PXI-8187. CPU load dicrease significantly...hope this help you... Hi Jeff, i had a lot of problems with writing to disk in LVRT 7.x destroying the determinism of my application, and finally just started streaming the data to my host program. However, I recently upgraded to LVRT 8.2, and now, using the same code, I have no problems streaming data directly to the hard drive of my RT controller. Quote Link to comment
Neville D Posted January 5, 2007 Report Share Posted January 5, 2007 Hi Jeff,i had a lot of problems with writing to disk in LVRT 7.x destroying the determinism of my application, and finally just started streaming the data to my host program. However, I recently upgraded to LVRT 8.2, and now, using the same code, I have no problems streaming data directly to the hard drive of my RT controller. I seem to remember reading somewhere that the file write functions now (LV 8.2 ?) automatically take care of the 512 byte chunking before writing to file.. maybe that is contributing to a more efficienct file read and write. Neville. Quote Link to comment
Jeff Plotzke Posted January 6, 2007 Author Report Share Posted January 6, 2007 Thanks for all the suggestions! I'm trying some tests to compare Binary/TDMS and I'll let you know how it turns out. In the mean time, I found this article on ni.com which gives some good information about optimizing RT applications with file I/O and DAQ: http://zone.ni.com/devzone/cda/tut/p/id/5424 Quote Link to comment
Herbert Posted January 19, 2007 Report Share Posted January 19, 2007 You might want to have a look at http://forums.lavag.org/index.php?s=&s...ost&p=23027. Herbert Quote Link to comment
Jeff Plotzke Posted January 21, 2007 Author Report Share Posted January 21, 2007 You might want to have a look at http://forums.lavag.org/index.php?s=&s...ost&p=23027. Herbert Thanks, Herbert! It sounds like TDMS may be a good choice, since I'm just writing 2D numeric arrays. Curious... with TDMS, should I still save up data so it's a multiple of 512 for a faster disk write? Also, would you suggest open/write/close for every time I write to the file, or a open/close once at the beginning and end and then many write/flushes in between? Thanks again! Quote Link to comment
Herbert Posted January 21, 2007 Report Share Posted January 21, 2007 Jeff, the 512 thing is something I cannot really confirm, since there's several "sweet spots" for chunk sizes that are floating around (the one I hear most often is 65535). Since we still add several bytes of index information every time we write, you would also need to use a different value to exactly hit 512. In general, I would rather use large data chunks to lower the number of times you access the disc. I would clearly recommend to open the file, then keep writing and finally closing it. The performance of opening the file does slightly decrease with the file size, because we're setting up index information for random access when we're opening a file. Flushing is something you only need if you have to make absolutely sure that all data you have acquired is on disc even in case of a complete system failure. Otherwise, just using TDMS Write will be perfectly sufficient. Hope that helps, Herbert Quote Link to comment
crelf Posted January 21, 2007 Report Share Posted January 21, 2007 Hope that helps,Herbert I'm really glad you joined us Herbert - your posts are insightful and well presented :thumbup: Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.