Write multiple images into TDMS file

[Thang Nguyen]

Hi,

I am working on a R&D project which need to acquire and record images at 180FPS. I intend to convert image data to byte array then to string then save to TDMS file. I just figured out TDMS does not save string with NULL value. I don't think it's a good idea to save byte array since it will be difficult to index images with a large size array.

Could you please give me some advice on what I should do?

Sincerely,

Thang Nguyen

[hooovahh]

Writing strings to TDMS that are a byte stream can be tricky.  I have a Read/Write Cluster to TDMS and it flattens it to a string.  But as you noticed there needs to be special consideration for null.  Well it also turns out that on different platforms there are other characters that need special attention too. You can absolutely do extra work on your images.  But I fear that you won't get the performance you want. TDMS is crazy fast and all, but it sounds like a bonkers amount of data real fast, and I don't know if it will be able to keep up, especially if there is special code needed to convert a string ready to be written to TDMS so the null (and any other characters you want) get written correctly.

Writing an array of bytes also sounds crazy for this application. The number of samples on the channels might approach NI's limitations for the data type, and then keeping track of where the offsets are for files would need to be a separate channel.  I've done stuff like this for logging raw CAN frames to TDMS files but that seems like way less data. Honestly you might be better served writing to binary files, then having the TDMS file just keep an index of the file names or locations for replay.  Not sure how I'd go about solving this.

[Antoine Chalons]

what is the size of you images and what's the max duration you need to save for?

180FPS is not that much, you could just save images to a SSD or if needed to a RAID drive

just make put acquisition in parallel of save and pre-allocate enough buffers to store the acquired images.

in 2007 (before SSD existed) I worked on an image streaming to disk system, up to 8000fps (images were tiny and monochrome), max duration was 30 seconds, but with a ggod SSD this is nothing.

[Thang Nguyen]

Thank you. I will try with SQLite then.

[infinitenothing]

In my experience SQLite will be much slower than TDMS. My recommendation is to reshape whatever array you have into a 1D array and then that can be your waveform. Store the image dimensions in metadata. Indexing is not challenging. You can get the the waveform size out of the channel properties (NI_ChannelLength) and divide that by how many pixels are in each image to get how many images are in your TDMS. You can pull out a given image index by multiplying the image index by the image pixel count and feeding that into the offset input for TDMS read and using the pixel count for the count input. Everything takes in i64s so you don't have to worry about large indexes. You just reshape the 1D array the read.