High speed, continuously stream the data from NI 5164 to NI 8267

[Tuan Nguyen]

Hello everyone,

It is long time not using LabView, and now I have one project that need to capture the data continuously with the following requirements:

• Number of channel: 02
• Sampling rate 1Gs/s simultaneously on both channel
• Continuously sample and save data in longer than 10 seconds

I am going to use the Labview with the following hardware:

• NI PXIe 5164: 2 channel, 1Gs/s,14 bits ADC
• NI PXIe 8267: 4T M2 SSD
• PXI chassis with controller

My question is: The NI 5164 at the maximums sampling rate will push to the host 3.2GB/s of data, can we save this data directly to the SSD without any interruption?

I check the Labview example and found there are example name: "niScope EX Stream to Disk". But I do not sure how it work, do the data transfer from Scope module directly write to the disk or the scope module transfer to the RAM memory then from the RAM the software write it back to the disk. 

Because the limitation of the bus bandwidth, I am afraid that if the data is transfer to RAM then write to the SSD there will be interruption during the data acquisition.

Please someone who have experience with this hardware or have been developed the same application help me. 

Thanks!

[Neil Pate]

Hi Tuan,

I would be a bit surprised if you could write to disk at 3.2 GB/s for any reasonable length of time even with a fast SSD. You say longer than 10s, how much longer? If you can buffer in memory all your data you could stream to disk as needed at a slower rate. 

Something I came across recently which I was surprised even existed, is a feature where an FPGA can use a DMA FIFO transfer to write directly to a TDMS file using a kind-of DVR (external DVR perhaps). Not sure if your hardware supports that but it might help with performance.

[Tuan Nguyen]

Thanks Neil,

For the record length, what I mean is: 10s is enough but longer is better.

The NI PXIe 8267 have read/wire speed of 5GB/s so from my point of view it still enough. I just not for sure how the Labview process the data flow, it come to the RAM first or it write directly to the SSD. 

Let me check the DMA FIFO. 

 

[Neil Pate]

this is the external DVR I was talking about 

 

[ShaunR]

You should speak directly to an NI Application Engineer.

I have worked with the PXI racks (PCI backplanes rather than PICe) and they had stipulations that there were slot boundaries so that you could only get the specified throughputs from certain slot configurations due to the backplane bus. They supplied me with with internal documents that detailed performance tests of the different configurations so they should have something similar for PCIe.

[Rolf Kalbermatter]

A Samsung EVO 960 has a maximum transfer speed of 1.5GB/s and that assumes that it uses NVE rather than SATA. With an EVO 970 you get close to 2GB/s. This are ideal rates and require that the PCI bus controller and disk have an optimal connection and the PCI bus controller has near perfect chipset drivers. The reality is generally somewhat below that and the software bindings inside user space are usually even less performant. Old SATA based SSDs max out at around 500MB/s and that is bus imposed. There is no way to go above that with SATA.

The FPGA DMA tech is pretty impressive but I would be surprised if they can go beyond 1GB/s.

Edited October 2, 2019 by Rolf Kalbermatter

[ShaunR]

1 hour ago, Rolf Kalbermatter said:

A Samsung EVO 960 has a maximum transfer speed of 1.5GB/s and that assumes that it uses NVE rather than SATA. With an EVO 970 you get close to 2GB/s. This are ideal rates and require that the PCI bus controller and disk have an optimal connection and the PCI bus controller has near perfect chipset drivers. The reality is generally somewhat below that and the software bindings inside user space are usually even less performant. Old SATA based SSDs max out at around 500MB/s and that is bus imposed. There is no way to go above that with SATA.

The FPGA DMA tech is pretty impressive but I would be surprised if they can go beyond 1GB/s.

To add to this. This document states that the maximum throughput of the bus is dependent on the number of links, each being 250MB. So a 16x slot shoud be capable of about 4GB/s. However it goes on to state that this is not the limiting factor for most NI products.

Edited October 2, 2019 by ShaunR