Multi-Device Acquisition

[JamesMc86]

Hi All,

I was hoping to crowdsource some of your experience today on a common application architecture that is bugging me slightly, but I can't decide whether it is genuinly bad or if it is just me. In the spirit of Steve Watts talk at the European CLA summit, something stinks!

The problem statement it solves is, I have multiple data sources, let's say they are the same rate but I have seen different rates as well, that I want to log to disk as a single file. I have seen the same solution several times involving FGVs, this is what makes me nervous!

What they do is have a seperate DAQ process per source of data and a single write to file process. Then either a)They have an FGV per DAQ process or b)A single FGV that different process write into different points in an array. Then in the log to file they either a)read from all FGVs or b)read from the FGV. I don't like this because:

1. There is no buffering, the only thing that guarantees getting the data is that the various loops stay in phase.
   
2. FGVs make the code pretty hard to read.
   

But it is hard to rip it to shreds when fundamentally people are succeeding with this. What I am interested in is what you guys have found to be a good solution to this problem. Ideally the data should be buffered which points to using queues, but another smell to me is waiting on multiple queues in one loop. One thought I have had is to have a data collator process, this would have to wait on multiple queues but then outputs the single set of data, or is this just b) from above again? I think this could be done in the actor framework as well where the collator then spawns the various DAQ actors.

So what are your thoughts, is there a deodorant to this or is this problem always going to smell!

Cheers,

James

[drjdpowell]

I usually save data as XY sets in a TDMS file, so different data sources can be at different rates and offsets. But if I wanted all data at the same time points, I would probably have a single queue to a “data logger” that all processes would send messages to (one queue, not many, with the source of the message as part of the message). Then the “Data logger” process could do any necessary resampling. This, IMO, has some advantages over using a Functional Global, though the advantages are pretty minor, at least for the typical use case of a steady data rate (where buffering is not needed).

— James

[mje]

I do this quite regularly. I have an application that fetches data from multiple devices each at different rates.

Each device has its own task running dumping data into a queue. Then there's a master task that at a regular rate grabs what's in the main input queues and generates output data and serializes as appropriate. The main task has to run at or slower than the slowest input rate.

What to do with data in the queues depends on your application. Mine simply flushes the queues grabbing whatever is there and runs it through a median filter to get the logged rate on par with the slowest component. I do this mostly because frequency aliasing and jitter means I won't always have the same number of elements waiting. Also has the benefit of improving the effective bit resolution on my measurements.

Correction, the median filter has no effect on resolution. Some of my tasks use mean filtering though, which does.

[ShaunR]

On the surface (from your description). it is a "many-to-one". So that really means Queue. My generic approach is to have multiple autonomous servers/processes/modules handling their own acquisition and placing data (after computation if necessary but sometimes to another module for crunching) onto a single "logging" queue. This "logging Queue (running also as a separate module/process) then saves data to a SQLite DB whenever there is something on the Queue. The DB can be queried in any which way you choose (between times, for a channel etc). Of course. Throughput is a consideration (how fast are you logging?) in which case the DB could be TDMS, however, it becomes much more difficult to extract the info once saved and you end up trading off resolution for convenience..

Edited April 24, 2012 by ShaunR

[viSci]

If you do not need more than 100,000 S/s and you own DSC, then you could stream to NI's very own real-time historical DB.

Too bad NI does not take Citadel out the DSC closet and make it available for general use.

[COsiecki]

Our method is similar to the ones described here. When we start recording, we pass references to all the channels we want to record to a process that extracts header info from the channels. It then creates a TDMS file and a common receive queue. Each channel then gets its own "data grabber" spawned. The data grabbers will dump data into the common queue when their channel updates (this action is event based). The original recording process will pull the data chunks from the common queue and write it to the appropriate channel in the TDMS file. When the recording stops, an event is fired that kills all the data grabbers. We use a parent data class with type-specific children as our queue elements so we can easily combine different data types in our file. We can also have multiple files grabbing the same data if we need to.