simultaneous read/write of files

[PA-Paul]

Hi All,

I'm in the planning stages of an application and was after a bit of information.

I'm doing some continuous DAQ and was wondering if it might be sensible to stream the data from the card (USB 6211) direct to disk in a binary (I've seen this a couple of times - binary being the quickest way to get the data to disk), but I also need to process that data relatively quickly after its been acquired (acquisition rate is 200Hz - 1KHz, but I need to get analysis done at at least 20 Hz).

Could I continuously acquire and stream to disk in one loop, whilst periodically reading back some of the data for processing in a second loop (running slower than the daq loop)?

Also, has anyone used the "USB signal stream" technology with the USB daq cards? the only thing I can find on it is this from the manual

"USB Signal Stream—A method to transfer data between the device

and computer memory using USB bulk transfers without intervention

of the microcontroller on the NI device. NI uses USB Signal Stream

hardware and software technology to achieve high throughput rates

and increase system utilization in USB devices."

and

"USB-621x devices have four dedicated USB Signal Stream channels. To

change your data transfer mechanism between USB Signal Streams and

programmed I/O, use the Data Transfer Mechanism property node

function in NI-DAQmx."

Does that mean the DAQmx code still looks essentially the same? I mean I still just use the "DAQmx" read VIs etc as I always have?

Anyways, as I say I'm in the planning stage, so I don't have any examples to post...

Any thoughts (however random!) would be welcome!

Cheers

Paul

[jdunham]

QUOTE (Ic3Knight @ Feb 13 2009, 02:04 PM)

I'm in the planning stages of an application and was after a bit of information.

I'm doing some continuous DAQ and was wondering if it might be sensible to stream the data from the card (USB 6211) direct to disk in a binary (I've seen this a couple of times - binary being the quickest way to get the data to disk), but I also need to process that data relatively quickly after its been acquired (acquisition rate is 200Hz - 1KHz, but I need to get analysis done at at least 20 Hz).

Could I continuously acquire and stream to disk in one loop, whilst periodically reading back some of the data for processing in a second loop (running slower than the daq loop)?

It might be possible, but it's certainly foolish.

A better approach is to make one process to write the DAQ data to disk, and make a separate process to crunch your data. The DAQmx API allows you to read the same data more than once, so each thread can keep track of its own read mark and operate independently. I have done this lots of times.

[PA-Paul]

QUOTE (jdunham @ Feb 14 2009, 03:03 AM)

It might be possible, but it's certainly foolish.

A better approach is to make one process to write the DAQ data to disk, and make a separate process to crunch your data. The DAQmx API allows you to read the same data more than once, so each thread can keep track of its own read mark and operate independently. I have done this lots of times.

I'm a little confused, could you post a quick example of what you mean about reading data more than once? Do you mean something like this:

http://lavag.org/old_files/monthly_02_2009/post-14639-1234610446.png' target="_blank">

Or do you mean create two separate daq tasks for the same channel?

Thanks for your help!

Paul

[jdunham]

QUOTE (Ic3Knight @ Feb 14 2009, 03:22 AM)

I'm a little confused, could you post a quick example of what you mean about reading data more than once? Do you mean something like this:

Or do you mean create two separate daq tasks for the same channel?

No it's just one task, but you need to work with the DAQmx property nodes to control where each thread is reading from. It's more complex than your sample, but nothing crazy hard. Check out this old post about a similar topic.

[PA-Paul]

Hi,

Thanks for the info, I've had a look through and will have a play with that...

One other DAQmx question that might help to clear things up for me... If I setup the following:

The DAQ device starts acquiring data from channel 0 at a constant rate (in this case 1000 Hz). My question is "where is that data stored"? I'm using a USB6211 multifunction DAQ device, is the data being streamed to my PC even if I don't call the read function? Or is it sat in a buffer on the card waiting for me to call the read VI?

Sorry if this is a silly question, I've been using DAQ for a while, but I've never delved too deeply under the hood!

Thanks for your help!

Paul

[LAVA 1.0 Content]

QUOTE (Ic3Knight @ Feb 15 2009, 12:14 PM)

The DAQ device starts acquiring data from channel 0 at a constant rate (in this case 1000 Hz). My question is "where is that data stored"? I'm using a USB6211 multifunction DAQ device, is the data being streamed to my PC even if I don't call the read function? Or is it sat in a buffer on the card waiting for me to call the read VI?

The data is stored in the on-board buffer, or in the systems ram (I am not sure). However if you wait too lang after a read you will get a buffer over-flow error.

What you want can be done with TDMS files, simultaneously write to the file and read from the file.

Make sure you open the TDMS file twice and make sure you don't read more than is in the file.

Ton

[jdunham]

QUOTE (Ton @ Feb 15 2009, 12:28 PM)

The data is stored in the on-board buffer, or in the systems ram (I am not sure). However if you wait too lang after a read you will get a buffer over-flow error.

Mostly in the system's RAM, or at least that's all you have to worry about. So DAQmx has a private circular buffer system which you don't have to worry much about. Except that the buffer looks linear to you (the offset from the start mark increases infinitely and is a U64 (though you can only access it with an I32 which is just a screw-up on their part--backwards compatibility etc. etc.). But under the hood, it's a circular buffer, so if you try to access data which is too old, you will get error -10846 (which is pretty familiar to people doing a lot of DAQ). Actually that was the NI-DAQ legacy error code and I don't remember whether DAQmx uses different codes.

So here's the part you have to care about: The circular buffer size determines how far back into the past you can retrieve data. DAQmx will probably take your sample rate and give you a reasonably-sized buffer (not sure exactly how big, since I haven't mucked with it for a while). If it is not giving you enough, you can ask for more with

C:\Program Files\National Instruments\LabVIEW 8.6\vi.lib\DAQmx\configure\buffer.llb\DAQmx Configure Input Buffer.vi

I actually thought it was via a DAQmx property node, but either senility is setting in or else they changed DAQmx.

Actually I checked again, and inside that VI is simply the property node. The property node is useful because you can change it to read-mode and find out what the current buffer size is.

[PA-Paul]

QUOTE (jdunham wrote)

Stuff about DAQmx

Thanks, that clears things up. For some reason, I thought the DAQ device (in my case the USB 6211) was storing the data "on-board" and waiting for my labview code to "go and ask" for the data before sending it to the PC... It makes more sense if DAQmx is getting it directly as fast as possible and I'm simply asking DAQmx for the data (also explains how the whole multiple reads of the same data can work!)

One more question (sorry!!), if I set up a continuous acquisition task, like the one in the picture I posted last, but never read back any data, will DAQmx throw up an error? Or will it only give the error if I try to read back samples its over-written in the circular buffer? For example, could I set up the continuous task, and then wait an hour (or whatever) and then use the "DAQmx read" property node to get the most recent sample? (I'm not sure why I'd want to do this, but it helps me build a better picture of how it all works!)

Thanks again for the help! I'm glad I found these forums!

Paul

[jdunham]

QUOTE (Ic3Knight @ Feb 16 2009, 02:35 PM)

Thanks, that clears things up. For some reason, I thought the DAQ device (in my case the USB 6211) was storing the data "on-board" and waiting for my labview code to "go and ask" for the data before sending it to the PC... It makes more sense if DAQmx is getting it directly as fast as possible and I'm simply asking DAQmx for the data (also explains how the whole multiple reads of the same data can work!)

Yes, the cool thing is that your code generally works the same on all DAQ boards, but you can just ask it to do those same things faster if it is an expensive board and/or has a big buffer versus a cheap board that has no buffer at all.

QUOTE (Ic3Knight @ Feb 16 2009, 02:35 PM)

One more question (sorry!!), if I set up a continuous acquisition task, like the one in the picture I posted last, but never read back any data, will DAQmx throw up an error? Or will it only give the error if I try to read back samples its over-written in the circular buffer? For example, could I set up the continuous task, and then wait an hour (or whatever) and then use the "DAQmx read" property node to get the most recent sample? (I'm not sure why I'd want to do this, but it helps me build a better picture of how it all works!)

Yes, absolutely. The circular buffer can acquire just about forever, and it never cares whether you read the data. But as you are finding, if you wait a long time and read with the default read function, you'll get data not found. A corollary to that, never adequately explained by NI, is that just because your read function throws an error, doesn't mean the acquisition is harmed, and you are perfectly free to try a different read from a different position.

You could easily have a system which is acquiring very fast, but you are only interested in the data when some other external event happens, like your motion control system gets everything aligned. Then you could record several seconds on either side of the event since your continuous acquisition was buffering the whole time. There is also a 'pretrigger' mode which does pretty much the same thing, but you would have to re-arm the acquisition to do it more than once. Once you have mastery over the buffer you can do anything you want.

QUOTE (Ic3Knight @ Feb 16 2009, 02:35 PM)

Thanks again for the help! I'm glad I found these forums!

Glad to help. I find it frustrating that DAQmx is so powerful, and there are a great many examples, but some of the coolest stuff you can do is entirely missing from any examples.

[PA-Paul]

Hi, Sorry to resurrect this...

I'm playing with reading back data from the DAQmx buffer...

What I wanted was one DAQ task - in my example, a simple AI read, continuous sampling, but I wanted to read the data back in two parallel loops... I set up the attached VI as a test... what should happen is the top loop simply reads back the most recent sample available and places it in a shift register (building an array - not the most efficient method I know). The second loop should read back all of the data (in blocks of 20), but I've delayed the loop so the read will ulitmately fall behind the buffer.

In this case, what you should see is the two waveform graphs "oscillating at different rates"... but there seesm to be some kind of "cross talk" going on... like the two "reads" and property nodes are interfering with each other... Is what I want to do actually possible? Or did I miss understand what was said above:

QUOTE (JDunham)

The DAQmx API allows you to read the same data more than once, so each thread can keep track of its own read mark and operate independently. I have done this lots of times.

Any help gratefully recieved!

Thanks

Paul

[jdunham]

QUOTE (Ic3Knight @ Feb 21 2009, 08:10 AM)

What I wanted was one DAQ task - in my example, a simple AI read, continuous sampling, but I wanted to read the data back in two parallel loops... I set up the attached VI as a test... what should happen is the top loop simply reads back the most recent sample available and places it in a shift register (building an array - not the most efficient method I know). The second loop should read back all of the data (in blocks of 20), but I've delayed the loop so the read will ulitmately fall behind the buffer.

In this case, what you should see is the two waveform graphs "oscillating at different rates"... but there seesm to be some kind of "cross talk" going on... like the two "reads" and property nodes are interfering with each other... Is what I want to do actually possible? Or did I miss understand what was said above:

How can you tell there is cross-talk? Your top loop should be producing an essentially random graph. It's just taking the last sample periodically, but with software timing, so there's no strict periodicity to your reading. how could you tell if it's not actually reading the last sample? Without seeing the data, it's hard to know whether it's DAQmx or just your program which produces the aberrant results.

The lower loop should be reading continuous data, so if it looks wrong, that's a stronger indicator that you are having problems.

It's certainly possible, and maybe even likely, that the property node just affects the task and when you call the DAQmx read VI, it's going to use the effect of the last property node that executed, regardless of which loop that was in.

The easiest way to check that is to put the property node and the DAQmx Read VI into a new subvi, and it has to be the same subvi and not reentrant, and then the subvi calls will block each other and you can guarantee that the property node and DAQmx Read are called in the order you are expecting. You could also use a semaphore to do the locking, but a subvi is usually easier. After that, if the output still doesn't look right to you, it's probably not DAQmx's fault.

BTW you've made me wonder if the last program I wrote using this kind of trick needs to have some semaphores added!!

[PA-Paul]

QUOTE (jdunham)

How can you tell there is cross-talk? Your top loop should be producing an essentially random graph. It's just taking the last sample periodically, but with software timing, so there's no strict periodicity to your reading. how could you tell if it's not actually reading the last sample? Without seeing the data, it's hard to know whether it's DAQmx or just your program which produces the aberrant results.

Sorry, should perhaps have explained a little more.

The daq device in the code (Dev2/AI0) is simulated (using MAX) and produces a very slow sinusoidal variation (I don't know if this is default behaviour for a simulated device?!). Since the sinusoid is very low frequency (compared to the 100 ms delay in the top loop), I expect both loops to produce a sinusoid. However, since the lower loop is not reading back quickly enough (the delay is set to 30 ms, and its reading back data in blocks of 20 with a sample rate of 1KHz), its sinusoid should appear slower than that of the top loop...

This works to an extent, however in the version I posted, the lower tends to produce a reasonable sinusoid with occasional "glitches", but the top loop produces a sinusoid but with lots of "glitches" which, if observed over a long enough period of time, show that its actually reading back the wrong data:

When I placed the loop contents into subvis, I think they argued a bit! there are short (1-3 s) interruptions where nothing appears to happen, and the sinusoid in the top loop became discontinuous (If it can really be called continuous in the above picture anyway!). This is the result of that:

So, basically, it seems that splitting the DAQ task between two parallel loops does not force DAQmx to create a separate, independent "thread"... so you can't (it would appear) access the buffer "parallely"...

I'm not really sure where to go next with this. What I was hoping for was a way to read data from the same daq task in two (or more) locations within one application. (similar to this little demo), but so far at least this doesn't look possible... (short of using TDMS and "buffering" the data to disk anyway...)

Any more thoughts?

Thanks for your help!

Paul

[jdunham]

QUOTE (Ic3Knight @ Feb 22 2009, 03:40 AM)

The daq device in the code (Dev2/AI0) is simulated (using MAX) and produces a very slow sinusoidal variation (I don't know if this is default behaviour for a simulated device?!). Since the sinusoid is very low frequency (compared to the 100 ms delay in the top loop), I expect both loops to produce a sinusoid. However, since the lower loop is not reading back quickly enough (the delay is set to 30 ms, and its reading back data in blocks of 20 with a sample rate of 1KHz), its sinusoid should appear slower than that of the top loop...

I've never used the simulated device. It's certainly possible that the simulator itself is buggy. If you have a typical NI DAQ card at your disposal, you can set up one of the analog outputs for continuous generation, and then use a pair of wires to bridge that into your analog input, and then you're testing against a real signal. On many of the cards, you can specify that the analog output is stored entirely on the on-board FIFO, and then you will get very precisely timed output no matter what the computer or processor is doing (short of rebooting). Just store one cycle of a sine wave in the analog task and make sure it is small enough to fit into the fifo, and then dig around the DAQmx properties and find where you force it to use the fifo.

QUOTE (Ic3Knight @ Feb 22 2009, 03:40 AM)

This works to an extent, however in the version I posted, the lower tends to produce a reasonable sinusoid with occasional "glitches", but the top loop produces a sinusoid but with lots of "glitches" which, if observed over a long enough period of time, show that its actually reading back the wrong data:

...

The graphs are very interesting, and obviously show that something is wrong. What I can tell you is that I have an application at 1kHz that has been using this technique, with one thread logging to disk for up to a few hours during a run, and another thread doing event-based real-time analysis on the same channel, and it has never shown any kind of data corruption.

[PA-Paul]

I tried again this morning with a real DAQ device (NI USB6211) and an external signal generator, and saw the same results as above. I then tried again, adding a semaphore to the code to protect the "Read property node and Read.VI" combinations in each loop. This worked a treat...

I guess, thinking about it, it makes sense. All I'm doing in my code is sending a DAQmx task NAME reference to each loop. There is still only one DAQ task... Its also quite possible, with the two parallel loops, that the top loop's read property node may be called after the bottom loop's property node but before the bottom loops Read VI, hence changing the properties for the bottom loop read... (which would explain what I was seeing in my graphs).

Using semaphores to protect each "property - read" combination seems to work fine and I presume won't affect performance significantly (unless, I suppose, the bottom loop is doing a particularly long read or having to wait for new samples to be acquired into the buffer...)

Anyway, since this isn't really a "data base and File IO" topic any more, can someone tell me how to get it shifted to a more relevant section? I'd be interested if anyone else has any thoughts on the matter!

Cheers for your support everyone!

Paul

[jdunham]

QUOTE (Ic3Knight @ Feb 23 2009, 01:24 AM)

I tried again this morning with a real DAQ device (NI USB6211) and an external signal generator, and saw the same results as above. I then tried again, adding a semaphore to the code to protect the "Read property node and Read.VI" combinations in each loop. This worked a treat...

I guess, thinking about it, it makes sense. All I'm doing in my code is sending a DAQmx task NAME reference to each loop. There is still only one DAQ task... Its also quite possible, with the two parallel loops, that the top loop's read property node may be called after the bottom loop's property node but before the bottom loops Read VI, hence changing the properties for the bottom loop read... (which would explain what I was seeing in my graphs).

Using semaphores to protect each "property - read" combination seems to work fine and I presume won't affect performance significantly (unless, I suppose, the bottom loop is doing a particularly long read or having to wait for new samples to be acquired into the buffer...)

Great. I think your analysis is spot-on. Note that you can use a common subvi in place of semaphores, but they should both be fast

[PA-Paul]

Hi All,

Sorry to resurrect this thread, but I've just discovered something relevant which I wanted to add....

In short - Beware the DAQmx read property node when using a USB based DAQ card!

The option for where to read from (i.e. relative to first read, most recent etc) does not work well when set to "most recent". In my application, I was acquiring data continuously at a fixed rate and then (at regular but not necessarily fixed intervals) using the "most recent" setting to try to read back only the most recent data point. I was then using this to provide feedback to control a 2nd device which would in turn influence what was read by the daq card. In essence, the daq device was providing data for a feedback loop. I found that my feedback loop, instead of providing a steady state for my system, was oscillating. After much testing and discussion with a colleague, we discovered that it was because the DAQ read was not actually providing the most recent data sample at all, but was always one iteration behind in the feedback loop! It appears that this arises from the way data is transferred from DAQ devices over the USB bus. The card buffers some data "internally", before sending "bulk packets" across the bus to improve efficiency (As opposed to say a PCI based card which uses DMA). When I set the DAQmx read property node to "most recent" and then read back the data, I was getting the most recent data from the DAQmx buffer, and then immediately afterwards DAQmx would go and get the next data from the card. So then, on the next iteration, I still only get what's in the DAQmx buffer which was actually sent from the card on the previous iteration of my feedback loop, meaning my calculations for control were not based on the most recent data at all, but were one loop iteration out - hence the oscillation!

There's a thread over on the NI forum which confirms this behaviour for USB based devices.

Anyways, as USB devices appear to be becoming more common for DAQ due to convenience, I thought I'd just post this up (not that this thread is really in the right forum for this anymore - any mods want to move it?).

Cheers

Paul