grab at high frame rates and save

[dispossessed]

Hi,

I need to take images continuously at high frame rates (~60fps) and then save these images on hard disk. I have no prior knowledge of how many images i will eventually be needing and i dont want to save the images in AVI format, but rather PNG.

I tried using buffers to store my images while the camera is still taking pics and then saving them on the hard disk when the camera has finished acquisition, however, the frame rate that the camera is acquiring images at seems to be always much lower than 60fps.

Does any one have an idea about how to take pics at 60fps and at the same time save them in PNG format?

PS: softwares im using are LABVIEW 8.5 and IMAQ.

Thanks,

Leen

[crelf]

QUOTE (dispossessed @ Jun 5 2008, 12:06 AM)

...however, the frame rate that the camera is acquiring images at seems to be always much lower than 60fps.

I've http://mplab.ucsd.edu/standards/FourCameraSetup/A300_Series_Data_Sheet.pdf' rel='nofollow' target="_blank">confirmed that your camera can acquire at 60fps. Can you please upload your code so we can see if there's anything in there that's causing the bottleneck?

[dispossessed]

QUOTE (crelf @ Jun 5 2008, 06:54 AM)

I've http://mplab.ucsd.edu/standards/FourCameraSetup/A300_Series_Data_Sheet.pdf' rel='nofollow' target="_blank">confirmed that your camera can acquire at 60fps. Can you please upload your code so we can see if there's anything in there that's causing the bottleneck?

attached is the code that i was trying to modify. i found this code on one of the NI website pages.

Thanks,

Leen

[bmoyer]

QUOTE (dispossessed @ Jun 5 2008, 10:00 AM)

attached is the code that i was trying to modify. i found this code on one of the NI website pages.

Thanks,

Leen

Although not the worst example I've seen, it certainly is not the best. Try looking at the LL Sequence.vi found in LabVIEW under the Help-->Find Examples... menu. Click the Browse tab, and select the Directory Structure option. Then find the IMAQ folder and under this is the IMAQ Low Level.llb where you'll find the LL Sequence.vi. Add to this VI the PNG saving portion of your example.

As for not knowing how long you need to acquire, it is fairly difficult to allocate memory "on the fly" without losing frames and very complicated. The best thing to do it to is to allocate a lot of frames and then terminate the sequence by using the IMAQ Stop.vi (you'll have to have a parallel loop monitoring the abort button). In this parallel loop you can also monitor the progress by calling the IMAQ Status.vi to get the current frame.

In a nutshell, what you are trying to do is fairly complicated and may take a lot of time to develop.

Bruce

This question is also in the NI Forums at:

http://forums.ni.com/ni/board/message?boar...d=77174#M328986

[MikaelH]

QUOTE (dispossessed @ Jun 5 2008, 03:06 PM)

I need to take images continuously at high frame rates (~60fps)

...maybe you should look at GigE Vision:

GigE Vision: Under the Hood Video

--Mikael

[stever]

Leen,

Were you able to find a solution? I'm trying to do exactly what you describe, and I've yet to find something that will work.

Thanks,

Steve

[Antoine Chalons]

QUOTE (stever @ Feb 2 2009, 04:01 PM)

Were you able to find a solution? I'm trying to do exactly what you describe, and I've yet to find something that will work.

I'm just realizing that this thread is related to http://forums.lavag.org/not-saving-at-required-frame-rate-t11317.html' target="_blank">another one here. Same person starting the thread and same application apparently.

I think the code I posted on this thread can be used to do image streaming, depending on your specs (frame rate, number of images to save) it might not be an appropriate solution.

Tell us a bit more on your application.

PS : the code I posted on linked thread is a light version of a working application that worked very well for me with an A302 basler camera, the specs were 30 images per second during 10 minutes, every images had to be processed and the display and also saved to disk.

[stever]

Antoine,

Thank you very much for your speedy reply. I will have a look at the code in the link you provided to the other thread.

Here is some info on my application:

- Need to acquire at full frame rate of the camera, which is currently 60 Hz, but might be faster in the future (up to a few hundred Hz)

- We cannot lose frames, and each frame needs a millisecond-based timestamp (this is a scientific application, not for human viewing of frames)

- Modern PC (will eventually be on quad-core Core 2 with 4GB RAM, XP 32-bit, SATA HD non-RAID)

- NI 1427 frame grabber

- Sensors Unlimited SU320KTSX camera

- We have to display frames, but not necessarily every single one (enough so the operator can see what's going on)

- It'd be nice to do some processing on these frames before display, but if that's too slow, not a problem to skip this

- Capture triggered by user pressing a button

- Capture needs to be able to run essentially indefinitely, but probably wouldn't be for more than a couple hours.

Using:

- LabView 8.6

- IMAQ 4.1

Thanks again,

Steve

[Antoine Chalons]

QUOTE (stever @ Feb 2 2009, 06:02 PM)

Antoine,

Thank you very much for your speedy reply. I will have a look at the code in the link you provided to the other thread.

Here is some info on my application:

- Need to acquire at full frame rate of the camera, which is currently 60 Hz, but might be faster in the future (up to a few hundred Hz)

- We cannot lose frames, and each frame needs a millisecond-based timestamp (this is a scientific application, not for human viewing of frames)

- Modern PC (will eventually be on quad-core Core 2 with 4GB RAM, XP 32-bit, SATA HD non-RAID)

- NI 1427 frame grabber

- Sensors Unlimited SU320KTSX camera

- We have to display frames, but not necessarily every single one (enough so the operator can see what's going on)

- It'd be nice to do some processing on these frames before display, but if that's too slow, not a problem to skip this

- Capture triggered by user pressing a button

- Capture needs to be able to run essentially indefinitely, but probably wouldn't be for more than a couple hours.

Using:

- LabView 8.6

- IMAQ 4.1

Thanks again,

Steve

I guess frame site is 320*240, right ?

At 60Hz, if you choose 12 bit resolution that's 60 [Hz] * (320*240) [pix] * 12 [bit/pix] / 8 [bit/byte] = 6.59 Mo/s

That's a fairly low rate in terms of data to write to disk, but you'll have to manage the disk space if it runs for many hours (each hour will be 23 Go )

I think the "concept" of code I posted in the other thread applies well to your application, feel free to post any question you might have.

Hope this helps.

[stever]

Thank you. I am looking at it now - it looks promising. One comment - I think the case statement in the upper loop needs to be reversed to write files when the buffer is > 0 ? You are my hero!

[Antoine Chalons]

QUOTE (stever @ Feb 2 2009, 07:30 PM)

I think the case statement in the upper loop needs to be reversed to write files when the buffer is > 0 ?

I may have put some intentional bugs to force potential users to put some effort to understand it

I'm really happy if it helps you

[stever]

Antoine,

That is a proven learning technique, and I applaud you for your efforts

I have a question about your VI:

I understand the "camera" circular buffer (you have this labeled as the acquisition buffer) - this appears to be the same thing as a "ring" in NI's language (e.g. the LL Ring.vi example). And I understand the upper while loop and the queue as well.

The lower while loop waits until the ID of the last buffer available is updated, then it proceeds into the True case. The goal here is to get the new frame referenced by this ID put on the back of the queue. In order to get this image sitting in the camera circular buffer into the queue, you have created a separate buffer of 20 (or whatever number) frames, and you index this buffer to get one of those frames. In this extracted frame, you IMAQ Copy the camera buffer frame into the image buffer frame and then shove that image into the queue. I understand the HistoryBuffer.vi takes care of getting the proper index into the image buffer.

My question is why do you need a separate multi-element buffer to pull out an image and put into the queue? Couldn't you just allocate a single-frame buffer (N=1 instead of N=20)? Will this buffer ever have more than one image in it at any one time?

There's an example from NI for IMAQdx that a coworker of mine found. Unfortunately, there's not an exact match to IMAQ VIs, and IMHO, the IMAQdx VIs are better, but I can't use them (because I'm using CameraLink). The example is located at:

http://zone.ni.com/devzone/cda/epd/p/id/5859

It's similar to your example in many respects, but different in others. I'm not facile enough yet with LabView/IMAQ to render an opinion as to which one is better for what I'm trying to do.

Thanks again.

Steve

[stever]

Is it possible that the Enqueue Element.vi only queues a pointer to the image? If so, I see the use in having the additional buffer, since that's where the frame data that the queue manages actually resides. Since frame data will be stored in this image/save buffer with indexes that only increase and then wrap at N (N=20 in the example), am I correct that this is a circular image/save buffer and therefore it's possible to overwrite frames? For example, if your image processing loop did so much stuff that frames were pulled into this buffer faster than they were pulled out, you'd end up overwriting good data, yes? Which perhaps is why you pointed the previous guy to the number of elements in the queue (if that gets > N, you can throw an error or something)?

Very clever. You are obviously quite skilled at this.

Steve

[Antoine Chalons]

QUOTE (stever @ Feb 3 2009, 07:57 PM)

My question is why do you need a separate multi-element buffer to pull out an image and put into the queue? Couldn't you just allocate a single-frame buffer (N=1 instead of N=20)?

Well.. It's not always necessary to have a ring buffer (N>1), it really depends on your application and I am used to always put more at least 5 even if I one would be enough ; but starting from this example I cn put N=1 if I like, when I started my application I didn't have the camera neither the frame grabber and I had little experience in image acquisition an streaming to disk, so to be safe I made it that way (separate multi-element buffer).

QUOTE (stever @ Feb 3 2009, 07:57 PM)

Will this buffer ever have more than one image in it at any one time?

Again, with your application, probably not. I have another application running with a Basler A504 camera and PCIe 1429, we acquire 4000 frame per second (1000*128*8bit) during 1 second, so at that speed we don't even try to stream to disk, it's about 650Mbytes/s.. a CD ! So be configure a ring buffer, start the acquisition, stop it when all the fames have been acquired then we have all the time we want to save to hard drive. Complete different application, but the code is just slightly different

QUOTE (stever @ Feb 3 2009, 07:57 PM)

There's an example from NI for IMAQdx that a coworker of mine found. Unfortunately, there's not an exact match to IMAQ VIs, and IMHO, the IMAQdx VIs are better, but I can't use them (because I'm using CameraLink). The example is located at:

It's similar to your example in many respects, but different in others. I'm not facile enough yet with LabView/IMAQ to render an opinion as to which one is better for what I'm trying to do.

I've never really used (I mean not in customer applications) IMAQdx or IMAQ 1394, as far as I know IMAQdx is the latest version and include support for both GigE cameras and 1394 cameras.

Until now I've only used CameraLink, I know the VIs are different from IMAQdx, I think it's a different way to work.

QUOTE (stever @ Feb 3 2009, 07:57 PM)

Is it possible that the Enqueue Element.vi only queues a pointer to the image? If so, I see the use in having the additional buffer, since that's where the frame data that the queue manages actually resides. Since frame data will be stored in this image/save buffer with indexes that only increase and then wrap at N (N=20 in the example), am I correct that this is a circular image/save buffer and therefore it's possible to overwrite frames? For example, if your image processing loop did so much stuff that frames were pulled into this buffer faster than they were pulled out, you'd end up overwriting good data, yes? Which perhaps is why you pointed the previous guy to the number of elements in the queue (if that gets > N, you can throw an error or something)?

I think you got it, in the application for which I used this "concept" I had to make I was not losing images at 40 (or 50Hz depending on the image size) in 10 minutes, the data rate was about 120Mo/s so I also had another VI that was just checking the queue size to make sure I never over write the secondary "circular buffer". It was interesting graphing the queue size as time was going by.. it allowed me to "see" that cache size of my hard drive.. I'll try to find the report I made (it's almost 2 years old now.. hope I'll find it).

In the end I think I put N=100 because in 10 minutes I would reach a max queue size of about 40... In the end it's a basic FIFO with one loop putting data in at a rate we want to keep constant, and a second loop that saves/processes/displays data as fast as possible, but at a rate that for sure will not be constant.

Let's be honest it took me a while and help from my coworker to get to this.

It think you're on your way to a solution

PS : Share what you know, learn what you can.

[stever]

Antoine,

I have it all up and running now. Thanks again. IMHO, you should consider submitting this as an official NI example...good stuff!

Steve

[Chris Davis]

QUOTE (Antoine Châlons @ Feb 3 2009, 05:20 PM)

Again, with your application, probably not. I have another application running with a Basler A504 camera and PCIe 1429, we acquire 4000 frame per second (1000*128*8bit) during 1 second, so at that speed we don't even try to stream to disk, it's about 650Mbytes/s.. a CD ! So be configure a ring buffer, start the acquisition, stop it when all the fames have been acquired then we have all the time we want to save to hard drive. Complete different application, but the code is just slightly different

You know, you could stream that Basler 504 to disk now. Checkout the latest hard drive comparisons and raid card comparisons at http://www.barefeats.com' rel='nofollow' target="_blank">www.barefeats.com. RAID cards from Highpoint and regular terabyte hard drive from Seagate could get really close to 800 MBs/sec without costing an arm and a leg. If you have an arm and a leg to spare, checkout the solid state disks from Intel and others tested at barefeats, and you could be looking at a 4 drive array instead of an 8.

[Antoine Chalons]

QUOTE (Chris Davis @ Feb 6 2009, 04:12 AM)

You know, you could stream that Basler 504 to disk now. Checkout the latest hard drive comparisons and raid card comparisons at http://www.barefeats.com' rel='nofollow' target="_blank">www.barefeats.com. RAID cards from Highpoint and regular terabyte hard drive from Seagate could get really close to 800 MBs/sec without costing an arm and a leg. If you have an arm and a leg to spare, checkout the solid state disks from Intel and others tested at barefeats, and you could be looking at a 4 drive array instead of an 8.

800MB/s ... that's scary !

Thanks for the link, I'll have a look.

EDIT :

The application also as a "long measure" mode which last 10 days, in this mode image acquisition is done continuously at 3900 Hz and ~200 images per seconds are processed (object tracking) but not saved...

At the very beginning of the project (about 2.5 years ago) the customer wanted to know if it was possible to save to HD 10 days of images at 4000 fps...

Now, it seems the data flow is no longer a problem, but remains HD capacity.. 535 TB

QUOTE

I have it all up and running now. Thanks again.

Great !

QUOTE

IMHO, you should consider submitting this as an official NI example...good stuff!

I will think about it, that's a good suggestion.

[gvstemp]

Hi Antoine Châlons

I have been looking for a solution for "high speed grab with saving to disk". This particular thread seemed VERY interesting, however, you mentioned that you posted an example VI in some other thread and I cant seem to find that linked thread.

Particularly you linked to

http://forums.lavag....ate-t11317.html

and this link just takes me back to the lavag forum main page.

I am very curious at looking at your celebrated solution for highspeed grabbing. Also you mentioned in one of your replies above

==

I have another application running with a Basler A504 camera and PCIe 1429, we acquire 4000frame per second (1000*128*8bit) during 1 second, so at that speed we don't even try to stream to disk, it's about 650Mbytes/s.. a CD ! So be configure a ring buffer, start the acquisition, stop it when all the fames have been acquired then we have all the time we want to save to hard drive. Complete different application, but the code is just slightly different

===

As for my attempts, I can tell the following:

I am trying to acquire images at 3000-4000Hz (400x400x8bit), for as long as I possible can. I am using a CMOS Mikrotron MC1362 camera with PCIe1433 card.

In my first attempt I am trying to acquire a set number of frames N (say N= 4000). So used Queues (of size N+1000) and Grab Acquire VI. As soon as I grab the image, I assign a time stamp to it and this cluster of (Image + time stamp), I write to a queue. Once the entire N frames are acquired, I Dequeue and post process.

I first tried to write the purple IMAQ image reference wire (rookie mistake, thinking that that was the actual image) to the queue and realised that I was writing the last frame N times to the queue.

(Hence I reached to this Forum).

At present my solution is: After the Grab Acquire.Vi, I convert Image to Array and write this array (IMAQ to Array.VI) to the queue (with time stamps). At the Dequeue end, I convert the array back to image for post processing.

This works.... but I am limited to 1500-1900 Hz only. The camera (at this image size) is capable of 3000Hz. The whole convert-to-array step slows down the system.

I would really like to look at your code that you used for 4000Hz acquisition, as well as the code for 'long measurement'.

Since I cannot seem to find the examples that you had uploaded in the linked thread, can you please upload the example again ? will appreciate much.

Thanks.

GVS

[jdunham]

In my first attempt I am trying to acquire a set number of frames N (say N= 4000). So used Queues (of size N+1000) and Grab Acquire VI. As soon as I grab the image, I assign a time stamp to it and this cluster of (Image + time stamp), I write to a queue. Once the entire N frames are acquired, I Dequeue and post process.

Aren't there native IMAQ functions for writing images to disk? Why don't you stream your input straight to disk, and keep a list of frame numbers and/or file names and timestamps in your queue? That gives you an index into your collection of disk image files, and you can access them in your post-processing step.

[gvstemp]

thanks for the reply !

Am a newbie at IMAQ functions. I looked again at the NI-IMAQ functions to find a VI that can do high speed (3k-4k frames per second) streaming to disk. Couldnt find.

Can you please elaborate which VIs would help ?

An example would also be greatly appreciated.

I was also thinking using the "LL Sequence.VI" to acquire N images. however I dont know and couldnot figure out a way to get time stamps for these frames. In post processing steps I use time information in the object tracking. So that might not be a solution for me.

Any other advice will be appreciated.

thanks

Aren't there native IMAQ functions for writing images to disk? Why don't you stream your input straight to disk, and keep a list of frame numbers and/or file names and timestamps in your queue? That gives you an index into your collection of disk image files, and you can access them in your post-processing step.

[gvstemp]

I have a relted question. Actually the more I read on this, the more questions I have

So seems like there are a few ways of doing this ( this = high speed imaging (3000Hz-5000Hz), then writing to disk and later post-processing)

1. Use Grab VI and attach a time stamp to it (like for the filename) and write this combo. to a queue, later dequeue to post process.

2. using "LL Sequence.VI"

3. using " LL ring.VI" or "buffered ring"

In Grab VI... the timestamps is the way to check FOR SURE that the frame rate is what I set it to be. ( Framerate = #frames/Total Time; and (N- (N-1)) frametimestamp = exposure time (should be constant)

however, in the "Sequence" or "Ring" option, is there a way to get accurate time stamps of the acquired frame ? Seems like I can only set total number of frames (N) and maybe measure total time for acquisition and get the average framerate. Not very good for me, since I cant be sure that every frame was acquired with deltaT = (1/framerate).

So whats the final word on "best way" to do highspeed image acquisition and writing to disk problem ?

[jdunham]

So seems like there are a few ways of doing this ( this = high speed imaging (3000Hz-5000Hz), then writing to disk and later post-processing)

1. Use Grab VI and attach a time stamp to it (like for the filename) and write this combo. to a queue, later dequeue to post process.

2. using "LL Sequence.VI"

3. using " LL ring.VI" or "buffered ring"

In Grab VI... the timestamps is the way to check FOR SURE that the frame rate is what I set it to be. ( Framerate = #frames/Total Time; and (N- (N-1)) frametimestamp = exposure time (should be constant)

however, in the "Sequence" or "Ring" option, is there a way to get accurate time stamps of the acquired frame ? Seems like I can only set total number of frames (N) and maybe measure total time for acquisition and get the average framerate. Not very good for me, since I cant be sure that every frame was acquired with deltaT = (1/framerate).

So whats the final word on "best way" to do highspeed image acquisition and writing to disk problem ?

Sorry I don't have IMAQ loaded on my current computer, but I can give some general observations.

If at all possible you should try for a hardware timed acquisition with frame buffer handling done at the driver level. With a really high frame rate like you have, it seems way too risky to attempt any kind of queue-based software buffering at the LabVIEW level.

I would try to use LL Ring.vi or Buffered Ring since your goal is continuous acquisition. I don't know how your frames are synced and timed on the hardware side, and so that could affect the system, but it seems like the Ring grab should be able to give you an accurate timing of frames. It shouldn't just be dropping frames, or if it does, you should try to fix that somehow so that you can grab without any dropped frames.

If you get to that point, then the frame timestamp should be a simple multiplication of the frame number and your frame rate, added to the start time.

If your framesync isn't even regular, like if it is based on some detection sensor, then you have a bigger problem. In that case I would get an NI mulitfunction or counter/timer card and use the RTSI bus to share the framesync with a hardware timer, and then you can build up a buffer of timestamps associated with framesyncs.

If you can get that hardware-timed ring grab working, then you should be able to set up a parallel thread to monitor the current frame number and write any new frames to disk, hopefully without converting them to 2D arrays first (which is really slow); instead using some kind of IMAQ function to save the file.

At any rate, my suggestion would be to get to the point where you can rely on every frame being acquired with a known delta time, because I don't think you'll be successful measuring it in software at those high rates.

Good luck,

Jason

P.S. It probably would have been good to start a new thread since it's a new problem.

[gvstemp]

Thanks a ton for the advice.

It seems like using LL Ring for my application is the best bet and to trust that at the camera end there is not jitter in the frame rate (once set).

I use a MC1362 Mikrotron camera, does any one has any experience/benchmarks on how reliable the frame-rate is ?

Anyway, I will try to set up the ring and see if I can record at 3-4kHz for long enough times.

Thanks a ton for your time and advises

Sorry I don't have IMAQ loaded on my current computer, but I can give some general observations.

If at all possible you should try for a hardware timed acquisition with frame buffer handling done at the driver level. With a really high frame rate like you have, it seems way too risky to attempt any kind of queue-based software buffering at the LabVIEW level.

I would try to use LL Ring.vi or Buffered Ring since your goal is continuous acquisition. I don't know how your frames are synced and timed on the hardware side, and so that could affect the system, but it seems like the Ring grab should be able to give you an accurate timing of frames. It shouldn't just be dropping frames, or if it does, you should try to fix that somehow so that you can grab without any dropped frames.

If you get to that point, then the frame timestamp should be a simple multiplication of the frame number and your frame rate, added to the start time.

If your framesync isn't even regular, like if it is based on some detection sensor, then you have a bigger problem. In that case I would get an NI mulitfunction or counter/timer card and use the RTSI bus to share the framesync with a hardware timer, and then you can build up a buffer of timestamps associated with framesyncs.

If you can get that hardware-timed ring grab working, then you should be able to set up a parallel thread to monitor the current frame number and write any new frames to disk, hopefully without converting them to 2D arrays first (which is really slow); instead using some kind of IMAQ function to save the file.

At any rate, my suggestion would be to get to the point where you can rely on every frame being acquired with a known delta time, because I don't think you'll be successful measuring it in software at those high rates.

Good luck,

Jason

P.S. It probably would have been good to start a new thread since it's a new problem.