Circular data logging

[scls19fr]

Hello,

I would like to control a weather station using serial com.

I will get wind speed, temperature, pressure and I will put it in a cluster (I know how to do it).

I would like to store for the current day every minutes all data.

I will also store every hour the average values.

The problem with this is that my program will run every day, 24 h !

So the amount of data will increase too much.

So I wonder if there is a way to store my data in a circular buffer.

(like the C++/Boost circular buffer http://www.boost.org/doc/libs/1_38_0/libs/...lar_buffer.html )

but not in memory but in a file... or a database.

I don't want to use the LabVIEW DSC (Datalogging and Supervisory Control).

Kind regards

[jcarmody]

QUOTE (scls19fr @ Apr 5 2009, 12:52 PM)

So I wonder if there is a way to store my data in a circular buffer.

I really wanted to answer with this, but the answer I wanted to give wasn't at the top of the list, so I'll answer with this one. Finally, I watched

a long time ago when I'd never heard of a circular buffer. It's pretty ok.

[ragglefrock]

This depends on the nature of your data that you're saving in the cluster. It may or may not be very difficult to implement a true circular buffer in a file.

If your data packets (clusters) that you are saving are a fixed size, and you don't demand that the data in the file is all in chronological order, then this should be possible. By fixed size, I mean that what you save will always take up the same amount of bytes in the file. This would not be the case, for instance, if you wrote numeric data as text to your file without using a fixed width. If these restrictions work for you, then it's easy enough to write your fixed size packets to the file, and simply have a file pointer that wraps around back to the beginning once it reaches the fixed size of the circular buffer. Use the Set File Position function for the file pointer to set it back to zero.

If your application doesn't fit those restrictions, then the easier thing to do to implement this is to have 2 files, let's call them A and B. All you do is write to either A or B until it reaches a fixed size. Once that happens, you close the file, delete the other, and replace it by overwriting it with new data. This is super-simple, and it guarantees that you have the most recent data covered.

[scls19fr]

Thanks jcarmody for this very explicit answer... Google it yourself !

Maybe you don't understand difference between storing in memory and storing in a file

If you know the difference maybe a better link should be

http://www.google.fr/search?q=circular+file+buffer+labview

or a better link

http://zone.ni.com/devzone/cda/epd/p/id/5040

If I'm asking this question here it's to have some feedback of developers about

using such data structure. In my mind ragglefrock answers much better to my question

even he doesn't use the NI implementation of a circular file buffer.

I would be interressed in knowing the limits of the NI implementation.

Can I use 3 or 4 data structure like this one without any problem ?

Why doesn't they provide thiss "out of the box" in LabVIEW ?

[ragglefrock]

QUOTE (scls19fr @ Apr 8 2009, 12:48 AM)

If I'm asking this question here it's to have some feedback of developers about

using such data structure. In my mind ragglefrock answers much better to my question

even he doesn't use the NI implementation of a circular file buffer.

I would be interressed in knowing the limits of the NI implementation.

Can I use 3 or 4 data structure like this one without any problem ?

Why doesn't they provide thiss "out of the box" in LabVIEW ?

The limits of the NI example you posted are quite clear:

• The data they are storing (numeric) is fixed-size. If your data type is not fixed-size (i.e. string), then this method will not work very well without a lot of extra work to pad the data to some fixed maximum size.
  
• The data is not timestamped or otherwise ordered. If you open the file, you can't determine the original order, since the start of the data might be somewhere in the middle.
  

I would guess LabVIEW doesn't supply this out of the box for many reasons:

• LabVIEW is a strictly-typed language. This makes it difficult to provide arbitrary solutions to these types of problems. Instead, NI would have to choose a fixed set of data types to support in the circular buffer file. As soon as a user wanted to do the work with a custom type, like a cluster, they'd be out of luck. And NI would have to duplicate the work many times over to support these different data types. Duplicated work leads to more bugs, and a greater degree of difficulty fixing any bugs that do exist.
  
• It's tough to say what the optimal file format would be for different users. Human-readable text or binary files for better performance and smaller file sizes? Any binary standard in particular? XML?
  

It sounds like you want a Standard Template Library of sorts for LabVIEW, which would be nice in theory, but would probably only be obtainable through LVOOP, which is not supported on all targets, and doesn't have integrated support for native data types. So there would be a LOT of ground-up work to do to get anywhere. I would guess the community would probably be more effective at targeting such a solution than NI, but I'm done guessing for tonight

[Aristos Queue]

The LabVIEW queues are circular buffers. You enqueue data at one end, dequeue from the other. All the mechanics of circular buffering are handled for you. You can bound the queue to a particular size or have it grow if it exceeds the current buffer size. See the online help for the Obtain Queue function to get started.

This works for in-memory buffering. If you really need to go to disk, you'll need to do something like the examples you posted. Why isn't something "out of the box" supplied? Because generally the in-memory solution works, and when it doesn't, almost always some custom solution is required to manage the data because of the size and scale -- there's usually some custom requirement about how the buffering works, when is it acceptable to toss data and when is it required to hold, what kind of compression can be applied, etc.

[jdunham]

QUOTE (scls19fr @ Apr 5 2009, 09:52 AM)

The problem with this is that my program will run every day, 24 h !

So the amount of data will increase too much.

So I wonder if there is a way to store my data in a circular buffer.

Well if that's your only problem, a true circular buffer is not the best answer. Why not just store the data in a flat file (one row per measurement), and make a new file every day (please use file names like Weather YYYY-MM-DD so they sort in order) and then delete every file more than a month old.

I'm not sure I understood totally, but it sounded like you wanted to store three numbers once a minute. Together with a timestamp, that's about 50 bytes per minute or 26MB per year, give or take a few MB. Why purge the old records at all? Are you recording a lot more data that you didn't mention? Even recording every second, you should have no problem finding a big enough hard disk.

[scls19fr]

Hello,

You are right jdunham... the size would be about 26MB... but I don't like to know that size is increasing too much so I thought the idea of using circular buffer interessting...

Moreover circular buffer are used inside LabVIEW itself for storing data with diffent size (see for example strip charts... you can set the number of stored points).

So I thought using a circular buffer inside LabVIEW much more easier.

I know the C++/STL an I was looking for a similar container in LabVIEW...

maybe as suggested Aristos Queue, I should have a look at queue. I never used them before.

Kind regards

PS : sorry for being so late in replying but email notification still doesn't works

[Mark Yedinak]

QUOTE (Aristos Queue @ Apr 12 2009, 09:38 PM)

The LabVIEW queues are circular buffers. You enqueue data at one end, dequeue from the other. All the mechanics of circular buffering are handled for you. You can bound the queue to a particular size or have it grow if it exceeds the current buffer size. See the online help for the Obtain Queue function to get started.

This is only true for LV 8.6 and above. Prior to that you had to write wrappers around the enqueue to make the queue work as a circular buffer. Otherwise depending on how you created your queue you would either have a queue that would grow until you ran out of memory or stop enqueueing data if the maximum numbers of elements was reached.

[scls19fr]

I'm running LV 8.5.

Do you have a LV 8.5 sample that use this kind of wrapper around the enqueue ?

but I can get 8.6 because we own it at work...

maybe you have some samples to demonstrate this.

I never use queue in LabVIEW before... what is, in your mind, the first document I should read first.

I also wonder if a queue can contain a cluster

But in my mind queue and circular buffer are a little bit different

because in a queue each element in a sort of structure that contain 3 things :

value

pointer to the next value

pointer to the previous value

so a circular buffer made with queue is not in a contiguous memory area...

I wonder if it doesn't decrease performances.

A "real" circular buffer is a contiguous memory area for storing values and

an integer area storing the position where the next value will be stored.

Kind regards

Sebastien

[Grampa_of_Oliva_n_Eden]

QUOTE (scls19fr @ Apr 14 2009, 12:49 PM)

I'm running LV 8.5.

Do you have a LV 8.5 sample that use this kind of wrapper around the enqueue ?

but I can get 8.6 because we own it at work...

maybe you have some samples to demonstrate this.

I never use queue in LabVIEW before... what is, in your mind, the first document I should read first.

I also wonder if a queue can contain a cluster

But in my mind queue and circular buffer are a little bit different

because in a queue each element in a sort of structure that contain 3 things :

value

pointer to the next value

pointer to the previous value

so a circular buffer made with queue is not in a contiguous memory area...

I wonder if it doesn't decrease performances.

A "real" circular buffer is a contiguous memory area for storing values and

an integer area storing the position where the next value will be stored.

Kind regards

Sebastien

I used to use circular buffers regularly but since the introduction of the polymorphic queue (which means the queeu can be of any data type you want... including clusters). Here is a screen shot of what may have been my last circular buffer from LV 6.0 (judging by the use of string for my actions since they were all the rage back then along with my use of red comments whcih I have since discontinued).

http://lavag.org/old_files/monthly_04_2009/post-29-1239728948.png' target="_blank">

The reason I don't use circlular buffers is because Queues are faster (under most circumstances).

So leave teh circular buffer for those cases where you want to be able to survive a power failure (data written to disk) and get comfortable with Queues.

Ben

[Mark Yedinak]

Here is a basic wrapper that would turn a queue (prior to 8.6) into a circular buffer. You can ignore the cluster with the exception of the queue and queue size. This snippet of code is taken from a general purpose dqGOOP log object. If you wanted to be safe you could put the dequeue in a loop to ensure that all elements past the maximum size would be removed. However if you manage the queue properly the queue should never be larger than the maximum size.