2D array size memory matters?

[Michael ten Den]

Hello,

I'm new on this forum and this is my first post, so excuses me if I'm posting it on the wrong place.

I busy with getting data from 3 lineair potentiometers.

Now is my question.

Does it matters for saving speed and memory if I store the data in seperated array's with time stamp in it or all 3 in 1 array with time stamp.

(see attachment, Excuses of the text with (int) in it. Off course is it a dbl)

____________

Michael ten Den

[Matt W]

Hello,

I'm new on this forum and this is my first post, so excuses me if I'm posting it on the wrong place.

I busy with getting data from 3 lineair potentiometers.

Now is my question.

Does it matters for saving speed and memory if I store the data in seperated array's with time stamp in it or all 3 in 1 array with time stamp.

(see attachment, Excuses of the text with (int) in it. Off course is it a dbl)

____________

Michael ten Den

With memory it's cheaper to have one array. Assuming the difference between times is constant then it's even cheaper to use waveforms which encode times as a starttime and the difference between times. Speed wise it depends on what you're doing to the data but it's probably going to be best as waveforms, then a single arrays then multiple arrays.

Edited December 14, 2010 by Matt W

[jgcode]

Hi Michael - welcome to LAVA

Does it matters for saving speed and memory if I store the data in seperated array's with time stamp in it or all 3 in 1 array with time stamp.

I agree with Matt - the first is preferred over the second as you have a copy of the time data in all arrays.

The answer may depend on the what you are doing with the data as well:

• Will the arrays be in memory at the same time?
• Do you need that format to be created i.e. to write to separate files?
• Are the arrays massive or really small?
• What platform will you run on (desktop or Real Time)?

LabVIEW is good in that, in general, it handles calls to the memory manager. You may find the second does not make a big difference and might be easier depending on what you are doing etc... If you are ever unsure its easy to run quick performance tests in LabVIEW too.

Cheers

-JG

[Michael ten Den]

Thanks jgcode & Matt,

I'm going to monitor the suspension of a car with 3 lineaire potetion meters. The important of all is that I have sample rates of 1 kHz. Higher is even better.

So the array will be quick huge. The LabVIEW program has to run the hole day on my laptop.

I'm getting my info trough UDP. There is a CAN network on and with a Analog2CAN converter i'm reading my potentio meters.

I get about 200 ID's in my program and I'm putting them in several arrays, but is this the best, less failure, fast way?

Is waveform then still better?

Looking forward for your answers,

____________

Michael ten Den

[ShaunR]

Thanks jgcode & Matt,

I'm going to monitor the suspension of a car with 3 lineaire potetion meters. The important of all is that I have sample rates of 1 kHz. Higher is even better.

So the array will be quick huge. The LabVIEW program has to run the hole day on my laptop.

I'm getting my info trough UDP. There is a CAN network on and with a Analog2CAN converter i'm reading my potentio meters.

I get about 200 ID's in my program and I'm putting them in several arrays, but is this the best, less failure, fast way?

Is waveform then still better?

Looking forward for your answers,

____________

Michael ten Den

You are probably better off logging to a file since you will have a huge dataset.

[SuperS_5]

Hi, I am guessing that you will be using polling then. I think that I would use the waveform data-type for short-term processing, and display. (Although I have used simple arrays for inherited code) Saving the data to file would be highly recommended if logging will be long-term. Also, If you don't care about the other 197 devices on the CAN network, I would recommend ignoring those datapoints.

[Mellroth]

...The important of all is that I have sample rates of 1 kHz. Higher is even better.

So the array will be quick huge. The LabVIEW program has to run the hole day on my laptop.

I'm getting my info trough UDP. There is a CAN network on and with a Analog2CAN converter i'm reading my potentio meters.

I get about 200 ID's in my program and I'm putting them in several arrays, but is this the best, less failure, fast way?...

I agree with Shaun on this one, the best is probably to stream data to disk.

If you go the memory path anyway, be aware of that having all data in one array might fail due to that there is not enough contiguous space in memory, even if the reported amount of memory is enough. Having many smaller arrays are therefore less likely to fail.

/J

[Michael ten Den]

I have now 4 while loops running:

- Continue receive and transmit CAN messages in/to queue

- Consumer loop, initialize, read the CAN message from queue and put them in the right array, a display case for my front panel update, and a few several items

- producer loop, time out 1 ms, sends 'read' in to consumer loop, update display, send CAN message and stop event

- Datalogging loop, save every 10 secs all arrays in same file and create every 5 min a new map for the data

I thought that put the messages first in arrays will be easier for me to plot in on my front panel and save it.

Apparently it isn't the right way.

@ superS-5 I will build a disable script for the ID's I don't want to measurement at that time, but can't delete them because they will be necessary for other test. Thanks

@ ShaunR I assume you mean direct logging to a file instead of array? But how to plot it then?

Some additional information:

- The program has to run several hours

- The x-as will be max. 5 minutes, what continous with the time.

- Sample rate of 3 pot. meters, 1 kHz.

It's not all clear to me right now what the best reliable methode is.

____________

Michael ten Den

[jgcode]

But how to plot it then?

One way is to keep a circular buffer of the latest data in memory for display.

[Michael ten Den]

One way is to keep a circular buffer of the latest data in memory for display.

Ok, but do I have to buffer lastest data of every ID in a different buffer, or is there a better solution?

[ShaunR]

There are a number of ways you can go about it. It depends on how you want to organise the data and what you want to do with it on screen.

If you are only going to show the last five minutes, then you can use a history chart/1 A 1Khz sample rate means about 300,000 samples (plot points) per channel which is a lot,so you will probably t have to decimate (plot every n points). However. It's worth bearing in mind, that you probably won't have 300,000 pixels in your graph anyway, so plotting them all is only really useful if you are going to allow them to zoom in. There are other ways (JGCodes suggestion is one, queues and a database are another). But that's the easiest and the hassle free way with minimum coding.

Ideally you want to stream the data into a nice text file - just as you would see it in an array (I use comma or tab delimited when I can). Then you can load it up in a text editor or spreadsheet and it will make sense and you won't need to write code to interpret it just to read it. You can always add that later if it's taking too long to load in the editor. If the messages are coming in 1,2,3,4,1234 etc then that's not a problem. However, it becomes a little more difficult if they are coming in ad-hock and you will need to find a way of re-organising it before saving so your text file table lines up. Hope you have a big hard-disk

Oh. And one final thought. Kill the Windows Indexing service (if you are using windows that is). You don't want to get 4 hours in and suddenly get a "file in use error"

Edited December 14, 2010 by ShaunR

[Michael ten Den]

Thank you for the respons.

I have a 1 kHz sample rate, but maybe its smarter to save that and only show 100 Hz or something like that. Because if their is a peak, it will also be showed with lower rate.

Or I maybe smarter to build a hysterese so if it get higher than a value it will give me the 1 kHz in an other chart

I'm now saving my files as .DAT file. I build a .DAT reader so I can open my data after measurement.

But saving to a text file will maybe be better. I just have to see it yet.

Unfortunaly I'm a windows user

____________

Michael ten Den

[asbo]

I'm now saving my files as .DAT file. I build a .DAT reader so I can open my data after measurement.

But saving to a text file will maybe be better. I just have to see it yet.

Check out the TDMS format Text files are more human-readable, but can grow to astronomical sizes quite easily.

[Michael ten Den]

Check out the TDMS format Text files are more human-readable, but can grow to astronomical sizes quite easily.

Thanks, going to check it out

[Michael ten Den]

Check out the TDMS format Text files are more human-readable, but can grow to astronomical sizes quite easily.

TDMS format is too big. Then I need lots of disk space.

[SuperS_5]

You don't want to get 4 hours in and suddenly get a "file in use error"

I have been bitten by that before.

[jgcode]

TDMS format is too big. Then I need lots of disk space.

TDMS in a binary format, that I have found weirdly enough to be quite large compared to other formats and thats not taking into account the index file it creates.

If disk space is a major concern then why not just use binary using the LabVIEW primitives?