how to save large datas

[simonch]

I want to save the datas more than 20M Bytes. But when run the program,there is a dialog box showing as Not enough memory to complete this operation. The memory size of my pc is 2G. So,how to save large datas? Thanks.The version of lv is 8.6

save large datas.vi

[jgcode]

I want to save the datas more than 20M Bytes. But when run the program,there is a dialog box showing as Not enough memory to complete this operation. The memory size of my pc is 2G. So,how to save large datas? Thanks.The version of lv is 8.6

Hi Simon

I did the maths and you are definitely trying to save more than 20MB... 400MB in fact

My question to you is - do you need the entire array in memory? If not stream the data to the disk. Additionally may I suggest a binary file and not ASCII for such a large file? You will save a lot of space (and it will be faster as it does not have to convert binary to ASCII etc...).

E.g.

Binary DBL 3.14159265141592 = 8 bytes

ASCII text 3.14159265141592 = 16 bytes

There maybe some LabVIEW guru's who can suggest a more optimized approach.

The code worked on my PC - but it took a while!

You may wish to span it when it reaches a certain size as well

[Ton Plomp]

I want to save the datas more than 20M Bytes. But when run the program,there is a dialog box showing as Not enough memory to complete this operation. The memory size of my pc is 2G. So,how to save large datas? Thanks.The version of lv is 8.6

Any reason to have all the data in memory?

I think the memory crash arises when converting to string, just dump the data to disc as binary (like jgcode showed).

The code worked on my PC - but it took a while!

Probably because the 0 ms timeout.

<br><br>Yes, 450 seconds with wait, 310 without wait.<br><br>I would advise to use TDMS files, using text-based files is asking for problems.

What if you data needs to be openend on a PC with a different decimal sign, what if you start to change the number of channels.

Ton

[jgcode]

I think the memory crash arises when converting to string

Yes the code originally failed on the string conversion.

Having the array in memory would slow things down I would imagine.

Probably because the 0 ms timeout. I would advise to use TDMS files

Well I ran it without the 0 wait in with respect to that comment.

It always worth adding a zero wait to free up the CPU with While Loops - nothing worse than an unthrottled loop.

As it was quite resource intensive I thought it would be a good idea for the For Loop in this case.

Great to see someone else recommending the TDMS format.

The new TDMS 2.0 format in 2009 has some great improvements (support for scaling properties, streaming directly from DAQmx, faster - bypassing the TDMS and Windows buffer straight to HDD etc..)

We use TDMS (or rather gTDMS) for embedded systems but I believe there is now VxWorks support for TDMS 2.0.

[simonch]

thank you, igcode, Ton. i made a mistake about the size of data.it shoud be 20M points rather than 20MB. i do not need the entire array in memory,i jist want to save the data to the disk.

igcode give a very good solution, thank you very much.Now, i use the NI pxi-5122 to acquire datas and save them to disk.there are two channels.the sample rate is 20MHz,and acquire 20M points,even more. in this case,how to save the datas to disk?

daq and save.vi

[simonch]

I found some examples with respect to the comment,but i still don't understand how to implement my use.

Some people said using Queue Operations to implement it.

acquire_lrge_binary.zip

Managing Large Data Sets in LabVIEW.zip

[jgcode]

I found some examples with respect to the comment,but i still don't understand how to implement my use.

Some people said using Queue Operations to implement it.

I think you are still going to have trouble converting the data to a string like that unless you do it using chunking or something similar.

You should be able to use the same example I have posted and stream the data to disk.

Just wire in the SCOPE data to the for loop.

[simonch]

I think you are still going to have trouble converting the data to a string like that unless you do it using chunking or something similar.

You should be able to use the same example I have posted and stream the data to disk.

Just wire in the SCOPE data to the for loop.

Thank you. I think fig.1 show what you mean. I'm not sure if the fig.2 is same as the fig.1. If they are same,then in fig.2, when N is 50M,still has the promble. So, i doubt fig.1 has the same promble, because the 2D array will be in the memory.Later i will try to run the program on my NI instruments.

In fig.3, N is 100M, there is no problem.

I'm a beginner, and i think my question is very easy for some gurus. Thanks for your patience to answer my questions.

Thank you, all. Thank you, igcode.

fig.1

fig.2

fig.3

[MikaelH]

Also rember to only use Single Precision SGL, that will half your data size, and is probably good enough.

So if your happy to get: 3.1415927 instead of 3.1415926 SGL should work fine

Cheers,

Mikael