Saving LARGE data files to file and SQL

[Draqo85]

Hi,

I have two questions about saving large files.

For example I want to save 2d array - 8 columns, 100000 rows to file. Numbers like : 0,000875 -1,086301 -0,0254 ...

I know it's sipmle - normal saving to file, but it takes lot of time and this saved file is big.

Is there any better and faster way to do it?? Most important is saving speed.

Second question is similar, also about saving large data.

Let's have the same data and we want to save it in Mysql.

What should be DATATYPE in our database sql if we want to save in it so big data in one row. How should it look like??

I thought about saving to binary files and stuff like that, but I don't know how to do it properly in labview with numbers like :0,000875 -1,086301 -0,0254 ...

Do you have any ideas how to do it??

[alukindo]

Hi Draqo85:

Regarding saving to database, perhaps you should also consider referencing the path to the data file from the database. In this way when the data is required, the database will pass the data file's 'saved path' rather than downloading the data from a row in the database.

Else, from an SQL Server perspective, ( I know you use MySQL, but it is likely that there is an equivalent approach from MySQL) there are two ways to upload large data files:

-- You can save the data in a single cell as a BLOB here is a link about BLOB data types.

-- The other way is to save your data in a results table using Bulk Insert or Bulk Copy procedure. You can then use a 'REFERENCE KEY' from a single row of your target table to link to the multiple rows in the results table. From LabVIEW, Bulk Insert procedure can upload 300,000 data rows per second.

Anthony L.

[MikaelH]

When saving to file I normally save it as binary, like this. If I stick to SGL it takes less then 500ms on my computer, for DBL it takes a minute.

When I need to save this is a database, I first ask my self do I need to search through the data or use any powerful calculations functions the database can offer.

If you need these functions, you have to save it as normal float values.

But if you only need to store it somewhere I would suggest only save the path in the database and then place the binary file on a file server.

Otherwise you can use a binary column type.

Cheers,

Mikael

[Maca]

For the column type in the SQL DB, I would recemond a "Float" data type. See: http://www.postgresql.org/docs/8.3/static/...#DATATYPE-FLOAT

From your original post it sounds like your currently saving the data as an array of strings, is this correct?

If it is that's why its taking so long, and you can probably just get away with converting the numbers to a double before saving them and that will fix your speed an size problems.

MikaelH has posted an excellent example that should get you out of trouble, but it might be worthwhile considering saving the data to TDMS files.

Also how do you plan to read/analyse the data? As this will have a bearing on how you should save it to disk.