Choice of logging format/style [TDMS, SQLite, some combination?]

[ShaunR]

1 minute ago, drjdpowell said:

Sorry, what “enumeration”?    

 

On 10/19/2016 at 11:08 AM, ShaunR said:

How about enumerating these alleged design cases for the OP and comparing the practicalities?. I'll start you off........

• TDMS is designed for high speed disk writing and can write to disk at more than 400MB/s with the right disks. SQlite can only manage about 0.5-1MB/s.

 

[drjdpowell]

1 hour ago, ShaunR said:

SQlite is single writer, multiple readers as it does table level locking. If drjdpowell had followed through with the enumeration it would (or should) have been in there. The high level API in the SQLite API for LAbVIEW insulates you from the Error 5 hell and "busy" handling (when the DB is locked) that you encounter when trying simultaneous parallel writes with low level APIs. So simultaneous parallel writes is not appropriate usage.....sort of .

Oh, that enumeration.  Not sure that would be on the list as only one thing can be written to a file at one time, even by TDMS.  SQLite is by default ACID compliant (now that would be on the list), but one can turn that off to get asynchronous disk writes, just like TDMS.  And as long as you writes are faster than your busy-handler timeout (I set that at 5 seconds, adjustable) then there are no Busy(5) errors.  The issue is just write speed, where TDMS wins.

[eberaud]

20 hours ago, Christian Butcher said:

Is this (1%) an approximate order of magnitude? Or just an example?

It's an example, but there is a an actual real number I can give you when I'll have taken the time to dig into my Excel sheets...

20 hours ago, Christian Butcher said:

What I'm meaning to ask is, you write an XY-array style, rather than using the waveform data type or an array of waveforms?

Exactly. I don't use Waveforms. Each sample has 3 columns for the time stamp and then X columns for the X channels. So X+3 columns total.