drjdpowell

12 hours ago, Manudelavega said:

Uhm, If I reall y want/need to understand this I guess I need to dig into type descriptors and so on?

Type descriptors?  No.   There are VIs such as  “Get Type Information.vi”, which give you the info.

Type descriptors come with flattening data, and flattening/unflattening data is slow.  The various Variant functions are much faster.  

A cluster is a fixed-size array.  I often use this to convert clusters (of mixed types) into an Array of Variants.

What’s a “cracked” module?

Anyway, back to netCDF.   Looks like no-one has used it, and, though it looks like a nice data storage API, it doesn’t offer any improvement over the existing HDF5 labview implementations.   Unless one needs compatibility with existing netCDF tools, but I think that is only if you are in the Atmospheric science community.  The MDSplus library I mentioned above is even more niche; only used by those doing research in nuclear fusion.

7 minutes ago, ShaunR said:

Multi-tasking <> multithreading. You seem to be conflating many different aspects of application execution.

I'm only interested in LabVIEW applications executing correctly when one is calling the same library from two parallel loops.  If libraries fail that, then I do not care if they are "multithreaded" by some technical definition.

I would first try copying your 4Gb file to a file system that can handle larger files, then try opening it.

9 minutes ago, ShaunR said:

The HDF5 library uses recursive locks and keeps a separate error stack for each thread. 

The “multithreaded” version of HDF5, you mean.  The “separate error stack” is actually an example of something you need a LabVIEW-level lock to solve.  LabVIEW does multi-tasking in the UI thread, so if one queries the error information after a function throws an error, it is quite possible that LabVIEW has called some other function from a parallel loop which has overwritten the error stack.  Using Thread-Local Memory to allow “multithreading” is based on the assumption that the threads are not multitasking.  Instead, you need to have a lock held over the original (error-throwing) function call and the secondary call to read the error information.

49 minutes ago, rolfk said:

Another way is to use a semaphore. That can be done explicitedly by the caller (what drjdpowell describes) or in the library itself but the later has the potential to lockup if the library uses multiple global resources that are each protected by their own semaphore.  OpenSSL which Shaun probably refers to, requires the caller to install callback functions that provide the semaphore functionality and which OpenSSL then uses to protect access to its internal global variables.

Ahh, so the callback is the locking mechanism of your choice.  Having the library do the lock means a lock is only made when necessary, in contrast to my Semaphore in LabVIEW, where I always lock before any call, even where it might not be needed.

I don’t have LabVIEW 2016 and can’t open your code, but you might be trying to change a parameter directly.  C calls pass parameters by-value, not by-reference, so changing any of the parameters in the function will not affect their value in the calling code.  

2 hours ago, ShaunR said:

But thread safety isn't about one problem. It is an umbrella term for multiple undefined behaviours when using multiple threads, including race conditions, pointer overflows, memory dereferencing, counter overruns,  and lots of nasty stuff that will crash LabVIEW-probably randomly and when already deployed to the customer site.

All those issues are about parallel actions; they are not specific to OS “threads”.  Only thread-specific memory is something that would require LabVIEW to use the UI thread.  Otherwise one can serialize prevent parallel calls by any number of means.

Well. Serialization means writing binary as strings to me (like the flatten functions,XML etc). So I'm already floundering with your question. 

Serialization is the act of serializing, which is to make something into a serial form, which is a set of things one following another.  Serial is often contrasted with parallel.  Used in multiple contexts in computer equipment, not just in converting memory data structures into serial bytes.

An HDF5 reference I found discussing its issues with thread safety.  From it I gather serialization would probably be sufficient.

especially for recursive mutexes

I used a mutex count for that, releasing the semaphore when the count reached zero.   That allowed me to call locking functions from inside other locking functions.

8 hours ago, ShaunR said:

You cannot mitigate thread safety in LabVIEW.

Depends what the issue is.  The MDSplus library I mentioned uses a global variable.  It doesn’t matter if function calls are made in different threads; it just matters that critical code sections that use that variable be protected from parallel access, and thus I used a semaphore for serialization.  So I may have been using “thread safety” imprecisely to include race condition issues.  What’s the proper term for a library that requires serialization but not use from a single OS thread?

That means netCDF isn’t thread safe either.  In a previous wrapping of a non-thread-safe library, MDSplus, I used a semaphore to serialize access, rather than the UI thread.  

...but am definitely interested in other developer feedback.

Hi Martijn,

Perhaps we should start a new topic on your library; I’ve had a look at it and could make some comments/questions.  For example, why are you making your dll calls in the UI thread?  Is HDF5 not thread safe?  

— James

Interesting discussion.

My bias against a middle C layer is partly related to my lack of C coding skills.  I can use a third-party library with the confidence that I can fix bugs or add features to the LabVIEW part of it, and with some confidence that the underlying dll (such as HDF5) is widely used and thus probably well tested and near complete.  But issues with the middle layer leave me stuck.  

It also matters how complex the library being wrapped is.   My understanding is that netCDF is a simpler API than HDF5 (with the tradeoff that it cannot do everything that HDF5 can do), and I haven’t found the LabVIEW code needed to call it directly is that complicated.

Regarding the Unreserve() and Abort() callbacks, I would much rather be able to register a VI callback when a VI hierarchy goes idle.  It could then do any cleanup actions like closing things.  Perhaps I should suggest this to NI.  I can, and sometimes do, use such cleanup functions using the “watchdog” asynchronous actions in Messenger Library, but I can’t add Messenger Library as a dependency for a reusable wrapper library (also, asynchronous watchdogs are problematic compared to synchronous callbacks due the possibility of race conditions).  

On December 16, 2016 at 6:45 PM, Stobber said:

djpowell, could you add support for extended error codes to this API? I'm struggling with a SQLITE_CANTOPEN error on a cRIO that never used to appear, and I don't have enough context to know what the heck is wrong.

Also, could you modify SQLite.lvlib:Format Error.vi to yield the full call chain? Figuring out where an error came from without it is sometimes really hard. If you don't want to stringify the call chain all the time, maybe make it an option I can toggle on Open?

See the 1.6.5 version now in the CR.   I added the extended code in the Error Description, after the standard error description.   So SQLITE_CANTOPEN(nnn), with nnn the full code, which you can lookup at http://www.sqlite.org/rescode.html.  At some point, I’ll implement the Extended Error names, but I don’t have time now.

On 19/12/2016 at 8:11 PM, GregSands said:

 The HDF5 libraries are mostly wrapped directly, with a small interface dll just for handling some of the memory allocation issues.

Just a side comment, but I think it is better to avoid a middle-layer dll if at all possible.  I have yet to find a memory allocation issue that could not be solved with LabVIEW.exe:MoveBlock.

So, no one has used netCDF then?  I'm trying to see if there would be any interest in a netCDF LabVIEW API.   The API I've made is incomplete, as I can only justify implementing the minimum needed for the client who's paying for it.   Seems like it's better to just use HDF5 directly, unless one already uses netCDF (which I think is mainly just the Atmospheric Science community).  

17 hours ago, Stobber said:

djpowell, could you add support for extended error codes to this API? I'm struggling with a SQLITE_CANTOPEN error on a cRIO that never used to appear, and I don't have enough context to know what the heck is wrong.

Also, could you modify SQLite.lvlib:Format Error.vi to yield the full call chain? Figuring out where an error came from without it is sometimes really hard. If you don't want to stringify the call chain all the time, maybe make it an option I can toggle on Open?

I will do both those things.  

17 minutes ago, shoneill said:

IIt seems the WAL file does not operate in synchronous mode at all.  I wonder if that affects fobustness at all?

You mean with PRAGMA synchronous=OFF?   With "NORMAL" that you are using I think there is a chance that the WAL will be corrupted on power loss, but that just means the loss of the latest few transactions.   On the default "FULL" setting each transaction to the WAL is synced.  

I would hazard a guess that it's something to do with the Write-Ahead-Log (WAL).  Perhaps an UPDATE is done on the main file normally, but if the main file is locked by a reader the UPDATE is recorded in the WAL file (which might be faster).   Then the completion of the UPDATE happens at the next WAL checkpoint.  I don't know, but regardless, WAL checkpointing makes benchmarking trickier; you might want to manually checkpoint by calling PRAGMA wal_checkpoint(FULL) as part of your performance test (just after the COMMIT) to make sure you are measuring the full cost of your UPDATEs.

I'm developing, for a client, a wrapper for netCDF4, a format that used for large atmospheric-science data sets that is implemented on top of HDF5.  Seems quite nice so far, and I was wondering if anyone had any experience with it?  Or HDF5 in general (which I have never used)?  It's kind of TDMS-like, but with n-dimension arrays.

8 hours ago, maxout said:

Another question is it possible to generate .db file with specific columns and tables using the wrapper you provided?

Tables are created using standard SQL statements.