drjdpowell

Anyway, back to netCDF. Looks like no-one has used it, and, though it looks like a nice data storage API, it doesn’t offer any improvement over the existing HDF5 labview implementations. Unless one needs compatibility with existing netCDF tools, but I think that is only if you are in the Atmospheric science community. The MDSplus library I mentioned above is even more niche; only used by those doing research in nuclear fusion.

I'm only interested in LabVIEW applications executing correctly when one is calling the same library from two parallel loops. If libraries fail that, then I do not care if they are "multithreaded" by some technical definition.

I would first try copying your 4Gb file to a file system that can handle larger files, then try opening it.

The “multithreaded” version of HDF5, you mean. The “separate error stack” is actually an example of something you need a LabVIEW-level lock to solve. LabVIEW does multi-tasking in the UI thread, so if one queries the error information after a function throws an error, it is quite possible that LabVIEW has called some other function from a parallel loop which has overwritten the error stack. Using Thread-Local Memory to allow “multithreading” is based on the assumption that the threads are not multitasking. Instead, you need to have a lock held over the original (error-throwing) function call and the secondary call to read the error information.

Ahh, so the callback is the locking mechanism of your choice. Having the library do the lock means a lock is only made when necessary, in contrast to my Semaphore in LabVIEW, where I always lock before any call, even where it might not be needed.

I don’t have LabVIEW 2016 and can’t open your code, but you might be trying to change a parameter directly. C calls pass parameters by-value, not by-reference, so changing any of the parameters in the function will not affect their value in the calling code.

All those issues are about parallel actions; they are not specific to OS “threads”. Only thread-specific memory is something that would require LabVIEW to use the UI thread. Otherwise one can serialize prevent parallel calls by any number of means.

Serialization is the act of serializing, which is to make something into a serial form, which is a set of things one following another. Serial is often contrasted with parallel. Used in multiple contexts in computer equipment, not just in converting memory data structures into serial bytes.

An HDF5 reference I found discussing its issues with thread safety. From it I gather serialization would probably be sufficient.

I used a mutex count for that, releasing the semaphore when the count reached zero. That allowed me to call locking functions from inside other locking functions.

Depends what the issue is. The MDSplus library I mentioned uses a global variable. It doesn’t matter if function calls are made in different threads; it just matters that critical code sections that use that variable be protected from parallel access, and thus I used a semaphore for serialization. So I may have been using “thread safety” imprecisely to include race condition issues. What’s the proper term for a library that requires serialization but not use from a single OS thread?

That means netCDF isn’t thread safe either. In a previous wrapping of a non-thread-safe library, MDSplus, I used a semaphore to serialize access, rather than the UI thread.

Hi Martijn, Perhaps we should start a new topic on your library; I’ve had a look at it and could make some comments/questions. For example, why are you making your dll calls in the UI thread? Is HDF5 not thread safe? — James

Interesting discussion. My bias against a middle C layer is partly related to my lack of C coding skills. I can use a third-party library with the confidence that I can fix bugs or add features to the LabVIEW part of it, and with some confidence that the underlying dll (such as HDF5) is widely used and thus probably well tested and near complete. But issues with the middle layer leave me stuck. It also matters how complex the library being wrapped is. My understanding is that netCDF is a simpler API than HDF5 (with the tradeoff that it cannot do everything that HDF5 can do), and I haven’t found the LabVIEW code needed to call it directly is that complicated. Regarding the Unreserve() and Abort() callbacks, I would much rather be able to register a VI callback when a VI hierarchy goes idle. It could then do any cleanup actions like closing things. Perhaps I should suggest this to NI. I can, and sometimes do, use such cleanup functions using the “watchdog” asynchronous actions in Messenger Library, but I can’t add Messenger Library as a dependency for a reusable wrapper library (also, asynchronous watchdogs are problematic compared to synchronous callbacks due the possibility of race conditions).

See the 1.6.5 version now in the CR. I added the extended code in the Error Description, after the standard error description. So SQLITE_CANTOPEN(nnn), with nnn the full code, which you can lookup at http://www.sqlite.org/rescode.html. At some point, I’ll implement the Extended Error names, but I don’t have time now.

Just a side comment, but I think it is better to avoid a middle-layer dll if at all possible. I have yet to find a memory allocation issue that could not be solved with LabVIEW.exe:MoveBlock.

So, no one has used netCDF then? I'm trying to see if there would be any interest in a netCDF LabVIEW API. The API I've made is incomplete, as I can only justify implementing the minimum needed for the client who's paying for it. Seems like it's better to just use HDF5 directly, unless one already uses netCDF (which I think is mainly just the Atmospheric Science community).

I will do both those things.

You mean with PRAGMA synchronous=OFF? With "NORMAL" that you are using I think there is a chance that the WAL will be corrupted on power loss, but that just means the loss of the latest few transactions. On the default "FULL" setting each transaction to the WAL is synced.

I would hazard a guess that it's something to do with the Write-Ahead-Log (WAL). Perhaps an UPDATE is done on the main file normally, but if the main file is locked by a reader the UPDATE is recorded in the WAL file (which might be faster). Then the completion of the UPDATE happens at the next WAL checkpoint. I don't know, but regardless, WAL checkpointing makes benchmarking trickier; you might want to manually checkpoint by calling PRAGMA wal_checkpoint(FULL) as part of your performance test (just after the COMMIT) to make sure you are measuring the full cost of your UPDATEs.

I'm developing, for a client, a wrapper for netCDF4, a format that used for large atmospheric-science data sets that is implemented on top of HDF5. Seems quite nice so far, and I was wondering if anyone had any experience with it? Or HDF5 in general (which I have never used)? It's kind of TDMS-like, but with n-dimension arrays.

Tables are created using standard SQL statements.

Thanks from me also.

Sorry, Shaun, I didn’t really follow (though I happily nod along with you bringing up “execution state”). I still don’t know what this has to do with OOP (or “services” or “non-reentrant subVIs” for that matter).

It doesn’t fully support it as message flow between loops. A message to a loop, that prompts further messages from that loop, is like a subVI call in regular LabVIEW programming, in that the action waits for the “input”to be available before it happens. However, this only works for ONE input; a subVI can wait for multiple inputs before it executes. My primary use of Futures is to extend this such that a loop can take an action only when multiple input messages are received. It also helps with ordering arrays. If I call three subVIs in parallel and combine their outputs in an array, that array has a defined order. But if i send a message to three loops, the reply messages can come in arbitrary order, unless I use Futures to specify the order. There are no OOP concepts involved.