drjdpowell

Indexing tunnels (conditional or not) follow a preallocation strategy of filling a larger-than-initially-needed array (later cutting the unneeded elements), while the “Build Array” primitive allocates a new array of exactly the right size on each iteration. So there is a lot fewer calls to the memory manager with an indexing tunnel.

Type descriptors? No. There are VIs such as “Get Type Information.vi”, which give you the info. Type descriptors come with flattening data, and flattening/unflattening data is slow. The various Variant functions are much faster.

A cluster is a fixed-size array. I often use this to convert clusters (of mixed types) into an Array of Variants.

What’s a “cracked” module?

Anyway, back to netCDF. Looks like no-one has used it, and, though it looks like a nice data storage API, it doesn’t offer any improvement over the existing HDF5 labview implementations. Unless one needs compatibility with existing netCDF tools, but I think that is only if you are in the Atmospheric science community. The MDSplus library I mentioned above is even more niche; only used by those doing research in nuclear fusion.

I'm only interested in LabVIEW applications executing correctly when one is calling the same library from two parallel loops. If libraries fail that, then I do not care if they are "multithreaded" by some technical definition.

I would first try copying your 4Gb file to a file system that can handle larger files, then try opening it.

The “multithreaded” version of HDF5, you mean. The “separate error stack” is actually an example of something you need a LabVIEW-level lock to solve. LabVIEW does multi-tasking in the UI thread, so if one queries the error information after a function throws an error, it is quite possible that LabVIEW has called some other function from a parallel loop which has overwritten the error stack. Using Thread-Local Memory to allow “multithreading” is based on the assumption that the threads are not multitasking. Instead, you need to have a lock held over the original (error-throwing) function call and the secondary call to read the error information.

Ahh, so the callback is the locking mechanism of your choice. Having the library do the lock means a lock is only made when necessary, in contrast to my Semaphore in LabVIEW, where I always lock before any call, even where it might not be needed.

I don’t have LabVIEW 2016 and can’t open your code, but you might be trying to change a parameter directly. C calls pass parameters by-value, not by-reference, so changing any of the parameters in the function will not affect their value in the calling code.

All those issues are about parallel actions; they are not specific to OS “threads”. Only thread-specific memory is something that would require LabVIEW to use the UI thread. Otherwise one can serialize prevent parallel calls by any number of means.

Serialization is the act of serializing, which is to make something into a serial form, which is a set of things one following another. Serial is often contrasted with parallel. Used in multiple contexts in computer equipment, not just in converting memory data structures into serial bytes.

An HDF5 reference I found discussing its issues with thread safety. From it I gather serialization would probably be sufficient.

I used a mutex count for that, releasing the semaphore when the count reached zero. That allowed me to call locking functions from inside other locking functions.

Depends what the issue is. The MDSplus library I mentioned uses a global variable. It doesn’t matter if function calls are made in different threads; it just matters that critical code sections that use that variable be protected from parallel access, and thus I used a semaphore for serialization. So I may have been using “thread safety” imprecisely to include race condition issues. What’s the proper term for a library that requires serialization but not use from a single OS thread?

That means netCDF isn’t thread safe either. In a previous wrapping of a non-thread-safe library, MDSplus, I used a semaphore to serialize access, rather than the UI thread.

Hi Martijn, Perhaps we should start a new topic on your library; I’ve had a look at it and could make some comments/questions. For example, why are you making your dll calls in the UI thread? Is HDF5 not thread safe? — James

Interesting discussion. My bias against a middle C layer is partly related to my lack of C coding skills. I can use a third-party library with the confidence that I can fix bugs or add features to the LabVIEW part of it, and with some confidence that the underlying dll (such as HDF5) is widely used and thus probably well tested and near complete. But issues with the middle layer leave me stuck. It also matters how complex the library being wrapped is. My understanding is that netCDF is a simpler API than HDF5 (with the tradeoff that it cannot do everything that HDF5 can do), and I haven’t found the LabVIEW code needed to call it directly is that complicated. Regarding the Unreserve() and Abort() callbacks, I would much rather be able to register a VI callback when a VI hierarchy goes idle. It could then do any cleanup actions like closing things. Perhaps I should suggest this to NI. I can, and sometimes do, use such cleanup functions using the “watchdog” asynchronous actions in Messenger Library, but I can’t add Messenger Library as a dependency for a reusable wrapper library (also, asynchronous watchdogs are problematic compared to synchronous callbacks due the possibility of race conditions).

See the 1.6.5 version now in the CR. I added the extended code in the Error Description, after the standard error description. So SQLITE_CANTOPEN(nnn), with nnn the full code, which you can lookup at http://www.sqlite.org/rescode.html. At some point, I’ll implement the Extended Error names, but I don’t have time now.

Just a side comment, but I think it is better to avoid a middle-layer dll if at all possible. I have yet to find a memory allocation issue that could not be solved with LabVIEW.exe:MoveBlock.

So, no one has used netCDF then? I'm trying to see if there would be any interest in a netCDF LabVIEW API. The API I've made is incomplete, as I can only justify implementing the minimum needed for the client who's paying for it. Seems like it's better to just use HDF5 directly, unless one already uses netCDF (which I think is mainly just the Atmospheric Science community).

I will do both those things.

You mean with PRAGMA synchronous=OFF? With "NORMAL" that you are using I think there is a chance that the WAL will be corrupted on power loss, but that just means the loss of the latest few transactions. On the default "FULL" setting each transaction to the WAL is synced.

I would hazard a guess that it's something to do with the Write-Ahead-Log (WAL). Perhaps an UPDATE is done on the main file normally, but if the main file is locked by a reader the UPDATE is recorded in the WAL file (which might be faster). Then the completion of the UPDATE happens at the next WAL checkpoint. I don't know, but regardless, WAL checkpointing makes benchmarking trickier; you might want to manually checkpoint by calling PRAGMA wal_checkpoint(FULL) as part of your performance test (just after the COMMIT) to make sure you are measuring the full cost of your UPDATEs.

I'm developing, for a client, a wrapper for netCDF4, a format that used for large atmospheric-science data sets that is implemented on top of HDF5. Seems quite nice so far, and I was wondering if anyone had any experience with it? Or HDF5 in general (which I have never used)? It's kind of TDMS-like, but with n-dimension arrays.