hooovahh

This year I went to NI Connect (the new NI Week) for the first time in 7 years. I thought I would be the exception and see all those that have gone year after year. But it turns out that many of us this was our first year back, or some last year was their first year back since COVID. In general I think this is a good sign, that things are moving in the right direction. NI has some new leadership that has a LabVIEW focus, and at least at the moment appear to want to push adoption. Reversing the subscription only is a welcome change, but for many it hurt the inertia of business. Once a ship starts moving in the wrong direction it takes a while to come back. Or put another way, respect is lost in buckets and gained in drops. Plenty of businesses have likely moved away from LabVIEW and NI because of poor decisions, that in my opinion, were so NI would look more valuable for an Emerson sale. I'm in the Detroit area, and plan to retire doing LabVIEW. At the moment I think I can do that. Not long ago I didn't think that would be the case. We were just blindly paying the SSP each year. The subscription only model, made management here reevaluated things. We took a few years off. Then perpetual licenses came back again so we renewed. I think we will likely get a new perpetual license every 4 years or so. This will hurt NI since this means less users on the newest release finding issues. Building back trust will take time here, and this will likely play out in a similar way around the world for other companies.

There are several SQLite libraries on VIPM.io. I personally use the one by JDP Science, but a few others also look promising. Changing over will obviously take some time finding and replacing one API for another, especially if there are differences in design choices in how they each implement similar functionality.

Can you describe the issue and steps you took?

VPN to the rescue. Yeah I see it. If you do a resource replace function outside of the scripting functions, you can probably get around the runtime environment requirement. But again I don't know what problem that solves, you still need to use LabVIEW in the development environment to use the controls. To me this is an extension of the IDE. Having something like QuickDrop be a separate application for instance doesn't solve anything, but adds unnecessary complications.

Several of the functions that this uses to generate the control are LabVIEW properties and methods that are only available in the full development environment. Control creation isn't the type of function NI had planned for LabVIEW to do at run time when it was created. What problem are you trying to solve with having this be a built application?

"I don't want your damn lemons!"

Yes thank you I see that now. The link to a login page, private hosting, and the listing for Commercial Pricing at the top of the page made me think it was more restrictive.

I'm unsure of your tone and intent with text, but I'll do my best to explain the vocabulary. The Free (as in beer) is like when I go to a friends house and they offer me a free beer. I can have this beer, and I am entitled to enjoying it without monetary cost. Of course someone else paid for it, but I get to enjoy it, they shared it with me. I do no have the rights or the freedom to reproduce this beer, market it, and sell it. My free-ness is limited to the license you provide. I guess an equivalent analogy here could be if my friend gives me a beer, but expects me to help him move. It is an agreement we both have. If I am "free to do what I want" that is a different kind of freedom. A freedom that is more like free speech, a right that we have. It isn't as limited as someone giving me a beer, it is closer to someone giving me a beer recipe. Typically this is associated with free open source software. Open source software is free (like beer) someone is giving it to me without having to pay. But the freedom I get goes farther than that. This probably is a problem with the english language, and how words can have many meanings and without context "free" can mean different things. The Beer/Speech is just there to help clear up what kind of free we are talking about.

Yup. Last time I checked.

Do you guys also disagree philosophically about sharing your work online? That link doesn't appear to be to a download but a login page, it isn't clear to me if the code is free (as in beer). I assume it isn't free (as in speech). In this situation I think Rolf's implementation meets my needs. I initially thought I needed more low level control, but I really don't.

All very good information thanks for the discussion. I was mostly just interested in reentrant for the ZLIB Deflate VI specifically. For my test I took 65k random CAN frames I had, organized them by IDs, then made something like roughly 60 calls to the deflate, getting compressed blocks for each ID and time. Just to highlight the improvement I turned the compression level up to 9, and it took about 400ms. In the real world the default compression level is just fine. I then set the loop to enable parallelism with 16 instances, which was the default for my CPU. That time to process the same frames at the same compression level took 90ms. In the real world I will likely be trying to handle something like a million frames, in chunks, maybe using pipelining. So the improvement of 4.5 times faster for the same result is a nice benefit if all I need to do is enable reentrant on a single VI. Just something for you to consider, and seems like a fairly low risk on this VI, since we aren't talking to outside resources, just a stream. I'd feel similarly for the Inflate VI. I certainly would not try to access the same stream on two different functions at the same time. Thanks for the info on time. I feel fairly certain that in my application, a double for time is the first easy step to improve log file size and compression.

I don't see any function name in the DLL mention Deflate/Inflate, just lvzlib_compress and lvzlib_compress2 for the newer releases. Still I don't know if you need to expose these extra functions just for me. I did some more testing and using the OpenG Deflate, and having two single blocks for each ID (Timestamp and payload) still results in a measurable level of improvement on it's own for my CAN log testing. 37MB uncompressed, 5.3MB with Vector compression, and 4.7MB for this test. I don't think that going to multiple blocks within Deflate will have that much of a savings, since the trees, and pairs need to be recreated anyway. What did have a measurable improvement is calling the OpenG Deflate function in parallel. Is that compress call thread safe? Can the VI just be set to reentrant? If so I do highly suggest making that change to the VI. I saw you are supporting back to LabVIEW 8.6 and I'm unsure what options it had. I suspect it does not have Separate Compile code back then. Edit: Oh if I reduce the timestamp constant down to a floating double, the size goes down to 2.5MB. I may need to look into the difference in precision and what is lost with that reduction.

Thanks but for the OpenG lvzlib I only see lvzlib_compress used for the Deflate function. Rolf I might be interested in these functions being exposed if that isn't too much to ask. Edit: I need to test more. My space improvements with lower level control might have been a bug. Need to unit test.

So then is this what an Idea Exchange should be? Ask NI to expose the Inflate/Deflate zlib functions they already have? I don't mind making it I just want to know what I'm asking for. Also I continued down my CAN logging experiment with some promising results. I took log I had with 500k frames in it with a mix of HS and FD frames. This raw data was roughly 37MB. I created a Vector compatible BLF file, which compresses the stream of frames written in the order they come in and it was 5.3MB. Then I made a new file, that has one block for header information containing, start and end frames, formats, and frame IDs, then two more blocks for each frame ID. One for timestamp data, and another for the payload data. This orders the data so we should have more repeated patterns not broken up by other timestamp, or frame data. This file would be roughly 1.7MB containing the same information. That's a pretty significant savings. Processing time was hard to calculate. Going to the BLF using OpenG Deflate was about 2 seconds. The BLF conversion with my zlib takes...considerably longer. Like 36 minutes. LabVIEW's multithreaded-ness can only save me from so much before limitations can't be overcome. I'm unsure what improvements can be made but I'm not that optimistic. There are some inefficiencies for sure, but I really just can't come close to the OpenG Deflate. Timing my CAN optimized blocks is hard too since I have to spend time organizing it, which is a thing I could do in real time as frame came in if this were in a real application. This does get me thinking though. The OpenG implementation doesn't have a lot of control for how it work at the block level. I wouldn't mind if there is more control over the ability to define what data goes into what block. At the moment I suspect the OpenG Deflate just has one block and everything is in it. Which to be fair I could still work with. Just each unique frame ID would get its own Deflate, with a single block in it, instead of the Deflate containing multiple blocks, for multiple frames. Is that level of control something zlib will expose? I also noticed limitations like it deciding to use the fixed or dynamic table on it's own. For testing I was hoping I could pick what to do.

😅 You might be waiting a while, I'm mostly interested in compression, not decompression. That being said in the post I made, there is a VI called Process Huffman Tree and Process Data - Inflate Test under the Sandbox folder. I found it on the NI forums at some point and thought it was neat but I wasn't ready to use it yet. It isn't complete obviously but does the walking through of bits of the tree, to bytes. EDIT: Here is the post on NI's forums I found it on.