OpenG Zip 4.2?

[Mads]

I noticed on sourceforge that there is a version 4.2 of OpenG Zip. Will it be released as a package anytime soon?

[Rolf Kalbermatter]

On 8/20/2019 at 12:54 PM, Mads said:

I noticed on sourceforge that there is a version 4.2 of OpenG Zip. Will it be released as a package anytime soon?

I did work on that mainly end of last year but found some time to resume testing recently. The code and VIs are more or less ready but I do need to do a bit more testing on Linux, Mac and the different real-time targets. Especially Mac and the real-time  targets proof to be quite a hassle. Mac because I don't work often on it nowadays and the real-time targets because debugging shared libraries on them is always quite some hassle and each flavor is again different.

I could however use some extra eyes for testing and I don't mean the shared library part itself but simply the general operation of it. I might be able to create a preliminary OGP package for installation through VIPM within a week or so if you want to test it. Let me know which platforms you would want to test it on and how and I will check what I can do.

[Mads]

Great. I would be glad to test it. I mainly work on ARM-based Linux RT targets myself, and the occational old VxWorks cFP-target.

[Rolf Kalbermatter]

8 minutes ago, Mads said:

Great. I would be glad to test it. I mainly work on ARM-based Linux RT targets myself, and the occational old VxWorks cFP-target.

I probably can't test the VxWorks targets for now but can create at least the RT image to be installed for them. No guarantee that it can even load though without having tested it before myself.

[GregSands]

It appears that the shared libraries are fully threadsafe, given the calls are all set to run in any thread, and I don't think the zlib library is multithreaded.  Would there be any issues with setting the VIs to "Shared clone reentrant" to allow multiple simultaneous calls?

 

[Rolf Kalbermatter]

13 hours ago, GregSands said:

It appears that the shared libraries are fully threadsafe, given the calls are all set to run in any thread, and I don't think the zlib library is multithreaded.  Would there be any issues with setting the VIs to "Shared clone reentrant" to allow multiple simultaneous calls?

It depends what you try to do. As long as you don't try to access a particular zip or unzip sessions from multiple places in parallel, the zlib zip library has always been safe for reentrant execution as it does not contain global state that spans across sessions. The underlaying zlib library (pure compression/decompression algorithms) itself is even less of a problem as it does not have the complex archive maintenance that is needed for a ZIP archive but only works on immediate memory streams.

If you try to open a zip (or unzip) session somewhere and then branch this to two different locations and try to write streams to them from both locations you are going to crash sooner or later. Each session stores state that is used across method invocations and even if I would protect the individual function calls with a per session mutex (or make the functions all execute in the UI thread) you would still potentially corrupt the zip archive stream or in the case of unzip operation retrieve a different stream than what you think you do in a particular location.

As long as you don't access a specific session (zip or unzip refnum) from multiple places in parallel you were always fine though and that will remain like this in the future. This pretty much is the same as trying to read or write to a specific file from multiple places (through he same refnum or a separately opened). You can do that but expecting the read and writes to work properly and have a proper data content in the file afterwards will be pretty much impossible. There is however no problem in writing (and/or reading) in parallel to two (or more) different files on disk.

So setting the VIs to shared clone should work (all the state is stored in the session behind the refnum) but I'm not going to do that for now.

Edited August 23, 2019 by Rolf Kalbermatter

[Rolf Kalbermatter]

Here is a new version 4.2.0b1 for the ZIP library. I didn't test it in every LabVIEW version on every platform. What I did test was MacOSx 32-bit and 64-bit LabVIEW 2014, Windows 32-bit and 64-bit LabVIEW 7.0, 7.1, 8.6, 2009, 2016, 2018, Linux 32-bit 7.1 and 8.6, and NI Linux x86 LabVIEW 2016. Other realtime targets I hadn't handy at the moment. Support for Linux 64-bit and NI Linux RT ARM as well as VxWorks and Pharlap is contained.

The realtime support will only get extracted when installing into LabVIEW for Windows 32-bit and through a seperate exe file that is invoked and will prompt for an administrative elevation of this installer. You then have to go into NI Max and into the Software part of your target and select to install additional components. In the list should be an OpenG ZIP Tools version 4.2.0 package visible. Select that to be installed on your target.

 

There are still following problems that I haven't implemented/fixed yet:

1) archives that contain file names with other encodings than your platform code page will go certainly wrong. This is probably not solvable without doing absolutely every file IO operation in the shared library too, since the LabVIEW file IO functions don't support any other encodings in the path.

2) If you try to zip up directories containing soft/hard inks then the current implementation will compress the actual target file/directory into the archive instead of a link and expanding zip archives that contain such links will expand just a small text file continaing the link destination. This is something I'm looking into to solve in the next release by allowing optionally adding a special link entry into the archive and create such a link on the filesystem when extracting. This is mostly of concern on Linux and MacOSX. While Windows also allows for such links nowadays it is still quite an esoteric feature and user accessible support for it is minimal (you have to use the command line or install additional third party tools to create/modify such links).

 

Hope to hear from other platforms and versions and how it goes there. Without some feedback I'm not going to create a release. 

oglib_lvzip-4.2.0b1-1.ogp

[Mads]

Downloaded it and so far I've tested it on LabVIEW 2019 and Linux RT ARM (cRIO-9063) with success (compression/decompression of files and folders and deflate/enflate on strings). I'll try a different target type later today.

Only trouble so far is with the package format - my VIPM does not like opening ogp files from Windows (get an access error), but I can open it and install it manually from VIPM...That might be a local issue though.

[ShaunR]

4 hours ago, Rolf Kalbermatter said:

archives that contain file names with other encodings than your platform code page will go certainly wrong. This is probably not solvable without doing absolutely every file IO operation in the shared library too, since the LabVIEW file IO functions don't support any other encodings in the path.

For windows, you only need to replace the file open with "CreatefileW". That also gives you the long path names. LabVIEW read/write/close are fine after that.

I got a popup warning after installation on LabVIEW 2009. Something about a dialogue error cluster being replaced. It was only a warning and made no difference to the operation, though. I'm guessing it is due to back-saving rather than being built in 2009.

Edited September 4, 2019 by ShaunR

[Rolf Kalbermatter]

1 hour ago, ShaunR said:

I got a popup warning after installation on LabVIEW 2009. Something about a dialogue error cluster being replaced. It was only a warning and made no difference to the operation, though. I'm guessing it is due to back-saving rather than being built in 2009.

That is normal. The VIs are still in LabVIEW 7.0 format and the used File Dialog in that version did not have an error cluster. So when mutating it to 8.0 and higher LabVIEW will insert some compatiblity code on the fly.

1 hour ago, Mads said:

Downloaded it and so far I've tested it on LabVIEW 2019 and Linux RT ARM (cRIO-9063) with success (compression/decompression of files and folders and deflate/enflate on strings). I'll try a different target type later today.

Only trouble so far is with the package format - my VIPM does not like opening ogp files from Windows (get an access error), but I can open it and install it manually from VIPM...That might be a local issue though.

Hmmm, it works fine on my computer but it is VIPM 2014 and I'm not fully sure if I didn't do some registry fiddling in the past to fix something. I rather would like to avoid having to do a vi package which only can be installed with VIPM of the same version or newer.

Edited September 4, 2019 by Rolf Kalbermatter

[Rolf Kalbermatter]

1 hour ago, ShaunR said:

For windows, you only need to replace the file open with "CreatefileW". That also gives you the long path names. LabVIEW read/write/close are fine after that.

Right but it's a bit more complicated than that, because the same issue although less extreme exists on all other LabVIEW platforms. And there are various trouble to turn the underlaying file id into a LabVIEW refnum. On Linux LabVIEW uses the posix functions, and so does it on MacOSX 64-bit but not on 32-bit or at least not the versions I tested (it could have changed later but that would be potentially even worse). In addition while modern Linux platforms use UTF8 throughout, that was different before and still can be configured differently (although I'm hard pressured to imagine why someone would do something stupid like that).

And to make matters worse there is no standardized way really to determine what codepage was used when a ZIP archive was created. There is a newer option to use UTF8 which is indicated by a flag in the file entry for each file, but if that flag is not set the entry is in whatever OEM codepage (not ANSI) the computer was using at the time the archive was created.  And that could and for most problematic archives also will be a different codepage than on the computer on which you want to extract the files.

It's a complicated problem and part of it is basically unsolvable.

Edited September 4, 2019 by Rolf Kalbermatter

[Mads]

Tested the 4.2 beta with success on a Linux RT x64 target today (cRIO-9030), where I have never gotten it to work previously.  

Compressed and decompressed folders with multiple files and subfolders, and used the inflate/deflate functions. The files that were compressed were also transferred to a PC to verify them there, and vice-versa.

[Rolf Kalbermatter]

9 hours ago, Mads said:

Tested the 4.2 beta with success on a Linux RT x64 target today (cRIO-9030), where I have never gotten it to work previously.  

Compressed and decompressed folders with multiple files and subfolders, and used the inflate/deflate functions. The files that were compressed were also transferred to a PC to verify them there, and vice-versa.

I believe you! 🙂  During testing of this release I came across a problem that first dumbfounded me. On most systems it seemed to work fine but when executing it in LabVIEW 7.1 on Windows it consistently crashed. The problem turned out to be memory alignment related. One of the data structures passed to the shared library happened to be 43 bytes long. Inside the shared library was however some assignment operator where an internal temperary variable of that structure located on the stack was first filled in and then assigned to the passed in variable. C does allow to assign whole structure variables by value and the compiler then generates code to copy the whole variable. Except that Visual C did not bother to make it exactly 43 bytes but simply copied 48 bytes which resulted in random trash from the stack being copied after the end of the variable. On most platforms LabVIEW seemed to align the parameters it was passing to the Call Library Node such that this extra buffer overwrite didn't collide with any of the other parameters , but in LabVIEW 7.1 it somehow always wanted to pack the paramers tightly so that this copying corrupted the buffer pointer passed to the next parameter of the function. This was normally supposed to be a NULL pointer but of course wasn't NULL after this assignment and then the shared library crashed.

I'm pretty sure that this was also the reason why it normally would encounter trouble on 64-bit Linux. And no this problem did not exist in pre 4.1 versions. This particular structure got extended when I incorporated the latest minizip 1.2 sources from Nathan Moinvaziri to support 64-bit archive operation. Previous versions used the standard stock minizip 1.1 sources included in the zlib source distribution.

[ShaunR]

8 minutes ago, Rolf Kalbermatter said:

C does allow to assign whole structure variables by value and the compiler then generates code to copy the whole variable. Except that Visual C did not bother to make it exactly 43 bytes but simply copied 48 bytes which resulted in random trash from the stack being copied after the end of the variable.

That's why you are paid so much. Or if you aren't; you should be.

[Antoine Chalons]

On 9/4/2019 at 10:23 AM, Rolf Kalbermatter said:

The realtime support will only get extracted when installing into LabVIEW for Windows 32-bit and through a seperate exe file that is invoked and will prompt for an administrative elevation of this installer. You then have to go into NI Max and into the Software part of your target and select to install additional components. In the list should be an OpenG ZIP Tools version 4.2.0 package visible.

If I understand well it means, LabVIEW 32-bit is needed on the computer to be able to deploy the package to a RT target, right?

What would be the way to deploy this package to a Linux RT Target from a Windows computer that has NI Max but not LabVIEW?

Is it possible to extract the EXE you mention from the package, run it and then deploy from MAX?

[Rolf Kalbermatter]

On 10/8/2019 at 10:00 AM, Antoine Chalons said:

If I understand well it means, LabVIEW 32-bit is needed on the computer to be able to deploy the package to a RT target, right?

What would be the way to deploy this package to a Linux RT Target from a Windows computer that has NI Max but not LabVIEW?

Is it possible to extract the EXE you mention from the package, run it and then deploy from MAX?

The OpenG package file is simply a ZIP file in disguise, with an ini style package file (called spec) in the root that describes where the different file groups should go to in your LabVIEW installation and with restrictions for what version and platform they apply for. If you have 7-Zip installed you can right click on a *.ogp file and select 7-Zip->Open Archive.

Then look in the directories for "File Group 8" and in there is the ogsetup.exe file. This is an Inno Setup file that installs the necessary packages into the correct NI Shared location for RT packages. I choose to do it this way as the files have to be installed in a location that has only write access when the process is evelated and rather than requiring the user to restart VIPM explicitly as admin (and trying to guess the correct location to write the files to from within a post install hook VI), I created an Inno Setup installer for the necessary files with an embedded manifest that requests elevation authorization from the user. After that and provided you have full cRIO support for NI Max for your target installed on your machine, you should be able to select the package in the Custom Software Install from within NI Max.

Basically I choose to only extract the ogsetup.exe file into a LabVIEW 32-bit installation, since this is the only way to program LabVIEW RT programs anyway. I figured that the chance that someone would want to install SW packages to a RT target from a computer that is not used to program that target too, would be a very unlikely situation.

Edited November 28, 2019 by Rolf Kalbermatter

[Antoine Chalons]

Perfect, it's quite easy to do and it works just fine for me.

 

My use case is that I want to update my rtexe on an NI Linux RT target, previous version didn't use OpenG Zip, new version does.

And to update the target I have a computer that only has NI Max.

Is there a way to build my rtexe from LabVIEW in such a way that it includes the dependencies of the OpenG Zip package to make the deployement easier?

 

[Rolf Kalbermatter]

1 hour ago, Antoine Chalons said:

Perfect, it's quite easy to do and it works just fine for me.

 

My use case is that I want to update my rtexe on an NI Linux RT target, previous version didn't use OpenG Zip, new version does.

And to update the target I have a computer that only has NI Max.

Is there a way to build my rtexe from LabVIEW in such a way that it includes the dependencies of the OpenG Zip package to make the deployement easier?

 

Not really unless you want to build some very custom stuff into your executable itself. And you would need an executable that does not statically link to the OpenG ZIP library as at the time you start it up, it can not reference the shared library anywhere or the loading will fail and the executable is not executable. To me that sounds like a lot more trouble than it's worth and you have the same issue with NI Toolkits. If you make use of (for instance DAQmx functions) you have to make sure the DAQmx driver is installed onto the target before you can deploy an executable that makes use of it.

 

The Pharlap ETS platform did include some means of deploying shared libraries directly together with VIs when you deployed the program from the project but that may have sounded like a cool thing initially when a Windows DLL was simply executable on Pharlap too, but nowadays most Windows DLLs won't load on Pharlap and have to be created specifically for the Pharlap ETS target. For non Windows like OSes like the NI Linux RT system, where the elf shared library is all but Greek for the Windows based LabVIEW environment, this is even more complicated.

Edited October 8, 2019 by Rolf Kalbermatter

[Antoine Chalons]

Ok, thanks a lot for the detailed explanations.

I'll stick with the ogsetup.exe

[Antoine Chalons]

Ha, today a new customer (a university) is requiring us to deploy our software on a Linux computer (not NI Linux RT).

So we have a Linux machine on which we have LabVIEW and VIPM, we compile our application on this computer and then we need to deploy to another Linux computer that doesn't have either LabVIEW nor VIPM installed.

If we simply manually place the so files from File Group 6 into a folder - which folder by the way? -, is it going to be enough to make it work?

Cheers!

[Rolf Kalbermatter]

21 hours ago, Antoine Chalons said:

Ha, today a new customer (a university) is requiring us to deploy our software on a Linux computer (not NI Linux RT).

So we have a Linux machine on which we have LabVIEW and VIPM, we compile our application on this computer and then we need to deploy to another Linux computer that doesn't have either LabVIEW nor VIPM installed.

If we simply manually place the so files from File Group 6 into a folder - which folder by the way? -, is it going to be enough to make it work?

Cheers!

The lvzip.so file on normal Linux should generally remain in the root folder of the lvzip package for source code distributions. That should make sure that the application builder will include the shared library into the application build in a support folder (usually called data) on all platforms where that matters. Doesn't that work for you? I don't really have experience with application builds on Linux. I only use that platform for testing in the LabVIEW IDE and only really have two LabVIEW versions available for that. If LabVIEW does not automatically include shared libraries into an executable build on Linux you should probably include it explicitly in the build specification as always included and make sure it is put in the root folder of the application or the support folder.

It's not as simple as taking the lvzlib.so from File Group 6 for any Linux application. I was trying to find a way in VIPM to directly specify which file was for which bitness in a package but it doesn't support that on file group basis so far, only on whole package base.

Whichever shared library you use from that File Group needs to be called lvzlib.so though to make it work with the VIs. Some of these restrictions were dictated by the fact that I needed to support pre LabVIEW 2009 versions and some of my non Windows test environments are still based on such versions so just blanket upgrade everything to 2009 or beyond to get rid of some of these restrictions is not a simple option.

Edited October 10, 2019 by Rolf Kalbermatter

[Antoine Chalons]

5 minutes ago, Rolf Kalbermatter said:

The lvzip.so file on normal Linux should generally remain in the root folder of the lvzip package for source code distributions. That should make sure that the application builder will include the shared library into the application build in a support folder (usually called data) on all platforms where that matters. Doesn't that work for you? I don't really have experience with application builds on Linux. I only use that platform for testing in the LabVIEW IDE and only really have two LabVIEW versions available for that.

I knew it would be the case on Windows and I was/am not sure about Linux because I have not compiled an application on Linux yet, it will be my first so I'll be careful with this.

Thanks again for you support.

[Antoine Chalons]

25 minutes ago, Rolf Kalbermatter said:

Some of these restrictions were dictated by the fact that I needed to support pre LabVIEW 2009 versions

Pre-2009 Wow... how for back do you go? The mythical 7.1?

[Rolf Kalbermatter]

The VI sources are currently still in 7.0. mainly because one of my Linux test systems has the 7.0 version available. (The other is 8.6). I used to have projects that needed 7.1 support for this library.

[hooovahh]

As always, thank you very much for this continuation.  Inflate/Deflate on Linux RT is important for a side project I have lately so this is awesome.  I did notice that in VIPM if I open your package and already have the previous OpenG zip package installed, it doesn't do an upgrade but instead performs a new install.  It appears that the internal name of the package, or versioning changed in a way that VIPM doesn't recognize it as a new version and will install it along side the last official release of OpenG which I think was 4.0.  I was able to edit the OGP spec file and create a package that convinced VIPM that it was an upgrade.  I was curious if this was intentional to distinguish it from the official releases.

Oh it seems the old package name was "OpenG LabVIEW ZIP Library", while the new one is "OpenG ZIP Library", I think this combined with some version changing is what I needed to edit.