Jump to content

Recommended Posts

I noticed on sourceforge that there is a version 4.2 of OpenG Zip. Will it be released as a package anytime soon?

  • Like 1

Share this post


Link to post
Share on other sites
On 8/20/2019 at 12:54 PM, Mads said:

I noticed on sourceforge that there is a version 4.2 of OpenG Zip. Will it be released as a package anytime soon?

I did work on that mainly end of last year but found some time to resume testing recently. The code and VIs are more or less ready but I do need to do a bit more testing on Linux, Mac and the different real-time targets. Especially Mac and the real-time  targets proof to be quite a hassle. Mac because I don't work often on it nowadays and the real-time targets because debugging shared libraries on them is always quite some hassle and each flavor is again different.

I could however use some extra eyes for testing and I don't mean the shared library part itself but simply the general operation of it. I might be able to create a preliminary OGP package for installation through VIPM within a week or so if you want to test it. Let me know which platforms you would want to test it on and how and I will check what I can do.

Share this post


Link to post
Share on other sites

Great. I would be glad to test it. I mainly work on ARM-based Linux RT targets myself, and the occational old VxWorks cFP-target.

Share this post


Link to post
Share on other sites
8 minutes ago, Mads said:

Great. I would be glad to test it. I mainly work on ARM-based Linux RT targets myself, and the occational old VxWorks cFP-target.

I probably can't test the VxWorks targets for now but can create at least the RT image to be installed for them. No guarantee that it can even load though without having tested it before myself.

Share this post


Link to post
Share on other sites

It appears that the shared libraries are fully threadsafe, given the calls are all set to run in any thread, and I don't think the zlib library is multithreaded.  Would there be any issues with setting the VIs to "Shared clone reentrant" to allow multiple simultaneous calls?

 

Share this post


Link to post
Share on other sites
13 hours ago, GregSands said:

It appears that the shared libraries are fully threadsafe, given the calls are all set to run in any thread, and I don't think the zlib library is multithreaded.  Would there be any issues with setting the VIs to "Shared clone reentrant" to allow multiple simultaneous calls?

It depends what you try to do. As long as you don't try to access a particular zip or unzip sessions from multiple places in parallel, the zlib zip library has always been safe for reentrant execution as it does not contain global state that spans across sessions. The underlaying zlib library (pure compression/decompression algorithms) itself is even less of a problem as it does not have the complex archive maintenance that is needed for a ZIP archive but only works on immediate memory streams.

If you try to open a zip (or unzip) session somewhere and then branch this to two different locations and try to write streams to them from both locations you are going to crash sooner or later. Each session stores state that is used across method invocations and even if I would protect the individual function calls with a per session mutex (or make the functions all execute in the UI thread) you would still potentially corrupt the zip archive stream or in the case of unzip operation retrieve a different stream than what you think you do in a particular location.

As long as you don't access a specific session (zip or unzip refnum) from multiple places in parallel you were always fine though and that will remain like this in the future. This pretty much is the same as trying to read or write to a specific file from multiple places (through he same refnum or a separately opened). You can do that but expecting the read and writes to work properly and have a proper data content in the file afterwards will be pretty much impossible. There is however no problem in writing (and/or reading) in parallel to two (or more) different files on disk.

So setting the VIs to shared clone should work (all the state is stored in the session behind the refnum) but I'm not going to do that for now.

Edited by Rolf Kalbermatter

Share this post


Link to post
Share on other sites

Here is a new version 4.2.0b1 for the ZIP library. I didn't test it in every LabVIEW version on every platform. What I did test was MacOSx 32-bit and 64-bit LabVIEW 2014, Windows 32-bit and 64-bit LabVIEW 7.0, 7.1, 8.6, 2009, 2016, 2018, Linux 32-bit 7.1 and 8.6, and NI Linux x86 LabVIEW 2016. Other realtime targets I hadn't handy at the moment. Support for Linux 64-bit and NI Linux RT ARM as well as VxWorks and Pharlap is contained.

The realtime support will only get extracted when installing into LabVIEW for Windows 32-bit and through a seperate exe file that is invoked and will prompt for an administrative elevation of this installer. You then have to go into NI Max and into the Software part of your target and select to install additional components. In the list should be an OpenG ZIP Tools version 4.2.0 package visible. Select that to be installed on your target.

 

There are still following problems that I haven't implemented/fixed yet:

1) archives that contain file names with other encodings than your platform code page will go certainly wrong. This is probably not solvable without doing absolutely every file IO operation in the shared library too, since the LabVIEW file IO functions don't support any other encodings in the path.

2) If you try to zip up directories containing soft/hard inks then the current implementation will compress the actual target file/directory into the archive instead of a link and expanding zip archives that contain such links will expand just a small text file continaing the link destination. This is something I'm looking into to solve in the next release by allowing optionally adding a special link entry into the archive and create such a link on the filesystem when extracting. This is mostly of concern on Linux and MacOSX. While Windows also allows for such links nowadays it is still quite an esoteric feature and user accessible support for it is minimal (you have to use the command line or install additional third party tools to create/modify such links).

 

Hope to hear from other platforms and versions and how it goes there. Without some feedback I'm not going to create a release. 

oglib_lvzip-4.2.0b1-1.ogp

  • Like 1

Share this post


Link to post
Share on other sites

Downloaded it and so far I've tested it on LabVIEW 2019 and Linux RT ARM (cRIO-9063) with success (compression/decompression of files and folders and deflate/enflate on strings). I'll try a different target type later today.

Only trouble so far is with the package format - my VIPM does not like opening ogp files from Windows (get an access error), but I can open it and install it manually from VIPM...That might be a local issue though.

Share this post


Link to post
Share on other sites
4 hours ago, Rolf Kalbermatter said:

archives that contain file names with other encodings than your platform code page will go certainly wrong. This is probably not solvable without doing absolutely every file IO operation in the shared library too, since the LabVIEW file IO functions don't support any other encodings in the path.

For windows, you only need to replace the file open with "CreatefileW". That also gives you the long path names. LabVIEW read/write/close are fine after that.

I got a popup warning after installation on LabVIEW 2009. Something about a dialogue error cluster being replaced. It was only a warning and made no difference to the operation, though. I'm guessing it is due to back-saving rather than being built in 2009.

image.png.36288cde679c3a13284d3d6247e419e5.png

Edited by ShaunR

Share this post


Link to post
Share on other sites
1 hour ago, ShaunR said:

I got a popup warning after installation on LabVIEW 2009. Something about a dialogue error cluster being replaced. It was only a warning and made no difference to the operation, though. I'm guessing it is due to back-saving rather than being built in 2009.

That is normal. The VIs are still in LabVIEW 7.0 format and the used File Dialog in that version did not have an error cluster. So when mutating it to 8.0 and higher LabVIEW will insert some compatiblity code on the fly.

1 hour ago, Mads said:

Downloaded it and so far I've tested it on LabVIEW 2019 and Linux RT ARM (cRIO-9063) with success (compression/decompression of files and folders and deflate/enflate on strings). I'll try a different target type later today.

Only trouble so far is with the package format - my VIPM does not like opening ogp files from Windows (get an access error), but I can open it and install it manually from VIPM...That might be a local issue though.

Hmmm, it works fine on my computer but it is VIPM 2014 and I'm not fully sure if I didn't do some registry fiddling in the past to fix something. I rather would like to avoid having to do a vi package which only can be installed with VIPM of the same version or newer.

Edited by Rolf Kalbermatter

Share this post


Link to post
Share on other sites
1 hour ago, ShaunR said:

For windows, you only need to replace the file open with "CreatefileW". That also gives you the long path names. LabVIEW read/write/close are fine after that.

Right but it's a bit more complicated than that, because the same issue although less extreme exists on all other LabVIEW platforms. And there are various trouble to turn the underlaying file id into a LabVIEW refnum. On Linux LabVIEW uses the posix functions, and so does it on MacOSX 64-bit but not on 32-bit or at least not the versions I tested (it could have changed later but that would be potentially even worse). In addition while modern Linux platforms use UTF8 throughout, that was different before and still can be configured differently (although I'm hard pressured to imagine why someone would do something stupid like that).

And to make matters worse there is no standardized way really to determine what codepage was used when a ZIP archive was created. There is a newer option to use UTF8 which is indicated by a flag in the file entry for each file, but if that flag is not set the entry is in whatever OEM codepage (not ANSI) the computer was using at the time the archive was created.  And that could and for most problematic archives also will be a different codepage than on the computer on which you want to extract the files.

It's a complicated problem and part of it is basically unsolvable.

Edited by Rolf Kalbermatter

Share this post


Link to post
Share on other sites

Tested the 4.2 beta with success on a Linux RT x64 target today (cRIO-9030), where I have never gotten it to work previously.  

Compressed and decompressed folders with multiple files and subfolders, and used the inflate/deflate functions. The files that were compressed were also transferred to a PC to verify them there, and vice-versa.

Share this post


Link to post
Share on other sites
9 hours ago, Mads said:

Tested the 4.2 beta with success on a Linux RT x64 target today (cRIO-9030), where I have never gotten it to work previously.  

Compressed and decompressed folders with multiple files and subfolders, and used the inflate/deflate functions. The files that were compressed were also transferred to a PC to verify them there, and vice-versa.

I believe you! 🙂  During testing of this release I came across a problem that first dumbfounded me. On most systems it seemed to work fine but when executing it in LabVIEW 7.1 on Windows it consistently crashed. The problem turned out to be memory alignment related. One of the data structures passed to the shared library happened to be 43 bytes long. Inside the shared library was however some assignment operator where an internal temperary variable of that structure located on the stack was first filled in and then assigned to the passed in variable. C does allow to assign whole structure variables by value and the compiler then generates code to copy the whole variable. Except that Visual C did not bother to make it exactly 43 bytes but simply copied 48 bytes which resulted in random trash from the stack being copied after the end of the variable. On most platforms LabVIEW seemed to align the parameters it was passing to the Call Library Node such that this extra buffer overwrite didn't collide with any of the other parameters , but in LabVIEW 7.1 it somehow always wanted to pack the paramers tightly so that this copying corrupted the buffer pointer passed to the next parameter of the function. This was normally supposed to be a NULL pointer but of course wasn't NULL after this assignment and then the shared library crashed.

I'm pretty sure that this was also the reason why it normally would encounter trouble on 64-bit Linux. And no this problem did not exist in pre 4.1 versions. This particular structure got extended when I incorporated the latest minizip 1.2 sources from Nathan Moinvaziri to support 64-bit archive operation. Previous versions used the standard stock minizip 1.1 sources included in the zlib source distribution.

  • Like 1

Share this post


Link to post
Share on other sites
8 minutes ago, Rolf Kalbermatter said:

C does allow to assign whole structure variables by value and the compiler then generates code to copy the whole variable. Except that Visual C did not bother to make it exactly 43 bytes but simply copied 48 bytes which resulted in random trash from the stack being copied after the end of the variable.

That's why you are paid so much. Or if you aren't; you should be.

  • Haha 1

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.