OpenG Zip 4.2?

[Rolf Kalbermatter]

Thank you for this report. It is not intentional at all but rather a potential problem as the two versions are not meant to be used alongside with possible cross linking causing many nasty problems.

I'm still busy working on 4.3 which will be a full release and have in the underlaying levels many completely incompatible changes, so it will be important to make sure it replaces any other versions in an exesting install (not machine wide but inside a specific LabVIEW installation). So if you can let me know about the actual changes you made I will try to incorporate that. I wasn't aware that the Package Name/Display Name was considered an identifying package identification feature and only assumed that the Package Name/Name was used, (which would make more sense).

[hooovahh]

Thanks, good to know.  I took your 4.1.0 release (maybe it was b2?) and edited it to make a package that was considered an upgrade.  I thought the changes I made were to the displayed name and version.  But now that I try to do the same with the 4.2 release in this thread I can't make it work.  I keep copying more stuff from the existing spec to the new one and it isn't working...I'll keep playing around and let you know if I get anything conclusive.

[hooovahh]

Okay I edited the spec file to start with the following:

[Package Name]
Name=oglib_lvzip
Version=4.2.0
Release=1
Display Name="OpenG LabVIEW ZIP Library"

Then I zipped the source back up and renamed the zip to oglib_lvzip-4.2.0-1.ogp.  I then installed the package and it didn't say "Upgrade" but instead said "Install".  But after installing there was only one entry and the result was the upgrade was successful.

[Antoine Chalons]

Hi,

Am I going to have problems if I change the re-entrancy settings of the VIs in GZip 4.2.0b1-1 from non-rentrant to pre-allocated clones?

I mainly use ZLIB Deflate.vi and ZLIB CRC32.vi

I use them both on Windows and NI Linux RT.

[Rolf Kalbermatter]

42 minutes ago, Antoine Chalons said:

Hi,

Am I going to have problems if I change the re-entrancy settings of the VIs in GZip 4.2.0b1-1 from non-rentrant to pre-allocated clones?

I mainly use ZLIB Deflate.vi and ZLIB CRC32.vi

I use them both on Windows and NI Linux RT.

I wouldn't expect real problems nor any specific advantages to doing so for these functions. The Call Library Nodes are already configured to run in any thread which allows the VI to execute the C function in whatever thread the VI is currently executing. As such it should not give you any serious performance improvements unless you intend to call these functions from many different locations in parallel and also only if you do this for pretty large string buffers. For short string buffers the actual execution time of these functions should be so short that the likelihood that one call has to wait for another call should be pretty small.

The disadvantage obviously is that you will have to redo this for every new version of the library. 😀

Edited March 3, 2020 by Rolf Kalbermatter

[Antoine Chalons]

Ok, thanks for details.

I will have between 1 and 5 clones of a dqmh module that send strings to different database via tcp or http, the strings can be large thus why I compress, but each module will only send a string every 1 to 10 seconds so, you're right, I probably wouldn't see much/any difference.

Edited March 4, 2020 by Antoine Chalons

[GregSands]

Are there any updates to this package?  The last I see in the thread is a beta version of 4.2.0, but the official package through VIPM seems to max out at 4.0.1-1.

[Rolf Kalbermatter]

10 hours ago, GregSands said:

Are there any updates to this package?  The last I see in the thread is a beta version of 4.2.0, but the official package through VIPM seems to max out at 4.0.1-1.

Well, I have a 4.3 alpha on github. But haven't been able to work on that for some time.

Edited April 22, 2021 by Rolf Kalbermatter

[jyoung8711]

I know this thread is pretty old, but I've been working with the beta version of 4.2.0, and came across something strange.

It seems like if I try to zip up files in LV 64-bit (2020/2021), if the files are at a network address (\\Network\file.ext) I end up with an error coming from the "fileutil.llb\FILE Resource File Info__ogtk.vi" file (and internally from the DLL call).

the 32-bit version seems to support network paths just fine, so this caught me a bit off-guard.  I took a look at the current version on github, and it looks like you were making some changes to the file utility functions, but it doesn't look like it resolved this issue (at least with a simple attempt on my side).

Any ideas what might be happening here?

Thanks in advance!

[Rolf Kalbermatter]

11 hours ago, jyoung8711 said:

I know this thread is pretty old, but I've been working with the beta version of 4.2.0, and came across something strange.

It seems like if I try to zip up files in LV 64-bit (2020/2021), if the files are at a network address (\\Network\file.ext) I end up with an error coming from the "fileutil.llb\FILE Resource File Info__ogtk.vi" file (and internally from the DLL call).

the 32-bit version seems to support network paths just fine, so this caught me a bit off-guard.  I took a look at the current version on github, and it looks like you were making some changes to the file utility functions, but it doesn't look like it resolved this issue (at least with a simple attempt on my side).

Any ideas what might be happening here?

Thanks in advance!

What would be the error you get?

[jyoung8711]

4 hours ago, Rolf Kalbermatter said:

What would be the error you get?

The DLL outputs an error "1". (the function call is LVPath_UtilFileInfo) - Apologies for not including in the previous post.

This is for the 4.2 beta version posted here with 64-bit LV (tried 2018, 2020, 2021). Windows 10 21H2

The same file, if moved to a local path, runs through the VI/Function just fine (or if executed in 32-bit version).

I haven't had the opportunity to try it on a second computer yet.

Thanks!

[Rolf Kalbermatter]

1 hour ago, jyoung8711 said:

The DLL outputs an error "1". (the function call is LVPath_UtilFileInfo) - Apologies for not including in the previous post.

This is for the 4.2 beta version posted here with 64-bit LV (tried 2018, 2020, 2021). Windows 10 21H2

The same file, if moved to a local path, runs through the VI/Function just fine (or if executed in 32-bit version).

I haven't had the opportunity to try it on a second computer yet.

Thanks!

Error 1 is the all generic "invalid parameter" error. could be indeed a problem in interpreting UNC paths somehow. I'll try to look into that.

Haven't really run the code yet on a 64-bit system with UNC paths, but I see where the error 1 seems to come from when looking in the source code. It looks like LabVIEW has changed its stance about what an UNC path represents between 32-bit and 64-bit. I use the function FIsAPathOfType() to check that the passed in path is an absolute path  ( I do not want to try to open relative, and of course empty or invalid paths, as I have no way to know where they should be relative to, and find the idea to use the current directory an atrocity that has absolutely no place in a modern multithreading application).

Going to verify that this is the culprit, as it could be also from somewhere else, but it looks suspicious and I know that internally UNC paths are treated as a different type in LabVIEW, but so far it considered them absolute too (which they are).

Edited March 17, 2022 by Rolf Kalbermatter

[Rolf Kalbermatter]

So I did take a look and yes it was that function but no, it doesn't only fail in 64-bit mode but also in 32-bit mode. So I'm a little lost why you feel it did work with an UNC path when using it in LabVIEW 32-bit. Going to do some more tests with this and trying to clean up a few related things.

Edited March 17, 2022 by Rolf Kalbermatter

[jyoung8711]

4 hours ago, Rolf Kalbermatter said:

So I did take a look and yes it was that function but no, it doesn't only fail in 64-bit mode but also in 32-bit mode. So I'm a little lost why you feel it did work with an UNC path when using it in LabVIEW 32-bit. Going to do some more tests with this and trying to clean up a few related things.

Ah... So, I know the reason for that! and it might help with troubleshooting...

I run the last "official" release for my 32-bit installations (4.0.1-1), and the 4.2 version found here for 64-bit -- [historical compatibility reasons... the newer dll caused issues with some older "common" code, while that version doesn't actually work in 64-bit].

 

Sorry for the confusion, but hopefully this might be helpful since it was working in the older version.

[Rolf Kalbermatter]

5 hours ago, jyoung8711 said:

Ah... So, I know the reason for that! and it might help with troubleshooting...

I run the last "official" release for my 32-bit installations (4.0.1-1), and the 4.2 version found here for 64-bit -- [historical compatibility reasons... the newer dll caused issues with some older "common" code, while that version doesn't actually work in 64-bit].

 

Sorry for the confusion, but hopefully this might be helpful since it was working in the older version.

It for sure helps. I was really thinking that I was overlooking something here. But with this explanation everything makes sense. The actual code in the DLL is substantially different to 4.0. In 4.0 most was just a very thin wrapper around existing LabVIEW functions. But those LabVIEW functions do not support Unicode paths so I have been refactoring that code substantially to support full Unicode paths for the underlying functions (and create compatibility wrappers that still use the old LabVIEW paths, which of course won't support Unicode path names).

The advantage of using full Unicode throughout the ZIP tools, eventually will be that path names can contain characters not present in your current ANSI locale and that path names can be almost arbitrarily long, namely 32k characters. The ZIP standard internally already supports UTF8 encoded file names, so once this is fully working you can also extract and create ZIP files that are using UTF8 filenames. But this complete porting to full Unicode support is not yet finished. Most of the actual programming is done, but it needs more testing.

Edited March 18, 2022 by Rolf Kalbermatter

[jyoung8711]

10 hours ago, Rolf Kalbermatter said:

It for sure helps. I was really thinking that I was overlooking something here. But with this explanation everything makes sense. The actual code in the DLL is substantially different to 4.0. In 4.0 most was just a very thin wrapper around existing LabVIEW functions. But those LabVIEW functions do not support Unicode paths so I have been refactoring that code substantially to support full Unicode paths for the underlying functions (and create compatibility wrappers that still use the old LabVIEW paths, which of course won't support Unicode path names).

The advantage of using full Unicode throughout the ZIP tools, eventually will be that path names can contain characters not present in your current ANSI locale and that path names can be almost arbitrarily long, namely 32k characters. The ZIP standard internally already supports UTF8 encoded file names, so once this is fully working you can also extract and create ZIP files that are using UTF8 filenames. But this complete porting to full Unicode support is not yet finished. Most of the actual programming is done, but it needs more testing.

Thanks for the additional context. That's helpful in understanding some of what's going on behind the scenes.

Also nice to know that it wasn't something "weird" on this particular system or something.  For the moment I've worked around this by moving everything into a temporary path on the system making the call, but long-term I'll look forward to the next release!

 

Thanks!

[bjustice]

Just installed the 4.2 beta package.  I'm looking for 64-bit support.

Is there some funkiness happening with the naming that is upsetting VIPM?  This beta package installs as a separate VIPM package with a separate palette, but it's still causing the non-beta ZIP library to "think" there's an upgrade available.

Is anyone else experiencing this?  It's causing some confusion here.

Thanks!

 

[Rolf Kalbermatter]

4 minutes ago, bjustice said:

Just installed the 4.2 beta package.  I'm looking for 64-bit support.

Is there some funkiness happening with the naming that is upsetting VIPM?  This beta package installs as a separate VIPM package with a separate palette, but it's still causing the non-beta ZIP library to "think" there's an upgrade available.

Is anyone else experiencing this?  It's causing some confusion here.

Thanks!

 

Read through this thread and read specifically Brian (Hoovahs) response.

[bjustice]

oops, sorry, I scanned a bit too fast then

Edit: Hooovah's spec changed also worked for me.  Upvote

Edited March 21, 2022 by bjustice

[Rolf Kalbermatter]

However, be aware that the github code as is is an experiment in construction. Lots of things don't work right yet and there are many bugs in the underlaying shared library that simply won't do the right thing yet. I'm slowly working on it but it is a side project and sometimes I just don't feel like debugging C code very much.

[bjustice]

On 3/22/2022 at 3:40 AM, Rolf Kalbermatter said:

However, be aware that the github code as is is an experiment in construction. Lots of things don't work right yet and there are many bugs in the underlaying shared library that simply won't do the right thing yet. I'm slowly working on it but it is a side project and sometimes I just don't feel like debugging C code very much.

Understood.

For what it's worth, this code met my personal needs.  Thanks for the community contributions!

[hooovahh]

So I've been using, and loving OpenG zip functions on my x86 based embedded cDAQ running RT Linux.  But I recently ran into an issue.  The newest versions of XNet only supports being installed as OPKG packages.  These can only be installed by using the System Image install option, and not the custom option that is needed for OpenG Zip.  As  a test I SSH'ed into my controller and copied the liblvzlib.so where it belonged, and my previously written code seems to find and use it.  Is there any plan to update the inno installer to somehow add feeds or something so that the OpenG Zip tools can be installed through MAX, on controllers that use the System Image option?

[Jordan Kuehn]

8 minutes ago, hooovahh said:

So I've been using, and loving OpenG zip functions on my x86 based embedded cDAQ running RT Linux.  But I recently ran into an issue.  The newest versions of XNet only supports being installed as OPKG packages.  These can only be installed by using the System Image install option, and not the custom option that is needed for OpenG Zip.  As  a test I SSH'ed into my controller and copied the liblvzlib.so where it belonged, and my previously written code seems to find and use it.  Is there any plan to update the inno installer to somehow add feeds or something so that the OpenG Zip tools can be installed through MAX, on controllers that use the System Image option?

You can call opkg install from the CLI and manually install a package you have copied over. Some discussion here where I built such a package. We use SystemLink which makes it easy to add the custom feeds and such, but hopefully this could help some?
 

 

[RomainP]

On 3/22/2022 at 9:40 AM, Rolf Kalbermatter said:

However, be aware that the github code as is is an experiment in construction. Lots of things don't work right yet and there are many bugs in the underlaying shared library that simply won't do the right thing yet. I'm slowly working on it but it is a side project and sometimes I just don't feel like debugging C code very much.

An official release of OpenG Zip with support of LabVIEW 64 bit would be great. Currently this library is the last one preventing our team from switching to LV 64 bit.
 

[Rolf Kalbermatter]

On 8/30/2022 at 4:44 PM, RomainP said:

An official release of OpenG Zip with support of LabVIEW 64 bit would be great. Currently this library is the last one preventing our team from switching to LV 64 bit.
 

I can't right now work on that. But I have plans to do that in the coming months. The story behind it is that I did a little more than just to make it 64-bits.

- The file IO operations where all rewritten to be part of the library itself rather than relying on LabVIEW file IO. While LabVIEW 8.0 and newer supports reading and writing files that are bigger than 2GB, it still has the awful habit to use internally old OS file IOs that are naturally limited to only supporting characters in file names that are part of your current local and they also normally are limited to 260 character long path names. If your drive is formatted in FAT32, that is all the drive can do for you anyhow, but except for USB thumb drives, you would be hard pressured to find any FAT formatted drives anymore. So having these limitations in the library feels very bad.

These two things are specially a problem on Windows. Mac is slightly less problematic and Linux has long ago pretty much solved it all internally in the kernel and surrounding system libraries.

- Modern ZIP files support things like symbolic links and I wanted a way to support them. For Linux and Mac that is a piece of cake. For Windows I may for now not be able to seamlessly support that as creating symlinks under Windows is a privileged action, so the user has to either be elevated or you have to set an obscure Developer flag in Windows that allows all users to create symlinks.

So in summary there was a lot of work to be done, most of it actually for Windows. Most of that is done but testing all that is a very frustrating job. And the non-Windows targets will then also take some more time for additional testing and making things that were modified for Windows compile again properly.

So yes, it's still on my to-do list and I'm planning to work on it again, but right now I have another project that requires my attention.

Because of the significant changes in the underlying shared library and internal organization of the VIs it will be almost certainly version 5.0. The official library API (those nice VIs with a green gift box in them) should remain compatible but if you want to make use of the new path name feature to fully support long path names with full character support, you may have to change to the new API, with the library specific path type, although if you use high level library functions, internal long path names will be ok, you just won't be able to access them with the normal LabVIEW file functions if they contain non local ANSI characters or are to long! It's the best I could come up with without the ability to actually changing the LabVIEW source code itself to add that feature into the internal Path Manager in LabVIEW. 😀

The according File Utilities Manager functions in the library will also be available for the user in a separate palette.

Edited September 1, 2022 by Rolf Kalbermatter