ZLIB Compress Files

[Joachim082]

Hi everyone,

I am trying to compress a file larger than 2 GB. Unfortunately I get the error code 4. Obviously there is a 2 GB file restriction. Is there a workarround to compress files larger than 2GB?

Best Regards,

--

Joachim

[ShaunR]

Hi everyone,

I am trying to compress a file larger than 2 GB. Unfortunately I get the error code 4. Obviously there is a 2 GB file restriction. Is there a workarround to compress files larger than 2GB?

Best Regards,

--

Joachim

The NI site discusses some work-arounds. Bit it 'aint pretty.

I don't know if the OpenG zip implementation is also restricted or not (2GB is a limitation of the original Zip spec, I think). But you could give that a try.

[Joachim082]

ShaunR,

I am already trying the openg implementation. is there also a 64bit implementation of the zlib-package? RAR would be an opportunity but I would prefer the zip-implementation.

[ShaunR]

ShaunR,

I am already trying the openg implementation. is there also a 64bit implementation of the zlib-package? RAR would be an opportunity but I would prefer the zip-implementation.

I'm not aware of 64 bit support for the OpenG Zip package. (I'm also not aware of a rar implementation at all). There was a discussion here a while back. Might be worth nudging Rolf nicely to see how far he got.

Edited January 3, 2011 by ShaunR

[hooovahh]

I'm always a proponent of the program 7-zip. 7-zip can make zips, rars, gzips, tars, 7-zip files, cab files and several other compressions types.

In the download section they have a command line version (which also comes in the installer) which can be used to create archives. The command line switches are a little complicated but there's tons of documentation online. You could always go with the system exec option in LabVIEW calling this EXE. There is a 64-bit version as well but I don't know if the command line program would be 64 bit, probably just the UI application.

EDIT: Also one other thing to think about and that is if your file system is FAT32 then the maximum file size is 2GB so changing to using a rar wouldn't fix this. Ensure that if you are using windows that the file system is NTFS.

[asbo]

EDIT: Also one other thing to think about and that is if your file system is FAT32 then the maximum file size is 2GB so changing to using a rar wouldn't fix this. Ensure that if you are using windows that the file system is NTFS.

The maximum file size for FAT32 is actually 4GB so I wouldn't expect that to be the problem (unless some other funky filesystem is being used), but I definitely second the vote for 7zip.

[ShaunR]

7zip isn't a compression method. It's really a unifying (and extendible) interface to several methods the default being LZMA. As such, it's really pants to try and wrap it in LV (ever wonder why no-one has written a LV wrapper?) It'd be much easier to compile the separate dlls (zip,LZMA et al) and use those.

But the OP only wants one....any one!, As long a it can do >2GB. If the openG implementation can do it then all he needs is the x64 dll and it'll plop straight in.

Edited January 3, 2011 by ShaunR

[jcarmody]

[...] it's really pants to try [...]

I had to look that up.

Definition

pants adjective

/pænts/ adj [after verb] UK slang

not useful or of bad quality This music is pants.

(Definition of pants adjective from the Cambridge Advanced Learner's Dictionary)

[ShaunR]

I had to look that up.

Definition

pants adjective

/pænts/ adj [after verb] UK slang

not useful or of bad quality This music is pants.

(Definition of pants adjective from the Cambridge Advanced Learner's Dictionary)

So obscure you had to look it up "TWICE"

[hooovahh]

The maximum file size for FAT32 is actually 4GB so I wouldn't expect that to be the problem

Thanks I thought it was 2GB but it was 4GB. Also random but the maximum partition size of FAT16 is also 4GB.

I had to look that up.

Definition

pants adjective

/pænts/ adj [after verb] UK slang

not useful or of bad quality This music is pants.

(Definition of pants adjective from the Cambridge Advanced Learner's Dictionary)

Thanks, I read that sentence several times wondering if it was a spelling mistake.

[Rolf Kalbermatter]

7zip isn't a compression method. It's really a unifying (and extendible) interface to several methods the default being LZMA. As such, it's really pants to try and wrap it in LV (ever wonder why no-one has written a LV wrapper?) It'd be much easier to compile the separate dlls (zip,LZMA et al) and use those.

But the OP only wants one....any one!, As long a it can do >2GB. If the openG implementation can do it then all he needs is the x64 dll and it'll plop straight in.

Actually 64Bit and support for >2GB ZIP archives are two distinct issue. I have in the current HEAD integrated a new version of zlib which would support 64Bit offsets internally which is what is required to support archives containing more than 2GB components. The same version also SHOULD be compilable as 64Bit code. While these two things happen to be incorporated in a working way in the current ZLIB library, they are not equivalent and even not really related.

I have come across some snatch in compiling the LVZIP DLL for 64 Bit and haven't really spent much more time on this until now. Well I can compile the code but it crashes and since I don't have a 64 Bit system at hand with a compatible source level debugger and LabVIEW installation (they both also need to be 64Bits) I can't at the moment debug it. The code as it is in the repository would apart from that crash be fully 64 bit capable and should also be able to handle >2GB components although that is nothing I have ever tested nor have thought to test so far.

So the OpenG library as it is in the repository SHOULD be able to handle >2GB files but that will also need re-factoring of some VIs to allow for 64 bit integers wherever offsets are passed to and from the DLL functions (those 64 bit capable functions have a special 64 postfix.

[ShaunR]

Actually 64Bit and support for >2GB ZIP archives are two distinct issue. I have in the current HEAD integrated a new version of zlib which would support 64Bit offsets internally which is what is required to support archives containing more than 2GB components. The same version also SHOULD be compilable as 64Bit code. While these two things happen to be incorporated in a working way in the current ZLIB library, they are not equivalent and even not really related.

I have come across some snatch in compiling the LVZIP DLL for 64 Bit and haven't really spent much more time on this until now. Well I can compile the code but it crashes and since I don't have a 64 Bit system at hand with a compatible source level debugger and LabVIEW installation (they both also need to be 64Bits) I can't at the moment debug it. The code as it is in the repository would apart from that crash be fully 64 bit capable and should also be able to handle >2GB components although that is nothing I have ever tested nor have thought to test so far.

So the OpenG library as it is in the repository SHOULD be able to handle >2GB files but that will also need re-factoring of some VIs to allow for 64 bit integers wherever offsets are passed to and from the DLL functions (those 64 bit capable functions have a special 64 postfix.

Indeed they are different issues .But LV x64 cannot load x32 dll's so even if the 32 bit dll is compiled with u64 pointers; it won't work in LV64 (which I think is why the OP was asking for an x64 dll). Anything that relies on dll's needs to be supplied with both and detect the bitness to work on both platforms (either on install or on the fly).

But I did play with the 32bit one when the OP asked, and by replacing all the I32 controls and coercions for u64 (in the write) actually got it to write a 4.3 GB file without complaining (quite surprised, although I can't see how the content would be valid.). The LV file functions are u64 so the coercions were causing it to throw the file error. But then it failed on the finalisation. The result was I could write a 4.3GB invalid file . Since finalisation was a direct call, there wasn't much further I could go without delving into the C source and thought I'd leave it to the expert .

Which repository?

It's great to hear your looking into x64

Edited January 8, 2011 by ShaunR

[Rolf Kalbermatter]

But I did play with the 32bit one when the OP asked, and by replacing all the I32 controls and coercions for u64 (in the write) actually got it to write a 4.3 GB file without complaining (quite surprised, although I can't see how the content would be valid.). The LV file functions are u64 so the coercions were causing it to throw the file error. But then it failed on the finalisation. The result was

Well changing only the offsets into 64 bit integers won't work. You also need to change the calls to use the 64 bit versions of the functions. If they simply changed the offsets to just support 64 bits, backwards compatibility would be broken. A 64 bit integer is passed to a function in two 32 Bit addresses on the stack so changing all 32 Bit offsets into 64 Bits only, it would seem strange that you could even run an entire chain of VIs as this mismatch should mess up the stack alignment. Although thinking about it, the functions are all cdecl and that means the caller cleans up the stack afterward so it may work ok anyhow, if there are no other parameters after the offset in the parameter list.

Which repository?

It's great to hear your looking into x64

Like all OpenG libraries it's all on SourceForge.

[ShaunR]

Well changing only the offsets into 64 bit integers won't work. You also need to change the calls to use the 64 bit versions of the functions. If they simply changed the offsets to just support 64 bits, backwards compatibility would be broken. A 64 bit integer is passed to a function in two 32 Bit addresses on the stack so changing all 32 Bit offsets into 64 Bits only, it would seem strange that you could even run an entire chain of VIs as this mismatch should mess up the stack alignment. Although thinking about it, the functions are all cdecl and that means the caller cleans up the stack afterward so it may work ok anyhow, if there are no other parameters after the offset in the parameter list.

Like all OpenG libraries it's all on SourceForge.

I did download and create the zlib from here a while ago (has x32 & x64 object assemblies). Not sure if it will do over 2GB (don't see why it shouldn't). But the issue here is that you use an intermediary DLL lvzip.dll so it's not a simple replace as I assumed originally.l Although I'm not sure why (maybe speed?); I have seen it a lot with labview interfaces to dll's.

[Rolf Kalbermatter]

I did download and create the zlib from here a while ago (has x32 & x64 object assemblies). Not sure if it will do over 2GB (don't see why it shouldn't). But the issue here is that you use an intermediary DLL lvzip.dll so it's not a simple replace as I assumed originally.l Although I'm not sure why (maybe speed?); I have seen it a lot with labview interfaces to dll's.

The intermediary DLL is not so much a seperate DLL as the combination of the original zlib source code with the additional zip support that is now official part of the contribution directory but used to be just an optional component by someone else, and on top of that some utility functions to help support with LabVIEW integration. I could have left that all in separate DLLs but since I had to modify some zlib header files anyhow (to make the functions cdecl instead of stdcall as they used to be in older versions on Windows, and later on to decorate the exported functions with my own prefix to workaround naming clashes in the cRIO systems with empty stubs included in the system image) I decided to derive from the concept of providing fully original DLLs. And since the entire build process already was required anyhow for the additional LabVIEW specific part it wasn't really much extra work. In fact it makes building the shared library easier and also the packaging and distribution since only one shared library is required. The drawback is that merging in new zlib/zip versions is a little bit more complicated but last time when I added the new zlib version to it, the actual merging was done in less than an hour since I still only have to manually check and adapt two of the header files.

[ShaunR]

The intermediary DLL is not so much a seperate DLL as the combination of the original zlib source code with the additional zip support that is now official part of the contribution directory but used to be just an optional component by someone else, and on top of that some utility functions to help support with LabVIEW integration. I could have left that all in separate DLLs but since I had to modify some zlib header files anyhow (to make the functions cdecl instead of stdcall as they used to be in older versions on Windows, and later on to decorate the exported functions with my own prefix to workaround naming clashes in the cRIO systems with empty stubs included in the system image) I decided to derive from the concept of providing fully original DLLs. And since the entire build process already was required anyhow for the additional LabVIEW specific part it wasn't really much extra work. In fact it makes building the shared library easier and also the packaging and distribution since only one shared library is required. The drawback is that merging in new zlib/zip versions is a little bit more complicated but last time when I added the new zlib version to it, the actual merging was done in less than an hour since I still only have to manually check and adapt two of the header files.

I didn't mean for you to defend the decision (apologies if it came across that way). I as just commenting on the fact that many people use an intermediary DLL which makes it difficult for me to just plop the latest version, pre-compiled library, when there's an update.

But I think it's great that you are investigating the x64 since it's prevented me from using the library in the past. LV now comes shipped with zip file support, but it doesn't enable in memory compression which I believe yours does support

[Rolf Kalbermatter]

I didn't mean for you to defend the decision (apologies if it came across that way). I as just commenting on the fact that many people use an intermediary DLL which makes it difficult for me to just plop the latest version, pre-compiled library, when there's an update.

Well I didn't feel I had to defend myself, but wanted to explain the issues as they apply to other use cases too. The work on this library actually prompted me to write some articles about External Code use in LabVIEW on the Expression Flow blog, and touches all these issues and some more.

But I think it's great that you are investigating the x64 since it's prevented me from using the library in the past. LV now comes shipped with zip file support, but it doesn't enable in memory compression which I believe yours does support

Yes it supports any combination of memory and file based archive and archive content datastreams (and I'm fairly proud of that accomplishment). This means that both for compressing and uncompressing, the source and target can be any combination of memory or file based data stream.

Accessing the memory stream of archive components was a requirement to allow copying password protected archives (for instance when adding a new component or removing an existing one). And no it doesn't compromise password protection since the entire encrypted stream is copied and to unpack it you still need to know the original password. You could easily create archives in that way that are hard to open in some normal unarchiving tools, since different internal components can end up having different passwords, but that is simply a usage problem, not a fundamental problem of the ZIP archive or its operation.

[Kas]

Has there been any more headway towards this? I get the attached error with files <2GB. The zip file is created up to 2GB but then the error comes up.

I'm using LabView 2012 32bit in Windows 7 64 bit.

Thanks

Edited November 8, 2012 by Kas

[Wouter]

Don't the VI's give you the option to create X number of zip files? Each Y Bytes big? Like you see often? filename.rar0, filename.rar1, etc...

[Kas]

Don't the VI's give you the option to create X number of zip files? Each Y Bytes big? Like you see often? filename.rar0, filename.rar1, etc...

I haven't really looked into that, but it wouldn't resolve the issue even if they do. Trouble is getting away from this 2GB limit. One workaround would be to go through cmd prompt, but I was hoping the native solution would work.

[Stobber]

Is this where I revive discussion of ZLIB support for 64-bit LabVIEW? Because I need it.

[ShaunR]

Is this where I revive discussion of ZLIB support for 64-bit LabVIEW? Because I need it.

 

LabVIEW already supports 64-bit (File IO>>Zip).

 

If you are after the OpenG stuff, it's all going on here.

[lordexod]

Can use "wim" file format

https://technet.microsoft.com/en-us/library/cc749478%28v=ws.10%29.aspx

 

Software "Imagex.exe" .

https://technet.microsoft.com/en-us/library/cc507842.aspx

 

WIMGAPI - wim api.

https://msdn.microsoft.com/en-us/library/windows/desktop/dd851918.aspx

https://msdn.microsoft.com/en-us/library/windows/desktop/dd834964.aspx