.net decompression GZip or deflate

[Benoit]

Hi everyone,

I am struggling in one weird behaviors concerning .net in LabVIEW.

 Why as the picture bellow show, I cannot select the method of compression/decompression?

In LabVIEW 2011 this method works as expected. It seems that the constructor changed since the old .net. but for some reason, I cannot decompress any string with any software I did with my LabVIEW 2017.

 

Anyone know how to do that or understand what is going on?

 

PLEASE!!! do not suggest me external tools. this is a requirements to use the native .net in Windows 10... Security reason... I cannot use external .dll and I have a s**t load of data in our database that cannot be accesses anymore. So I need to use the native .net.

 

Thanks everyone for your support.

 

Benoit

[smithd]

You could try going back to the older .net version:

https://knowledge.ni.com/KnowledgeArticleDetails?id=kA00Z0000019LDcSAM

The Gzipstream msdn doc says:

Starting with the .NET Framework 4.5, the DeflateStream [which~=Gzipstream] class uses the zlib library for compression. As a result, it provides a better compression algorithm and, in most cases, a smaller compressed file than it provides in earlier versions of the .NET Framework.

 

I'm assuming that a zip is a zip and the algorithm doesn't matter, but maybe something else changed then as well.

[Stinus Olsen]

While I know this topic is a bit old now, I just wanted to point you to this thread that I created a while back.

The behavior you're observing is a known bug in LabVIEW that currently doesn't have a solution - just workarounds that all relies on external components ?

[Benoit]

Thanks,

But if you read this thread, you will notice that I posted already.

I cannot use external code from anyone outside... possible security threat. We had no choice to do as you said anyway. The problem with NI is that they prefer to release new stuff with bug than fixing the old one since they believe that there is a work around... ?

Benoit

[candidus]

Don't always believe what others say :-) When the .NET support of LV doesn't suit your needs use .NET reflection. This way you can get exactly the constructor you want. Here is some code to show the idea:

GZipStream.zip

[Benoit]

OK I see code, but I see no explanation. I see no way to get the string in and out. As you may notice, I am not good at .net.

If your solution is really working, Please add some comments so I can understand it.

Moreover, there is plenty of people that are struggling to find a way out. If there is, like you said, please share your knowledge.

Thanks

Benoit

[Gribo]

The inputs and outputs are streams. You can convert them to strings.

[candidus]

Benoit, sorry about the missing documentation. The snippet above is just the constructor call

new GZipStream(Stream stream, CompressionLevel compressionLevel)

expressed via .NET reflection. Yes, it is that complicated but at least it's possible.

For a more complete example see the ZIP file appended to my first post. I will write a more detailed explanation tomorrow.

[Benoit]

It will look very basic my question, but it's because my understanding of the .net is very basic.

what do you mean by reflection?

I'm waiting impatiently for your example with more documentation.

Benoit

[smithd]

https://dotnetcademy.net/Learn/4/Pages/1

Its roughly analogous to combination of VI server/scripting + lvoop class loading + variant data type inspection VIs + openg type code inspection VIs...except with a more consistent design

His point is just "if you can't call the constructor directly because labview is broken, you can have .net call the constructor manually, by name, using reflection". The direct labview equivalent of what he posted is https://zone.ni.com/reference/en-XX/help/371361P-01/glang/get_lv_class_default_value/

Edited November 13, 2018 by smithd

[candidus]

17 hours ago, smithd said:

https://dotnetcademy.net/Learn/4/Pages/1

Its roughly analogous to combination of VI server/scripting + lvoop class loading + variant data type inspection VIs + openg type code inspection VIs...except with a more consistent design

Great article, thanks! And a pretty cool way to explain things in LV terms

 

OK, here the detailed explanation as promised:

You can find a very simple example in the ZIP file attached to my first post, GZipStream.Example.vi . There's a file Hello.txt, if you run GZipStream.Example.vi you'll get the compressed file Hello.txt.gz . To decompress it, activate the other frame of the diagram disable structure and run it. The output will be Hello2.txt . There's not much to say about this example VI, the code consists of just .NET invoke nodes and properties.

The code of CreateGZIPStream.vi is that what matters here. We use .NET reflection to call a constructor that would be otherwise inaccessable for LV. Reflection is the ability to query a .NET assembly (DLL or EXE) for the classes (and other types) contained within and ask these classes for their properties, methods, events and constructors. It is built into the .NET framework and thus always at your service if the .NET support of LV isn't enough. The most important .NET data type when using reflection is System.Type .

OK, let's now dissect the code. Sorry about using C# here but I know no better way to explain .NET code.

This is what we want to do:

GZipStream CreateGZipStream(Stream stream, CompressionMode compressionMode) {

 // Call the constructor.

 return new GZipStream(stream, compressionMode);

}

Since LV doesn't allow us to select this constructor we need another way to get and invoke it. Reflection makes that possible, although the code becomes much more complex. CreateGZipStream.vi is roughly equivalent to the following C# code:

GZipStream CreateGZipStream(Stream stream, CompressionMode compressionMode) {

// Get the types.

Type streamType = typeof(Stream);

Type compressionModeType = typeof(CompressionMode);

Type gzipStreamType = typeof(GZipStream);

// To get the desired constructor we need its argument types and bundle them into an array.

Type[] constructorArgTypes = new Type[] { streamType, compressionModeType };

// Get the constructor.

ConstructorInfo gzipStreamConstructor = gzipStreamType.GetConstructor(constructorArgTypes);

// Bundle the constructor arguments into an array.

object[] constructorArgs =new object[] { stream, compressionMode };

// Call the constructor and cast the result.

return gzipStreamConstructor.Invoke(constructorArgs) as GZipStream;

}

Unfortunately LV complicates things a little further: We have no direct equivalent of typeof(). We can work around that by either using GetType() or loading the type from the assembly by name. Using GetType() is easier but it requires a valid .NET reference. Thus we need both:

GZipStream CreateGZipStream(Stream stream, CompressionMode compressionMode) {

 // Get the types.

 Type streamType = stream.GetType();

 Type compressionModeType = compressionMode.GetType();

 Type gzipStreamType = compressionModeType.Assembly.GetType("System.IO.Compression.GZIPStream");

 // To get the desired constructor we need its argument types and bundle them into an array.

 Type[] constructorArgTypes =new Type[] { streamType, compressionModeType };

 // Get the constructor.

 ConstructorInfo gzipStreamConstructor = gzipStreamType.GetConstructor(constructorArgTypes);

 // Bundle the constructor arguments into an array.

 object[] constructorArgs =new object[] { stream, compressionMode };

 // Call the constructor and cast the result.

 return gzipStreamConstructor.Invoke(constructorArgs) as GZipStream;

}

But that's still not enough. For LV CompressionMode is an enum, in .NET an enum is a full featured object. So we have also to construct a CompressionMode .NET object in LV and set its enum value. There's no direct equivalent to the 'value__' property in C#, I guess LV does something special here. Now we're ready to invoke the constructor in LV. The snippet again:

We construct the CompressionMode object and use its Assembly property to get the GZIPStream Type object. Then we ask for the constructor using the parameter types bundled into an array. Finally we call that constructor with the arguments bundled into an array and cast the result to GZipStream.

A note about the style of the code: When it comes to external reference-based code I use a single error wire through all nodes to enforce some kind of control flow. Furthermore it's important to close all open .NET references after using them.

I'm using .NET reflection to do things that are not supported by LV for a couple of years now, in my case .NET Remoting with events, see here (no reflective constructor invocation, though).

 

 

 

 

 

[Benoit]

?

OK I'll try to digest that. I'll need a couple of days.

But thank you very much.

Even that I can use your example and make something working right now, I still do not understand it enough to create it by myself from scratch.

But I'll get to that point very soon.

Thanks again.

I leave all of you with an unanswered question... Why NI do not provide this as a workaround? They always said create an external .dll to access .net function...

Benoit

[Michael Aivaliotis]

17 hours ago, Benoit said:

Why NI do not provide this as a workaround?

NI doesn't know everything.

[Novgorod]

On 11/14/2018 at 7:55 AM, candidus said:

OK, here the detailed explanation as promised:

[...]

Hi there from the future! I stumbled upon the same problem due to this exact .NET bug in labview, which apparently is known to NI since at least 2014 and hasn't been fixed 5 years later. The issue is that you can't select the constructor with CompressionMode as parameter - it's always overridden with the version using CompressionLevel as parameter. So you can use it for compression (because the default CompressionMode is compression) but you can't decompress.

Anyway, thanks for the great workaround and the excellent explanation - that's rare to find!

[Phillip Brooks]

I contributed a GZIP compress/decompress function on the NI forums many years ago. That function used a MemoryStream instead of a file.

I added the .NET reflection functionality from above to the original decompress VI based on a MemoryStream.

https://forums.ni.com/t5/Example-Programs/GZIP-compress-uncompress-of-string-using-NET/ta-p/3507908

 

Edited October 21, 2019 by Phillip Brooks

[X___]

A simpler way might be to do this using native LV functions:

Of course you need to create two temporary files, but this is year 2020...

[hooovahh]

The native LV functions don't implement the inflate/deflate from zlib but instead a different standard (pkzip?).  Also if you do want to compress a stream of bytes using the native zip feature you don't need to create the initial temporary file.  You can compress data using an array of bytes saving it to a zip.  I made a VIM and posted it here which supports an array of bytes, in memory TDMS, DVRs, and a couple other data types.

[X___]

It does seem to do exactly what I needed though, which was emulate that piece of MATLAB code:

zipbuf = fread(fid, offset, 'uint8=>uint8');
tmp_dir= tempname;
[SUCCESS,MESSAGE] = mkdir(tmp_dir);
if SUCCESS
    fnz= fullfile(tmp_dir,'temp.zip');
else
    disp(MESSAGE);
    error('Cannot create folder.')
end
tmp_fid = fopen(fnz,'w');
fwrite(tmp_fid, zipbuf);
status = fclose(tmp_fid);
fnu=char(unzip(fnz,tmp_dir) );
tmp_fid = fopen(fnu, 'r');
data = fread(tmp_fid, length, type_str);
status = fclose(tmp_fid);
[SUCCESS,MESSAGE] = rmdir(tmp_dir, 's');
if ~SUCCESS
    disp(MESSAGE);
    error('Cannot delete folder.')
end
end
 

[hooovahh]

That's good.  I just know that for other things it can be more selective on the compression type.  I wanted to make a way to create a PNG file outside of NI's native way due to some limitations.  I read up on the documentation and it honestly wasn't all that complicated.  The only issue is that part of the file is compressed with zlib.  Using OpenG it made good normal PNGs file that could be viewed in any image viewer.  But using LabVIEW's native API for compression created an image that only some image viewers would display, chrome was not one of them.  I also made a BLF file API.  This is a logging file format for raw automotive CAN data.  It also has chunks of data that are compressed.  Using NI's native method created a file that Vector tools couldn't open and I also needed OpenG.

[X___]

The NI_Unzip.lvlib VIs are all password protected (probably because underneath, they are using some .NET calls they don't want people to mess up with). So yes we don't know what they do, although the help tells us what they can't do:

Unzips the contents of Zip file to the Target directory. Set Preview only? to TRUE to preview the list of files in Zip file before you unzip the zip file. This VI cannot unzip a zip file that is password protected.

Why wouldn't it support password protected archives?

 

Anyway, the compression/decompression algorithms are not necessarily particularly well described or unique for that matter and it is therefore not surprising that some implementations might be incompatible (although they shouldn't):

.NET documentation

https://docs.microsoft.com/en-us/dotnet/api/system.io.compression.deflatestream?view=netcore-3.1

An old 40,000 ft description:

https://zlib.net/feldspar.html

quoted in this interesting "from-the-ground-up" implementation (quoting the previous reference):

http://www.integpg.com/deflate-compression-algorithm/

[hooovahh]

1 hour ago, X___ said:

The NI_Unzip.lvlib VIs are all password protected (probably because underneath, they are using some .NET calls they don't want people to mess up with). So yes we don't know what they do, although the help tells us what they can't do:

It is a series of call library nodes.  I think if it were .NET NI would have a hard time getting it to run on other targets like Linux, Pharlap, VXWorks, and issues with ARM/x86.  In the two examples I linked to I show how these are called to do compression with an array of bytes by calling them.  Still a pure G implementation would be nice, just looking over the source and documentation is quite daunting based on my lack of understanding compression techniques, but those links do help.

[X___]

Are you sure of that? The hierarchy shows System Exec call, so it might be even cruder than a .NET call indeed:

 

[hooovahh]

Yes I'm sure.  I just opened the block diagram of the Add File to Zip, and Unzip functions in 2020 and I believe they are unchanged from 2018 at least calling functions from within LabVIEW.  NI is calling system exec to touch the zip file, setting the file attribute that I believe is the file creation date, of the file that was uncompressed.  Obviously the date will be right now since the file was just extracted.  After it is done being uncompressed NI wants to touch the file, so it has the creation date that is stored for that file within the zip.  This can be seen in the unpassword protected VI Set Unzip File Date Time.  The VIM I linked to earlier shows how these call library nodes create the zip.

Edited May 20, 2020 by hooovahh

[X___]

OK. What is the idea of password protecting those CLN and not, say, those in the Analysis VIs. Are we at risk of blowing up the planet?

[hooovahh]

You'd have to ask NI, but I suspect they don't want to have to support users calling the functions directly, and instead they only want to support the users calling the VIs that they create, exposing the functions.  It gives them tighter control, and likely means less unit testing if they know a function will only ever be called a certain way.  For instance the Write PNG File has no compression.  The function internally does have a compression input, and setting it does create a more compressed PNG image.  But it is possible that compression isn't tested or supported on all platforms.  So they can either support compression on some targets and not others, either making duplicate VIs, or erroring out on those targets.  Or they just take that ability away.  Well that or someone just didn't realize that input would be requested and left it off the connector pane.