My ZLIB Deflate and Compression in G

[hooovahh]

So a couple of years ago I was reading about the ZLIB documentation on compression and how it works.  It was an interesting blog post going into how it works, and what compression algorithms like zip really do.  This is using the LZ77 and Huffman Tables. It was very education and I thought it might be fun to try to write some of it in G.  The deflate function in ZLIB is very well understood from an external code call and so the only real ever so slight place that it made sense in my head was to use it on LabVIEW RT.  The wonderful OpenG Zip package has support for Linux RT in version 4.2.0b1 as posted here. For now this is the version I will be sticking with because of the RT support.

Still I went on my little journey trying to make my own in pure LabVIEW to see what I could do.  My first attempt failed immensely and I did not have the knowledge, to understand what was wrong, or how to debug it.  As a test of AI progression I decided to dig up this old code and start asking AI about what I could do to improve my code, and to finally have it working properly.  Well over the holiday break Google Gemini delivered. It was very helpful for the first 90% or so. It was great having a dialog with back and forth asking about edge cases, and how things are handled.  It gave examples and knew what the next steps were.  Admittedly it is a somewhat academic problem, and so maybe that's why the AI did so well.  And I did still reference some of the other content online.  The last 10% were a bit of a pain. The AI hallucinated several times giving wrong information, or analyzed my byte streams incorrectly.  But this did help me understand it even more since I had to debug it.

So attached is my first go at it in 2022 Q3.  It requires some packages from VIPM.IO. Image Manipulation, for making some debug tree drawings which is actually disabled at the moment.  And the new version of my Array package 3.1.3.23.

So how is performance?  Well I only have the deflate function, and it only is on the dynamic table, which only gets called if there is some amount of data around 1K and larger.  I tested it with random stuff with lots of repetition and my 700k string took about 100ms to process while the OpenG method took about 2ms.  Compression was similar but OpenG was about 5% smaller too.  It was a lot of fun, I learned a lot, and will probably apply things I learned, but realistically I will stick with the OpenG for real work.  If there are improvements to make, the largest time sink is in detecting the patterns. It is a 32k sliding window and I'm unsure of what techniques can be used to make it faster.

ZLIB G Compression.zip

[Rolf Kalbermatter]

On 1/8/2026 at 10:37 PM, hooovahh said:

So a couple of years ago I was reading about the ZLIB documentation on compression and how it works.  It was an interesting blog post going into how it works, and what compression algorithms like zip really do.  This is using the LZ77 and Huffman Tables. It was very education and I thought it might be fun to try to write some of it in G.  The deflate function in ZLIB is very well understood from an external code call and so the only real ever so slight place that it made sense in my head was to use it on LabVIEW RT.  The wonderful OpenG Zip package has support for Linux RT in version 4.2.0b1 as posted here. For now this is the version I will be sticking with because of the RT support.

Still I went on my little journey trying to make my own in pure LabVIEW to see what I could do.  My first attempt failed immensely and I did not have the knowledge, to understand what was wrong, or how to debug it.  As a test of AI progression I decided to dig up this old code and start asking AI about what I could do to improve my code, and to finally have it working properly.  Well over the holiday break Google Gemini delivered. It was very helpful for the first 90% or so. It was great having a dialog with back and forth asking about edge cases, and how things are handled.  It gave examples and knew what the next steps were.  Admittedly it is a somewhat academic problem, and so maybe that's why the AI did so well.  And I did still reference some of the other content online.  The last 10% were a bit of a pain. The AI hallucinated several times giving wrong information, or analyzed my byte streams incorrectly.  But this did help me understand it even more since I had to debug it.

So attached is my first go at it in 2022 Q3.  It requires some packages from VIPM.IO. Image Manipulation, for making some debug tree drawings which is actually disabled at the moment.  And the new version of my Array package 3.1.3.23.

So how is performance?  Well I only have the deflate function, and it only is on the dynamic table, which only gets called if there is some amount of data around 1K and larger.  I tested it with random stuff with lots of repetition and my 700k string took about 100ms to process while the OpenG method took about 2ms.  Compression was similar but OpenG was about 5% smaller too.  It was a lot of fun, I learned a lot, and will probably apply things I learned, but realistically I will stick with the OpenG for real work.  If there are improvements to make, the largest time sink is in detecting the patterns. It is a 32k sliding window and I'm unsure of what techniques can be used to make it faster.

ZLIB G Compression.zip 348.33 kB · 4 downloads

Great effort. I always wondered about that, but looking at the zlib library it was clear that the full functionality was very complex and would take a lot of time to get working. And the biggest problem I saw was the testing. Bit level stuff in LabVIEW is very possible but it is also extremely easy to make errors (that's independent of LabVIEW btw) so getting that right is extremely difficult and just as difficult to proof consistently.

Performance is of course another issue. LabVIEW allows a lot of optimizations but when you work on bit level, the individual overhead of each LabVIEW function starts to add up, even if it is in itself just tiny fractions of microseconds. LabVIEW functions do more consistency checking to make sure nothing will crash ever because of out of bounds access and more. That's a nice thing and makes debugging LabVIEW code a lot easier, but it also eats performance, especially if these operations are done in inner loops million of times.

Cryptography is another area that has similar challenges, except that security requirements are even higher. Assumed security is worse than no security.

I have done in the past a collection of libraries to read and write image formats for TIFF, GIF and BMP. And even implemented the somewhat easier LZW algorithm used in some TIFF and GIF files. On the basic it consists of a collection of stream libraries to access files and binary data buffers as a stream of bytes or bits. It was never intended to be optimized for performance but for interoperability and complete platform independence. One partial regret I have is that I did not implement the compression and decompression layer as a stream based interface. This kind of breaks the easy interchangeability of various formats by just changing the according stream interface or layering an additional stream interface in the stack. But development of a consistent stream architecture is one of the more tricky things in object oriented programming. And implementing a decompressor or compressor as a stream interface is basically turning the whole processing inside out. Not impossible to do, but even more complex than a "simple" block oriented (de)compressor. And also a lot harder to debug.

Last but not least it is very incomplete. TIFF support is only for a limited amount of sub-formats, the decoding interface is somewhat more complete while the encoding part only supports basic formats. GIF is similar and BMP is just a very rudimentary skeleton. Another inconsistency is that some interfaces support the input and output to and from IMAQ while others support the 2D LabVIEW Pixmap, and the TIFF output supports both for some of the formats. So it's very sketchy.

I did use that library recently in a project where we were reading black/white images from IMAQ which only supports 8 bit greyscale images but the output needed to be 1-bit TIFF data to transfer to a inkjet print head. The previous approach was to save a TIFF file in IMAQ, which was stored as 8-bit grey scale with only really two different values and then invoke an external command to convert the file to 1 bit bi-level TIFF and transfer that to the printer. But that took quite a bit of time and did not allow to process the required 6 to 10 images per second. With this library I could do the full IMAQ to 1-bit TIFF conversion consistently in less than 50 ms per image including writing the file to disk. And I always wondered about what would be needed to extend the compressor/decompressor with a ZLIB inflate/deflate version which is another compression format used in TIFF (and PNG but I haven't considered that yet). The main issue is that adding native JPEG support would be a real hassle as many PNG files use internally a form of JPEG compression for real life images.

Edited 2 hours ago by Rolf Kalbermatter