My ZLIB Deflate and Compression in G

[hooovahh]

So a couple of years ago I was reading about the ZLIB documentation on compression and how it works.  It was an interesting blog post going into how it works, and what compression algorithms like zip really do.  This is using the LZ77 and Huffman Tables. It was very education and I thought it might be fun to try to write some of it in G.  The deflate function in ZLIB is very well understood from an external code call and so the only real ever so slight place that it made sense in my head was to use it on LabVIEW RT.  The wonderful OpenG Zip package has support for Linux RT in version 4.2.0b1 as posted here. For now this is the version I will be sticking with because of the RT support.

Still I went on my little journey trying to make my own in pure LabVIEW to see what I could do.  My first attempt failed immensely and I did not have the knowledge, to understand what was wrong, or how to debug it.  As a test of AI progression I decided to dig up this old code and start asking AI about what I could do to improve my code, and to finally have it working properly.  Well over the holiday break Google Gemini delivered. It was very helpful for the first 90% or so. It was great having a dialog with back and forth asking about edge cases, and how things are handled.  It gave examples and knew what the next steps were.  Admittedly it is a somewhat academic problem, and so maybe that's why the AI did so well.  And I did still reference some of the other content online.  The last 10% were a bit of a pain. The AI hallucinated several times giving wrong information, or analyzed my byte streams incorrectly.  But this did help me understand it even more since I had to debug it.

So attached is my first go at it in 2022 Q3.  It requires some packages from VIPM.IO. Image Manipulation, for making some debug tree drawings which is actually disabled at the moment.  And the new version of my Array package 3.1.3.23.

So how is performance?  Well I only have the deflate function, and it only is on the dynamic table, which only gets called if there is some amount of data around 1K and larger.  I tested it with random stuff with lots of repetition and my 700k string took about 100ms to process while the OpenG method took about 2ms.  Compression was similar but OpenG was about 5% smaller too.  It was a lot of fun, I learned a lot, and will probably apply things I learned, but realistically I will stick with the OpenG for real work.  If there are improvements to make, the largest time sink is in detecting the patterns. It is a 32k sliding window and I'm unsure of what techniques can be used to make it faster.

ZLIB G Compression.zip