Jump to content
gb119

LabVIEW 2019 Map data type

Recommended Posts

So I got very excited when I saw that LabVIEW 2019 has a new native map type (aka hash array, associative array, dictionary) and so decided to have a play and see how it compared to my home-rolled LVOOP has array that uses variant attributes and I must admit that I'm slightly underwhelmed....

I've now benchmarked the 2019 native map class and a simple variant attribute by creating maps of 1 million elements of randomly generated 8 byte keys and then reading back 10,000 randomly selected elements and fairly consistently the native map is about an extra 50% slower than the variant attribute for both read and write operations. I'll admit my benchmark code is quite naive, but it's still a little disappointing that there is quite this performance hit....

Can any of the NI folks here comment on the performance - is it just that in fact variant attributes are blazingly fast?  I know I shouldn't be churlish and it's really great that 2019 has so many new features and a native map and set type have been long overdue...

 

Edit: and yes I've spotted that the variant type conversion was wired wrong and I should have been generating an array of 10000 I32 not error clusters - but no it doesn't make a significant difference....

map snippet.png

Variant Snippet.png

Edited by gb119
  • Like 2

Share this post


Link to post
Share on other sites

You don't need to type cast to string when using maps. Change the keys in your Map code to be U64s instead of strings and let's see the benchmarks. Type casting is expensive. One of the benefits of using the map data type is that you no longer are forced to use strings as your keys.

Share this post


Link to post
Share on other sites

So for my usual use cases I do want the keys to be strings - the type cast is just a means to generate a random string. Its also the same in both versions of the benchmark so cannot on its own explain why the map is slower. A better benchmark would probably be to pre-calculate the keys and then use the same keys for both map and variant attribute.

Share this post


Link to post
Share on other sites

Variant attributes were optimized quite a bit over the years. For string-based keys, they are very efficient, especially if you don't need to cast to/from variant for the values. 

Share this post


Link to post
Share on other sites

I would always recommend wiring up the output of the read nodes to an array indicator (outside the sequence so that it has no effect on timing).  Compiler optimisations can do weird things when you're not actually wiring up certain outputs.  For example, you're not using the data output of the variant but you are of the map. Not saying it explains the differences, but I've seen things like that wildly affect performance in the past.

I would look at the code, but I'm currently trying to get 2019 installed in a VM.

  • Like 1

Share this post


Link to post
Share on other sites

Side note: I tried to look at the code, but I couldn't manage to get the snippets dragged into a LV 2019 diagram. Is there something special I have to do to download the .pngs so that they have the LV meta data necessary to make the snippet be droppable code?

Share this post


Link to post
Share on other sites
53 minutes ago, Darren said:

Side note: I tried to look at the code, but I couldn't manage to get the snippets dragged into a LV 2019 diagram. Is there something special I have to do to download the .pngs so that they have the LV meta data necessary to make the snippet be droppable code?

Sorery dunno what has happened there - I created the snippets in LV 2019 64bit, saved as aPNG files and dragged it ontot he post in Chrome - perhaps somewhere between Chrome, and LAVAG it stripped the embedded LabVIEW code?

From your comments, I think the answer is simply that variant attributes are in fact quite fast and at least for string keys is the way to go for now.

Share this post


Link to post
Share on other sites

LAVA has been known to strip out that meta data in the past.  I remember one instance where it was seen when the image wouldn't fully fit on the post and it would do some scaling on the server side.  I'll ping Michael but until then I'd recommend attaching the VI.

Share this post


Link to post
Share on other sites

I had a go at recreating the benchmark (but with preallocated string keys), and maps did perform slightly worse than variants, but still within about 5% of each other. Flipping the map key type over from string to U64 increased the map performance by ~10%.

You may be seeing the sorting overhead of maps (and sets). AFAIK variant attributes are unsorted, whereas key/value pairs in maps are sorted on insert/delete. This exchange on Twitter has some more info:

aq_sets_rationale.PNG.04911af33ed3f1859d5e3643914a1d61.PNG

Share this post


Link to post
Share on other sites

The ~5% slower Map vs Variant Attributes is consistent with what @altenbach reported in his NI Week presentation, for very large datasets.

Since one set is ordered, the other is not, it might be interesting to benchmark the "delete" operation and see if it is symmetrical. My intuition here would be that finding the key and deleting it would be ~5% faster in favor of Maps (for large datasets).

Since I rarely deal with large sets, I'm egotistically happy with NI's choice to make those sets and maps ordered.

Share this post


Link to post
Share on other sites

Altenbach has done numerous benchmarks on the NI forums and found that maps perform 2% - 3% better than variant attributes in one particular use-case: https://forums.ni.com/t5/LabVIEW/LV2019-Maps-vs-Variant-Attributes-Performance/m-p/3934796#M1118297

He'll publish his code in a few days.

Share this post


Link to post
Share on other sites
4 hours ago, Antoine Chalons said:

Anyone knows if there is a way to do a RegExp search on a maps?

There doesn't appear to be a native way to do this, but if you write a VIM for this then you should be able to reuse that on any map type.

  • Like 1

Share this post


Link to post
Share on other sites

Great idea!

Although I attended your NIWeek presentation in 2017 VIMs are still not a reflex for me... I need to dig in!

Share this post


Link to post
Share on other sites

I'd be interested to see a comparison with SQLite (Altenbach didn't post his code). Those times seem really slow for the number of entries.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.