Secondary Project Provider for ByRef Architecture Scaffolding

[UnlikelyNomad]

INB4 Demo Video: YubTub Linky

Monday of this week I finally saw NI's Center of Excellence and after not being at NIWeek for a few years I was blown away at how far the technical sessions have come. I saw the LV Hooks presentation and of course had to try getting back into the rusty nails that I started trying to play with a year ago. I previously tried creating XNodes that could replace getters and setters for data hidden behind DVRs however I hit a road black when trying to access DVRs that were private to a class. I have finally gotten around to implementing my next approach, and it works!

This is the first of a set of features to help me speed up development when I'm creating ByRef classes. My ByRef architecture involves creating a DVR and usually having that be the only data member of the class private data cluster. This allows me to branch the class wire while still accessing the same DVR contents in multiple locations. This allows me to avoid having to use additional frameworks and patterns like QMH or AF to easily support parallel tasks. This design lets you choose between synchronous and asynchronous actions and most importantly: inheritance and dynamic dispatch are still fully supported! Want to use a queue? Do it! Want to use a bool to shutdown a background loop? Sure thing! Want to use events! Go right ahead! Want to mix ByRef and ByValue classes? Okay! I know there are already project provides that try to automate some ByRef implementation details however I don't like the idea of functionality being hidden in some "core" and wanting to do something slightly differently this time. The basic idea of everything that will come of this provider is that it automates creating the VIs you would use directly, and only those VIs. Everything is visible and available and you can use the provider to scaffold the common starting points and then move on from there.

 

I'm curious what y'all think! Does anyone do similar ByRef designs? Anyone have ideas or suggestions? Once I get the DVR member accessor scripting implemented I'll uploading this to GitHub for public critique consumption.

[G-CODE]

There's this reference design framework, but as far as I know it doesn't have scripting to create accessor methods.

https://github.com/JKISoftware/JKI-State-Machine-Objects

[UnlikelyNomad]

34 minutes ago, planet581g said:

There's this reference design framework, but as far as I know it doesn't have scripting to create accessor methods.

https://github.com/JKISoftware/JKI-State-Machine-Objects

I rarely use state machines in my applications so I've never been inclined to look into that.  After seeing that readme I'm definitely going to take a look at it and see what all it does though. [Mass compiling now in VIPM]

I suffer from being very picky in my ability to be flexible and tend to avoid [heavier] frameworks when I can. The main idea of this is that it isn't a framework; it is IDE tooling to more easily develop ByRef designs

[UnlikelyNomad]

That DAQmx template is slick. Definitely not a state machine and definitely already handles some of the features I was thinking of like templating asynchronous background tasks.

[UnlikelyNomad]

I'm curious on the effort done to cast DVRs back and forth between an int.Googling doesn't seem to bring anything up so I'm curious how this became a thing and what benefits it provides? Does it stop LabVIEW from doing some tricky thing with DVRS? Does it make the scripting simpler?

[smithd]

25 minutes ago, UnlikelyNomad said:

I'm curious on the effort done to cast DVRs back and forth between an int.Googling doesn't seem to bring anything up so I'm curious how this became a thing and what benefits it provides? Does it stop LabVIEW from doing some tricky thing with DVRS? Does it make the scripting simpler?

Don't do this, it can crash labview. If you need to cast, I think a variant will at least be safer (var to data should return a NaRefnum if its the wrong type, while cast to and from int will have labview attempt to dereference a random pointer you told it was good). There may be guards against it, but its not worth it.

[smithd]

14 hours ago, UnlikelyNomad said:

I'm curious what y'all think! Does anyone do similar ByRef designs? Anyone have ideas or suggestions? Once I get the DVR member accessor scripting implemented I'll uploading this to GitHub for public critique consumption.

As a general rule I prefer this style to the DVR-of-a-class (eg session framework: https://forums.ni.com/t5/Reference-Design-Content/Extensible-Session-Framework-ESF-Library/ta-p/3522019)

The argument against it, which is also very valid, is that its hard to indicate that your class has by-ref features because by definition that dvr inside is private. So they must operate as if your class is by-value unless they know the class and all children have no by-value components. You could make the same argument about sticking a visa reference in your class -- its a reference, but you wouldn't go calling your class a by-reference class.

Using a DVR-wrapped class and the malleable VI features added in 2017sp1 would seem to be a nice route. You can template the DVR unwrap and define an interface which the malleable VI will adapt for you.

For context to the below post, I previously commented that I'd prefer a tools menu item because the project provider scares me, but I looked around and theres a lot of examples out there so I decided to remove it...at the same time it was being replied to.

Edited October 19, 2018 by smithd

[UnlikelyNomad]

I could convert this to a Tools item. Make it an either/or. I definitely like the ability to interact with the item itself. I've come back to this first paragraph several times to take out reasons why I'm less favorable to the tools menu; maybe that'll be next week's side project.

What I am avoiding is conditional menus based on current state of an item; you use it right or you don't. The actions to check compatibility but it's after you select it and not while you're waiting to see if LabVIEW is going to render the popup menu correctly. I was toying with the idea of storing state in the class (user tags) or assessing the state but I'd like to maintain portability (no deps or conditional/extra class state) and everything in native LabVIEW. The tools menu does seem to be context sensitive based on project selection but that feels like a weak link to me. I think if I did a tools menu entry I'd want a UI to confirm class selection, and then I might was well make different actions available from that UI, and then suddenly GOOP or AF.

I'm avoiding DVR wrapped classes (versus my DVR wrapped in a class) because unless I'm missing something... dynamic dispatch doesn't work with that mechanism. I'd much rather either continue doing this manually or create some tooling to significantly cut down on the bootstrapping effort.

[G-CODE]

9 hours ago, smithd said:

Don't do this, it can crash labview.

Can you show me how to make LabVIEW crash by doing this? I created and destroyed 1 million DVRs that were cast to integer and cast back to DVR and couldn't get LabVIEW to crash. Attached VI made with LabVIEW 2013.

 

DVR to integer_2013.vi

[smithd]

10 hours ago, UnlikelyNomad said:

I'm avoiding DVR wrapped classes (versus my DVR wrapped in a class) because unless I'm missing something... dynamic dispatch doesn't work with that mechanism. I'd much rather either continue doing this manually or create some tooling to significantly cut down on the bootstrapping effort.

DD doesn't work directly on a DVR (except property nodes), hence my malleable VI suggestion. Of course malleables just make it easier to call -- you still need to have the method in the class, so your tool would still be useful.

1 hour ago, planet581g said:

Can you show me how to make LabVIEW crash by doing this? I created and destroyed 1 million DVRs that were cast to integer and cast back to DVR and couldn't get LabVIEW to crash. Attached VI made with LabVIEW 2013.

To clarify, it can crash labview if you get it wrong and uncast to the wrong type.  Or so I'm told. I've never attempted to do this because a variant is a type-safe way of accomplishing the same task.

Edited October 19, 2018 by smithd

[UnlikelyNomad]

1 hour ago, planet581g said:

Can you show me how to make LabVIEW crash by doing this? I created and destroyed 1 million DVRs that were cast to integer and cast back to DVR and couldn't get LabVIEW to crash. Attached VI made with LabVIEW 2013.

 

DVR to integer_2013.vi

Still unsure why this is a thing though. Duck typing? Union like tricks? Seems like this should be used in very specialized instances and not to transport data around. Even in C++ pointers have type and this completely erases type safety.

[smithd]

9 hours ago, smithd said:

To clarify, it can crash labview if you get it wrong and uncast to the wrong type.  Or so I'm told. I've never attempted to do this because a variant is a type-safe way of accomplishing the same task.

I tried this out and can confirm that accidentally swapping types has the following behavior:

dbl->dbl = OK

string->dbl = always displays the prior value of the DVR -- probably something bizarre to do with the fact that I started with dbl-dbl and its in debug so the memory space stayed the same. No error though, and the value didn't update as I changed the dvr (string) value.

int(0)->str=OK

int(positive number)->str=crash

 

[Rolf Kalbermatter]

On 10/19/2018 at 5:11 PM, planet581g said:

Can you show me how to make LabVIEW crash by doing this? I created and destroyed 1 million DVRs that were cast to integer and cast back to DVR and couldn't get LabVIEW to crash. Attached VI made with LabVIEW 2013.

 

DVR to integer_2013.vi

It does not crash if you typecast the integer to the right refnum (be it DVR or any other LabVIEW refnum), But it can badly crash if you ever convert DVR_A into an int and then try to convert it back into DVR_B or any other LabVIEW refnum. The int typecast removes any type safety and LabVIEW has no foolproof way to detect in the Typecast to Refnum conversion that everything is alright. LabVIEW does have some safety built in there such that part of the uint32 value of any refnum is reserved as a random identifier for the refnum type pool it resides in (so a file refnum has another bit pattern for those bits than a network refnum) but it can not verify safely that the int coming from a DVR_X is fully compatible to the DVR_Y you want to convert it to. And then as soon as it attempts to access the internals from DVR_Y according to its datatype it badly reaches into nirvana.

So if you do the typecasting all safely behind the private interface of a class so that nobody else can ever meddle with it AND you are very very careful when developing your class to never ever typecast the wrong integer into a specific DVR (or Queue, Notifier, Semaphore, etc) you are fine, but there is no LabVIEW holding your hand here to prevent you from shooting in your own feet!

The original idea comes from before there where DVRs and LabVIEW GOOP. In LabVIEW 6 or 7 someone developed a first version of LVOOP, which used a private "refnum" repository implemented in the LabVIEW kernel and accessed through private VIs using a Call Library Node. It was basically like a DVR that was also the class wire. Obvriously DD was not something that would work seemlessly but had to be programmed explicitedly by the class developer and hence was seldom used. The pointer (which was back then an int32 since all LabVIEW versions were 32 bit only) was then typecast into a datalog file refnum with a specific enum type for typesafety so that you had a typesafe refnum wire to carry around as class wire.

Some of the DVR based class design (reference based classes) in the Endevo/NI GOOP Framework still reminds of this original idea, but now using proper LabVIEW classes as the class wire. Additional bonus is that this buys them automatic dynamic dispatch capability.

Edited October 20, 2018 by rolfk

[Francois Normandin]

On 19/10/2018 at 0:49 AM, UnlikelyNomad said:

I'm curious on the effort done to cast DVRs back and forth between an int.Googling doesn't seem to bring anything up so I'm curious how this became a thing and what benefits it provides? Does it stop LabVIEW from doing some tricky thing with DVRS? Does it make the scripting simpler?

The reason for this typecast is to break type propagation that plagues edit-time performance which occur when you have a large class hierarchy. It is not because it makes scripting easier (on the contrary). In essence, when you edit a class' private member, the class' mutation history is modified and the LabVIEW IDE goes through a series of checks and recompiles to propagate the changes. For very small projects, this is completely trivial and does not affect edit-time performance, but for larger hierarchies (think Actor framework...) it becomes a real issue. One way to reduce this performance lag is to never put typedefs in the class private data... but this is up to the developer to stick to that rule or not. When you use an integer, using typedefs does not affect the edit time performances at all.

As Rolf mentions, this pattern was popularized by GOOP (now OpenGDS).

When creating a framework, you can never predict how the developers will use it... Obviously, if you use a framework and it starts to lag as your project's complexity increases, you are quickly going to blame the framework for the performance degradation (just like we blame Windows for running slower after we've bloated our computers with a bunch of apps, but that's another topic...). So it is very tempting to use tricks like this typecast to an integer to break this slow type propagation.

As was mentioned by Smithd, it is risky... but if it stays in the private methods of your class, it is totally safe.

In JKI SMO, you'll notice that this is used through two private methods and the DVR type is typedef'ed. You can very well create SMOs that do not use this trick. 

Edited October 20, 2018 by Francois Normandin

[Francois Normandin]

On 19/10/2018 at 0:12 AM, UnlikelyNomad said:

That DAQmx template is slick. Definitely not a state machine and definitely already handles some of the features I was thinking of like templating asynchronous background tasks.

Indeed, not a State Machine... "JKI State Machine Objects" gets its name from the use of "JKI State Machine" as its core "Process.vi", which is not a state machine. It is an Event-driven Queued Message Handler (QMH). State Machine was a misnomer and the name stuck around as it was (still is) a very popular template.

Edited October 20, 2018 by Francois Normandin

[UnlikelyNomad]

Francois, thanks for the elucidation! I don't use frameworks like AF so I haven't run into too much hierarchy bloat and I tend to compile lower level libraries into PPLs to be used by higher levels; I'm thinking that limits how much LabVIEW can check for during type propagation since I only have one editable component open at one point.

Since I'm doing ByRef I often do have to break type safety and store some relationships as base objects and then always cast them when pulling them out of the DVR for use (since you can't have a DVR reference another DVR that references the first) so this may also cause a similar optimization. IE: a server keeping a list of open connections and those connections keeping a reference to their originating server.

[smithd]

6 hours ago, Francois Normandin said:

The reason for this typecast is to break type propagation that plagues edit-time performance which occur when you have a large class hierarchy. It is not because it makes scripting easier (on the contrary). In essence, when you edit a class' private member, the class' mutation history is modified and the LabVIEW IDE goes through a series of checks and recompiles to propagate the changes. For very small projects, this is completely trivial and does not affect edit-time performance, but for larger hierarchies (think Actor framework...) it becomes a real issue. One way to reduce this performance lag is to never put typedefs in the class private data... but this is up to the developer to stick to that rule or not. When you use an integer, using typedefs does not affect the edit time performances at all.

Never heard of this, but makes sense. I do the same (using variants of course) when it comes to static VI refs (eg 'call and forget') because any change to a type def in the connector pane breaks controls/indicators -- type prop doesn't update them. So I have my create reference in one subVI that spits out a variant and then everywhere I want to launch an instance I just have another copy of the static ref and a variant-to-data function which casts it back. Avoids annoying labview bugs that way.

Edited October 20, 2018 by smithd

[UnlikelyNomad]

17 minutes ago, smithd said:

Never heard of this, but makes sense. I do the same (using variants of course) when it comes to static VI refs (eg 'call and forget') because any change to a type def in the connector pane breaks controls/indicators -- type prop doesn't update them. So I have my create reference in one subVI that spits out a variant and then everywhere I want to launch an instance I just have another copy of the static ref and a variant-to-data function which casts it back. Avoids annoying labview bugs that way.

 

Any chance you have an example of this? Not sure I've ever run into something like this before. I update typedefs all the time and haven't had my start async calls get boogered up. Though I do currently have NI pursuing a bug where a typedef isn't getting namespaced properly when compiling to a PPL and thus breaks all the calls once compiled because the static ref still specified SomeLib.lvlib:SomeClass.lvclass:SomeType.ctl instead of SomeLib.lvlibP:SomeClass.lvclass:SomeType.ctl which of course I can't recreate in dependent components. As long as I'm testing the uncompiled component with the static ref, everything updates as expected for me.

[smithd]

2 hours ago, UnlikelyNomad said:

 

Any chance you have an example of this? Not sure I've ever run into something like this before

Probably did not explain it right, but attached is an example. Open main.vi, open the type def and add a field to it (eg a 2nd string). Save and close the type def and the code breaks because while the VI ref is tied to the type def, the controls appear to be tied to the version of the type def at the time of their creation -- eg, type prop fails. The solution for bigger projects is to type def the control or wrap it into a class or whatever, and then you only manually update it in one place, but I find that pretty annoying as well so I do the variant form (terriblesolution.zip) as this requires zero manual updates. On a related topic, I would have no objection to someone coming here and saying "you're an idiot, you can make it update if you just do X" but I haven't found any X so far.

bug.zip

terriblesolution.zip

[UnlikelyNomad]

Making progress on the accessor! Just need to make the two templates now and script up the template editing.

[UnlikelyNomad]

15 hours ago, smithd said:

Never heard of this, but makes sense. I do the same (using variants of course) when it comes to static VI refs (eg 'call and forget') because any change to a type def in the connector pane breaks controls/indicators -- type prop doesn't update them. So I have my create reference in one subVI that spits out a variant and then everywhere I want to launch an instance I just have another copy of the static ref and a variant-to-data function which casts it back. Avoids annoying labview bugs that way.

Ohhhh I gotcha now. This I do see. I'd say this isn't a bug and is consistent behavior. Change the data type feeding into a string indicator and you get a broken wire instead of a new indicator. Type defs are the only thing that propagate updates so without a type def to update you would break data types to controls/indicators.

[smithd]

Except that in this case we're talking about a reference which refers to a type def. Saying this is consistent behavior would be like saying "its OK for any queue refnum that references a type def to break whenever the type def changes" -- something that obviously does not happen.

[UnlikelyNomad]

Oh yeah, queue terminals do update. So inconsistent behavior.

[styrum]

In his "Don’t Wait for LabVIEW R&D…Implement Your Own LabVIEW Features!" presentation Darren Nattinger advised to not mess with project providers. So, he didn't give any links to anything on this subject. How did you figure out how to do it? Please provide links/code on how to make your own project provider. Is this the best place to start on this topic?

https://forums.ni.com/t5/LabVIEW-Project-Providers/gp-p/bymqyodmkc?profile.language=en

Any  advice/help/examples you can offer from your own experience making your own project provider?

Edited April 26, 2019 by styrum
Found more info myself