Ben Leedom

@smithd, I updated the nipkg attached to the v0.1.0 release (at https://github.com/ni/rebar/releases/tag/v0.1.0-alpha2) so that it should install with NIPM 18.5.1. Please try it out if you get a chance and let me know if it doesn't install for you.

Rebar will take yet another cue from Rust in the C interop case. Rust has the notion of an unsafe context, which can be either an entire function or a block within a function, and within which you can do certain actions that would be not be permitted otherwise, like using raw pointers or calling any kind of external code. The language basically says, "At this point you assert that you know what you're doing; I will still enforce some rules, but it's up to you to ensure that this C DLL isn't going to violate data safety." (You can read more about unsafe Rust here.) Once interop with arbitrary C DLLs becomes important, I'd expect to add a similar kind of unsafe context--probably a special structure that marks off the unsafe area from the rest of the surrounding code--plus the ability to use calls targeting NXG's Shared Library Interface (SLI) document. That way, you can pass pointers into C functions without performing extra data copies, as long as you pinky-swear that you know what you're doing. 😀

41 minutes ago, smithd said:

How does this work together with the existing system?

This is a very good question, and while I will try to answer, at the moment I have mostly a hazy vision that I have not fully worked out the details of. My priority is and has been to elaborate a good design for the core of Rebar, then create enough of an implementation of it that you can do some useful things in it and can see a path towards other useful things. How much the design and implementation of Rebar/VI interop gets worked out will depend very much on how much demand there is for it.

A VI calling a Rebar function should look about the same as calling a subVI. A Rebar function's signature will contain information equivalent to the inplaceness information computed by the VI compiler for a VI, so the VI compiler will know when it need to copy the data that it sends to Rebar. Similarly, a Rebar function should be able to call a VI; the Rebar compiler may need to dig out inplaceness information for the VI that is normally invisible to the user. One difference in this direction would be that Rebar should be explicit that it is obtaining a specific clone of the VI and calling that, which it might wrap up into a closure-like object.

So then the question is what kind of data you can pass back and forth between Rebar and VI. The rules here are that Rebar cannot pass any raw values to VI that VI will not guarantee the invariants of. Specifically:

• Rebar can't pass its own references, or anything containing its references to VI, since VI won't do lifetime checking.
• Rebar can't pass values that have destructors, because VI isn't guaranteed to call them when appropriate.

For any types that Rebar cannot pass raw to VI, it must wrap them in refnums. This amounts to having a reference-counted shared object between Rebar and VI, so there are still some Rebar types that wouldn't qualify, but it should be enough to allow the most interesting Rebar-created values to VI and have the runtime maintain their invariants. In this way, you could define a TCP connection type, a file handle type, an IMAQ image type, or whatever you want in Rebar, and provide an API for it back to VI with refnums. This would have the nice result of allowing you to re-implement many parts of the LabVIEW runtime in Rebar.

That's about as far as I've got on interoperability. Obviously the devil is very much in the details here, and often creating an interop system between two different languages is way harder than designing each one in isolation. Like I said above, though, none of this matters much unless Rebar is interesting enough that people want to use the two side-by-side.

1 hour ago, smithd said:

The local variable lifetime is kind of my point -- its life isn't really limited because the data space for a reentrant VI remains live for the duration of an application. Similarly, I was under the impression the shift register value is also permanent. As an example, I seem to recall a while back running a loop that allocates a giant (>>1mb) array on a shift register but never wire it out of the VI. if you do that, the large array remains part of the VI's data usage. From the perspective of the details you mentioned, yes, that giant array is on the heap somewhere, but all that data is still live until something overwrites it, making that data location permanent as well, even if initialized.

I think it's important to distinguish between what the language semantics are and what the compiled code and runtime actually do. For an initialized shift register on a loop, its value for a particular execution of that loop is only accessible during and immediately after the loop; the language does not let you get that value at any other point, and the value will be overwritten when it is initialized on the next loop execution. (NB: I'm not talking about individual iterations of the loop, but the execution of the loop over all iterations.) The compiled VI does store the shift register value in its dataspace, so yes, that value will be retained in the dataspace between loop executions, but from the language's perspective this doesn't matter because it's inaccessible. The language does not require the value to be retained or its allocation to be reused between loop executions; that's an optimization that the runtime provides.

1 hour ago, smithd said:

Good explanation, but for the refnum point I keep coming back around to -- yes, this is something I absolutely would love to see improved and resolved, but it seems like a bandaid over an underlying flaw in how labview does dataflow.

I'm not sure if we're agreeing or disagreeing here. My claim is that the way LabVIEW does dataflow--meaning wires carry immutable values that can be used in any number of concurrent places downstream--means that refnums are probably the best way of referencing objects that is available to LabVIEW. (I'm not enough of a theoretician to prove that reference systems with better properties aren't possible, but I do trust that if there were a better way, somebody smart around here would have discovered it in the last 30 years or so. However, another hint in this direction is the fact that channel wires have properties that approach what I'm proposing with Rebar, and channel wires are fundamentally different from normal wires.) If you want a reference system with better properties, I believe you must fundamentally change the way LabVIEW does dataflow. And so, to your next point:

1 hour ago, smithd said:

Now thats something I can definitely get behind. If NXG can have "gvi"s or "good, baggage-free VIs without depending on the design decisions from 3 decades ago that might not have aged as well" I might switch sooner
But in all seriousness, it seems like you've made your own type of VI with compiler rules associated with it. Is there a way to expand that more broadly, making life simpler enough for the compiler (per your points above) that it can figure out the lifetime of objects? IE in order to reach the goal of stateless functions, but without relying on the user to manage lifetimes?

Yes, this is a new type of VI, or a new Model of Computation, or a new set of compiler rules, or whatever you want to call it--though I'd prefer you just call it a function, to distinguish it from what a VI is, which is more than a function. Importantly, it is not merely an extension of VIs, because it works differently enough that I do not think it should be mixed with G on the same diagram. Interoperating with VIs is a different story, and I think there is an interesting set of possibilities there.

But I'm not sure I understand your question, "is there way to expand that more broadly?" I think to make life simpler for the compiler, you have to change the language, like I said above--maybe that answers it? Also, rather than "relying on the user to manage lifetimes," I would state it as "providing a set of safe and complete rules for lifetimes and guiding the user into writing code that follows the rules."

4 hours ago, smithd said:

All of the below is without having installed it. I tried, but it requires NI package manager 19. NIPM 19 is not released as far as I can tell (I downloaded the latest, 18.5.1, from ni.com and when it loads it doesn't seem able to reach the server). More to the point, why does NIPM continuously have this issue where it requires a specific version to unpack the files? I honestly don't think I've ever successfully installed a package with NIPM on the first go <_<

Apologies for that! I failed to realize that NIPM 19 isn't publically available yet. I will try to spend some time creating a package that is compatible with NIPM 18.5.1 soon.

And yes, I share your frustration with NIPM versioning. I go long enough without having to fiddle with NIPM on my dev machine that when I do I wind up reinstalling it from scratch almost every time.

Quote

To start, I think you need to get more technical on the overview. For example, you say things like local variables and shift registers are "places" and that "places" have lifetimes...what is the lifetime of a local variable or a shift register? The explanation of how in rebar the sets of wires correspond to a single 'place' which...sounds just like a standard labview buffer allocation. My point is that you're making something for the type of person who might say "oh neat a rust+labview mashup!!" but then have a 1000ft explanation of how it works 👻

Fair enough! Over the last week or so I've been trying to get things into a presentable state, but I agree that there is a need for a lot more documentation and explanation. The overview you've been quoting was indeed intended to be a high-level summary of why one might want to use this new thing that looks superficially like the old thing. I hope to spend more time soon adding more in-depth content to the wiki on github. But you're asking the technical questions, so let's get technical.

I had a draft of the overview that listed the specific lifetimes of places, but then removed it. A local variable's lifetime is bounded by that of its containing VI's clone, because it is stored in the VI's dataspace--that is, it persists not just for the entirety of calls to the VI but also across calls. A shift register's lifetime is bounded by the loop where it is defined, unless it is uninitialized, in which case its lifetime is like that of a local variable's--it persists across calls.

I was using the word "place" to avoid using the word "variable," since that has specific and different meaning in G. What I mean is much closer to a local variable or a member variable in a text-based language like C or Java, something that can be defined and then reassigned to. You can reassign values to shift registers or local variables (places) in G by wiring inputs to them, but the language doesn't express directly reassigning to the buffer for a wire (a value); that's the difference.

Quote

This is just a nitpick, but labview already makes data copies explicit. What it doesn't make explicit is optimizations that remove the need for copies .

Also, they aren't copy dots, they are buffer allocations...

Okay, they're not copy dots. "Buffer allocations" is not a precise enough term in my mind, either. VIs allocate data in basically two ways: the dataspace for each VI clone, and heap buffers. Each clone of a VI gets the same size dataspace, so when an object's size can be known at compile time, it generally gets put entirely in the dataspace, and otherwise its dataspace presence is a handle to a heap buffer--the latter includes arrays, strings, paths, value classes, and variants. Substrings and subarrays are handles in the dataspace separate from the main array or string handle but pointing into the same data. As Stephen's post says, LabVIEW does not distinguish between places where it just adds more to the dataspace and where it allocates both in the dataspace and the heap. Moreover, just because LabVIEW shows a buffer being allocated for an array at a particular place, does not mean that that place may not reuse a heap buffer on its second execution that it obtained on its first execution.

Quote

Its obvious that LabVIEW knows what to do with a visa refnum (and file refnums, and any other refnum I can think of) when the application closes...which I think is a positive for LabVIEW, so I'm not sure how often its actually valuable to define cleanup (especially given it sounds like the cleanup is off diagram). That having been said, I can think of one super exciting use for this which is hardware access via a dll. If this tool lets you say "this pointer sized int is a pointer and when the program exits call this function" thats pretty awesome. And much better than the unreserve callback nonsense in the CLFN.

"Hardware access via a DLL" is not altogether different from what a VISA refnum, or any other refnum, does. Refnum types, whether implemented internally in the runtime or with plugins, are basically bundles of pointers to DLL functions, with special functions to use for resource acquisition and cleanup (not unlike the CLFN unreserve callback nonsense).

But yes, part of the point of this is to allow you, the graphical programmer, to define types that wrap pointers and DLL functions, or any other sort of "resource," and have them clean themselves up automatically. And we can do one better than "when the program exits": we can clean up the object at the precise time when we know it is no longer used, which gets to your next question:

Quote

To me the problem with the existing system is that the refnums point at global things. For example, if I allocate a queue inside of A.vi, and I have no terminal which outputs the value of the queue refnum, if A.vi finished executing the compiler SHOULD kill the queue.....except that queues are global. I'd much much much much rather have the compiler figure out the lifetime of the queue for me and kill it when the wire can't go anywhere else. It seems clear that this addon allows for that, which is awesome, but why do we need an addon? I thought the whole point of dataflow was that the compiler knew about the lifetime of data and could make intelligent decisions based on that?

The compiler does try to determine the lifetime and usage of data and make decisions about when to copy it based on that. There are several factors that confound the compiler's analysis and force it to be conservative:

• The compiler tracks signature information for each VI about how its outputs and inputs are in-place to each other. Dynamic dispatch calls represent calling one of many possible signatures, not all of which may be known at compile time (in the case of dynamic loading), so those calls generally have to be conservative with copying data.
• Calling a VI by reference asynchronously also requires being conservative, because you can no longer directly tie when a call begins to when it ends.
• Refnums are particularly bad for determining lifetime, because you can copy the refnums themselves anywhere--uninitialized shift registers, global variables, queues, DVRs, fields of derived classes stored in base class wires, variants--such that it is impossible to determine statically how many different places might all point to the same object. The runtime does not increment a reference count every time it copies a refnum, so it has to be extremely conservative about when an object can be freed--it's pretty much either, "the user just called Close Reference on this" or "the program's going away now." It's not just that some refnum types let you globally access instances by name, it's more that refnums are all the concurrency-unsafe baggage of pointers without the advantage of referring to a literal memory address.

Quote

I don't know enough about iterators and generators, but why do you think anonymous functions are aided by this? To be honest its not clear to me why anonymous functions aren't possible right now, nor closures.

To be clear, I do think anonymous functions are possible in G today. Actually, if you get right down to it, a VI is pretty much what closures are in other languages, in terms of retaining state across calls. I think it's likely that anonymous functions would have to be presented as and implemented as nearly identical to VIs, which means providing configuration options for things like reentrancy.

One of the goals of Rebar is to provide a basic function document that retains absolutely no state across calls, and thus can completely dispense with the notion of reentrancy as a property of a function. That in turn simplifies the notion of defining an anonymous function.

Quote

I wanted to finish up with an application-specific question. One of the big use cases for avoiding copies is image processing, but unfortunately imaq images are not only global, but they are not reference counted like named queues (ie worst of both worlds). So if, for example, you are an image acquisition and you want to share your latest snap with all your closest friends (the logging loop, the network loop, the processing loop, the display loop) its difficult to share the references in such a way that when all the other loops are done, the image is freed. If it were a queue, you could obtain several named references to the same queue and the queue only goes away when all the references are dead. I ended up implementing this myself with a DVR, but it still requires that users call the 'release' function -- ideally, and what I think rebar brings to the table, is an automatic release. Does this rebar system solve that? If so, sweet. You should share it with the imaq group

There are two ways that Rebar might let you safely share an image between multiple places like this. If you want to restrict all the loops to running in lockstep (i.e., each one finishes working with one image before any starts on the next), then you can pass simple references (pointers) to each loop. If that's too restrictive, then the language will provide a basic reference-counted wrapper around any type:

• Cloning the wrapper automatically increments the refcount, and the clones can be sent anywhere;
• When the wrapper goes out of scope, it automatically decrements the refcount;
• When the refcount goes to zero, it destroys the underlying object;
• All of the reference counting is done atomically, so that it's safe to use in parallel.

Thanks for your questions! Let me know if you have any others, or if I need to explain anything better.

Cross-posting from the ni.com LabVIEW forum:

Hello! This is a brief announcement that there's a new community here:

https://forums.ni.com/t5/Rust-and-G/gp-p/5381

I started this to discuss a project called Rebar, which is an experiment in combining semantics of the Rust language (https://www.rust-lang.org/) with G syntax, implemented as an addon for LabVIEW NXG. You can read more about it and find a link to install the addon over at the community. Please take a look!