Are Events Lossless?

[AlexA]

I couldn't find a clear answer to this while googling, so:

Are Labview Events lossless like a queue, or are they lossy like notifiers?

 

Further to that question, has anyone ever implemented a process responsible for File IO that gets its data via Events (I.e. subscribes to an event)? Especially, big data such as a continuous image stream?

 

Finally, has anyone ever used Events with a DVR? The very idea of it kind of doesn't make sense to me as you're broadcasting a locking resource, but I thought I'd inquire anyway.

 

Thanks in advance,

Alex

[hooovahh]

Events are Queues from everything I've heard, loss less and all.  Now there are some interesting issues that can be seen, like if you generate an event, before being subscribed for it, then that event will be lost.  Jack had a neat presentation at NI Week 2013 that demonstrated a few memory issues and pitfalls but I can't seem to find it at the moment.

 

I have used User Events for messaging between actors with large amounts of data and I've not seen any issues other than the fact that copies need to be made under certain conditions.  File I/O, high speed DAQ, real-time FIFO, FPGA FIFO, communication data, sequence control, UI updating data, no problem with the transport layer it self.  You can shoot your self in the foot but that isn't really the User Events fault.

 

I've never used a DVR with a User Event, but I see no reason it wouldn't work.

[mje]

To be pedantic, there is nothing lossless or lossy about events. Events by themselves are only a means of tracking subscribers. It is the registration refnum that wraps the event queue, and yes is lossless.

[AlexA]

Thanks for the info guys!

[mje]

From the context of your question I got the impression you were asking about user events and dynamic registration.

To be clear also realize that the lossless generalization falls apart for statically registered events (the ones configured directly in an event structure linked to a control). You can configure those to act via a FIFO buffer so they can be lossy.

[hooovahh]

To be clear also realize that the lossless generalization falls apart for statically registered events (the ones configured directly in an event structure linked to a control). You can configure those to act via a FIFO buffer so they can be lossy.

Yeah I guess the technical answer is the better one.  That is odd that dynamic events can't be limited like non-dynamic, to make them lossy.  You can invoke the Flush Event Queue which I've never used but I assume it...well flushes the events in the queue.  That could make it lossy sorta.  I mean you won't really be lossing any event, just ignoring them.

 

EDIT:  Oh 2013 and newer which is when new event stuff came around.

[ShaunR]

 You can configure those to act via a FIFO buffer so they can be lossy.

 

Can you elaborate on that a bit more? Lossy, how?

[hooovahh]

Can you elaborate on that a bit more? Lossy, how?

Again added in 2013.

 

http://zone.ni.com/reference/en-XX/help/371361K-01/lvdialog/edit_events_dialog_box/

 

http://zone.ni.com/reference/en-XX/help/371361K-01/lvupgrade/labview_features/

 

Very handy for handling things like window resize where 100 events can pile up quickly and you really just care about the last event so you limit it to 1.  Same with a scrollbar that is set to double.

[drjdpowell]

That is odd that dynamic events can't be limited like non-dynamic, to make them lossy.  

I wish they would extend this to User Events.  Last-only User Events could be very useful.

[AlexA]

@Hooovahh

 

I've been thinking about this, and it's not obvious to me how pushing a DVR through an Event makes any sense. Specifically, if multiple processes subscribe to a DVR event, who is responsible for destroying the DVR? My specific use case is as follows:

I have a process producing high speed image data via DVR.

I have a UI which needs relatively slow updates (25-30 FPS, from Cam Data at 100 FPS+)

I have a File IO process which needs to get all that data.

 

To me, the File IO is the obvious place to delete the DVR as its the only process gauranteed to receive all the data (at least in my current architecture). If both processes subscribe to the event, it's possible that the File IO process clears the DVR before the UI has a chance to access it resulting in an error at the UI. Do you just handle that error and ignore it? I guess that's one way I could do things.

[AlexA]

Ok, further question:

 

If I have a buttload of events, and I register them as a cluster, it doesn't seem to be possible to re-register specific elements of that cluster? Here's what I'm doing currently but it doesn't work:

 

There's no way that I can see to make the Event Registration terminal accept a sub-element of a cluster.

 

Attempting to re-register as I am in the picture will destroy the other event references in that cluster (as you would expect).

Is this just not possible, or am I missing something obvious?

[mje]

Interesting problem, AlexA. I've only ever done that with individual events, that is ones which are not clustered. I don't know if you can pick off an individual cluster element to register, I've definitely never thought to try.

[Neil Pate]

I think it can be done, but rather than doing a Reg Event, just replace the actual cluster element.

 

Note: this is quite old code I wrote some time ago when I was just learning about User Events (LV 8.5), I am not sure I would do things like this now if I had to!

Edited October 19, 2014 by Neil Pate

[hooovahh]

Well my suggestion with who kills the DVR comes from who makes it?  Does any one actor create the DVR?  or is it created before the actors startup?  If it is created before the actors, then my suggestion is to destroy it after all actors stop.

 

As for your other issue with reregistering events.  In the past instead of having one event for a giant cluster that I knew I would need parts of.  Instead of registering them as a cluster, you can register them as an array, but only if all the data types are of the same type.  So I choose to make N user events, all of the data type Variant.  Then if you register for the array, any event within the array will cause that case in the event structure to be called.  Then you detect which of the user events were generated, and cast the data from a variant back to the type of that specific event.  A bit of a pain but with typed data is is quite easy to update and manage.  

[AlexA]

Good point, the DVR is generated by the "Acquire Read Region" method on a Target-to-host DMA FIFO from an FPGA. As I understand it, a new DVR is generated on each call to the method. This DVR must be destroyed so that the method can place data into that region the next time it cycles around to it.

 

The thing is, I can't think of a way for this producer to elegantly destroy the DVR. The key being, when does it make the decision to destroy the reference?

 

My thoughts are below:

The producer can't destroy the DVR on the same cycle it generates it so it must store it to be destroyed later.

The obvious storage structure is an array (circular buffer).

I guess you could have a destruction operation acting some indices ahead of the current, for arguments sake lets say 5. On the first cycle through, you get a whole bunch of errors pertaining to invalid refs, but you just squash them. Then the next time through you're destroying refs ahead of your current index (wrapping) so they're gauranteed to be free.

 

But, if the consumer (File IO) is slower than the producer, eventually you're going to catch up and destroy a reference that you're working on (writing to disk).

 

I guess this is a problem regardless of whether its a DVR or actual data, the symptom being a memory leak if its actual data.

 

I don't know if I'm missing some piece of information/subtlety with DVRs or with that particular method which would inform a cleaner solution?

[ShaunR]

Good point, the DVR is generated by the "Acquire Read Region" method on a Target-to-host DMA FIFO from an FPGA. As I understand it, a new DVR is generated on each call to the method. This DVR must be destroyed so that the method can place data into that region the next time it cycles around to it.

 

The thing is, I can't think of a way for this producer to elegantly destroy the DVR. The key being, when does it make the decision to destroy the reference?

 

My thoughts are below:

The producer can't destroy the DVR on the same cycle it generates it so it must store it to be destroyed later.

The obvious storage structure is an array (circular buffer).

I guess you could have a destruction operation acting some indices ahead of the current, for arguments sake lets say 5. On the first cycle through, you get a whole bunch of errors pertaining to invalid refs, but you just squash them. Then the next time through you're destroying refs ahead of your current index (wrapping) so they're gauranteed to be free.

 

But, if the consumer (File IO) is slower than the producer, eventually you're going to catch up and destroy a reference that you're working on (writing to disk).

 

I guess this is a problem regardless of whether its a DVR or actual data, the symptom being a memory leak if its actual data.

 

I don't know if I'm missing some piece of information/subtlety with DVRs or with that particular method which would inform a cleaner solution?

 

I haven't played with this yet, but from reading everything about this acquisition method it strikes me that it doesn't fit with an asynchronous buffering producer consumer architecture. It seems to be a synchronous method targeted at efficient memory management and high throughput..

 

It states in the docs:

 

"You must delete this external data value reference before the driver can write new data to the specified portion of the buffer."

 

I read that as you must destroy the ref in order for the acquisition to acquire the next block of data. i.e. if you stick it on a queue/fifo you won't get another acquire until you've popped it off the queue and destroyed it which defeats the object.

Edited October 20, 2014 by ShaunR

[hooovahh]

Ahh I guess that makes sense.  Any transport layer that relies on a reference, and is asynchronous, could have the potential problem of destroying the reference before all consumers have consumed the data.  I don't know how your application is structured, but would it be possible to create this reference before any actors start, and destroy it after all actors have been stopped.  That is if you must rely on a DVR.  

 

What was the reason you were interested in using DVRs in a user event?  Instead of just use the user event which is lossless (more or less) as we've mentioned.

[AlexA]

Hey guys,

 

The DVR choice was made a while ago due to resource constraints on the system. Specifically, I use the DVR with the "TDMS Advanced Asynchronous Write (Data Ref)" function. We've since moved to a bigger system but I'd obviously prefer to keep the execution tight if I can.

 

The way that "Acquire Read Region" method works, you can configure the FIFO buffer size to be any multiple of your data chunk-size (for me, 1024 x 1024 U8 pixels). The method then iterates over the available buffer generating a new DVR for each chunk. This DVR must be destroyed so that chunk can be used again. Things I'm not sure of:

Whether all the DVR's are assigned at start-time or generated on the fly?

 

Since the above post, I've implemented a simple Lossy queue circular-buffer in the producer. The lossy queue is "Size of FIFO Buffer (in chunks) minus one" when the Lossy Enqueue pushes a DVR onto the queue, I delete the DVR that comes off the end.

I push raw pixel data to the UI by copying the data out of the DVR and generating a pixel data event.

I push the DVR to the File IO by generating a DVR event.

 

I've made a number of assumptions:

1) That copying the data out of the DVR and generating an event doesn't actually do anything if there are no listeners subscribed.

2) Furthermore, if the listener is subscribed, but doesn't do anything with the data (a conditional structure eats the data to prevent the UI updating to quickly), then I assume that doesn't perform an un-necessary copy. This is a big assumption and may be giving too much credit to the LV compiler.

3) The consumer of the DVR (File IO) can operate faster than the producer can fill the circular buffer (I've yet to see this fail, though I definitely need to check this explicitly).

 

The reason it's crucial to delete the DVR outside the File IO is that the user can be using the UI to observe the image, without requesting that the images be streamed to disk. Thus the File IO is not subscribed to the DVR Event.

 

In all honesty, I'm hacking this Events based paradigm over a number of old decisions. So I'm really not sure if I'm papering over some redundancy.

 

Edit: Interesting behaviour that might be useful is that "Delete DVR Reference" has an output terminal for the data that the reference was linked to, I haven't thought if I can take advantage of this somehow to update the UI.

Edited October 20, 2014 by AlexA

[todd]

Jack had a neat presentation at NI Week 2013 that demonstrated a few memory issues and pitfalls but I can't seem to find it at the moment.

https://github.com/wirebirdlabs/LabVIEW-User-Events-Tips-Tricks-and-Sundry

[hooovahh]

https://github.com/wirebirdlabs/LabVIEW-User-Events-Tips-Tricks-and-Sundry

That's the one thanks.

[AlexA]

So after much modification of code I've run into the following phenomenon.

 

Even for relatively low data update rates (20 Hz, 4.8 kB/s) the front panel becomes unresponsive. Buttons are sometimes handled, sometimes not, sometimes they seem to be handled behind the scenes (control is enabled) but the boolean control in question remains in the false state.

 

As I understand it, Dynamically registered events can not be configured such that the front panel remains unlocked while they're being handled? Am I correct? If so, this is a big caveat that I wish I'd been aware of before starting down this path. The paradigm seems really nice, but with some fundamental flaws lurking below the surface.

 

Bugger.

[Neil Pate]

I think the standard recommendation of not doing too much processing in the event structure applies if you are also using it to handle GUI events. Are you doing this in your code?

[drjdpowell]

Am I correct? 

No.   There’s a “Lock Panel until handler completes†shortcut menu item on the Register for Events node.

[AlexA]

No.   There’s a “Lock Panel until handler completes†shortcut menu item on the Register for Events node.

 

Not for User Events i.e. the ones which you fire by calling Generate Event with some data.

 

Edit: I hunted extensively for such an option, and checking again, I can confirm it's not there. See attached screen shot

Edited October 28, 2014 by AlexA

[hooovahh]

Correct.  You can't set the behavior of locking or not locking the front panel from a user event.  If you wired a control reference to the register for events, and picked something like Mouse Enter then you can right click and choose to lock or not the front panel.

 

As for a user event locking the front panel. I've never had this be a problem because very little happens in the event structure.  For me the event structure is in a state machine, and after getting an event will go and o what it needs to.  During this time the UI is no longer locked and will respond.  So one solution is to make the code in the event structure very minimal so it locks the UI but only for a fraction of a millisecond.