Jump to content

DVR or data copy? How to deal with large data in AOP projects


Recommended Posts

I'm working on a large project and have run into a quandary about how to best deal with a large shared data set.

I have a set of classes that define 3 data structures.  One is for a script read from disk that the application executes.  Another is the output data from the program, generated as the script is executed.  The last is a summary of the current state of execution of the script.  The first two can be significant in size and are unbounded.

In my application, I have one actor (with sub actors) responsible for reading the script from disk, executing it and collecting the data.  It also updates the summary status data.  Let's call this the control actor.  I have another actor that takes the data and displays it to the user, allowing them to navigate through it while the script is being executed.  Lets call this the UI actor.  The last actor is responsible for communicating the summary to another application over the network.  We will call this the comm actor.

So, the control actor is generating the data and the other two are consumers of the data.  I had originally thought to have all the data stored as state data in the control actor and then as it is updated/created just message it to the other actors.  But then they would essentially have to maintain a copy of the data to do their job.  This seems inefficient.  Then I thought I could wrap the data classes in DVRs and send the DVRs to the other classes.  That way they could share the same data.  The problem with that is they would not control when their data gets updated.  And I am violating the philosophy of actors by creating what is essentially a back channel for data to be accessed outside of messages.  Also, I could block the 'read' actors when I am writing to the DVR wrapped class.  I would have to be careful when updating subsections to lock the DVR in an in-place structure to do the modify and write.  Then comes the question of how to best alert the readers to changes made to the data by the writer.  Simpy send them a message with the same DVRs in it?  Or send a data-less message and have them look at the updated values and take appropriate action?

 

So, any best practices or thoughts on how to resolve this issue?  I appreciate any feedback..

 

-John

Link to post

I have an application where there is a lot of data moving around and I use DVRs for the data transport and user events to signal updates to any interested processes. Some of the processes are started with the application, while others are spawned and destroyed as users select various things to view in the UI portion (separate application). I haven't really looked at the actor framework to know if this is a violation of the intent, though.

-Chris

Link to post
  • 1 month later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Similar Content

    • By Axelwlt
      Hi
      I am reading big TDMS files (~1GB) and I try to minimize memory usage.
      I read a signal once and I want to keep it for later usage, so I tried to write it to a DVR in the hope that I have only one instance of the data in memory.

      However the task manager tells me that uses 2 times more memory than it should.
      I think the problem is that the array is not deallocated although I don't need it after writing it to the DVR.
      Is there a solution to make sure I don't have copies of data?
    • By Ram Prakash
      Can anyone please tell what a DVR [ Data value reference ] is ? I want to know at what situation it will be used and what are the advantages we get by using DVR. I am really confused in this topic . If someone has any code in which they have worked with DVRs. kindly share it to me.
       
      Thank you.
    • By Zyl
      Hi everybody,
       
      I'm running into something I don't really understand. Maybe you can help me here !
      I've got a LVLIB that is used as an 'Interface': it exposes public VIs which wrap around public functions of a private class (see code attached) . The class is private because I want to force the users to use the 'interface' functions.
      In one of my interface VI, I create a DVR on the private class (Interface_init). The DVR is stored into a typedef (FClass_DVR.ctl) and this typedef is the 'reference' that link all the interface public functions.
      In TestCode.vi (which is not part of the lvlib and illustrates the standard code that a user can create to use my driver), I can call my public interface functions and link them without any problem.

      But as soon as I create an indicator on that reference (to create a state-machine-context-cluster for example), my TestCode VI breaks !

      The error returned is : This VI cannot use the LabVIEW class control because of library access scope. The LabVIEW class is a private library item and can only be accessed from inside the same library or libraries contained in that library.
      I understand that the class is private. But the DVR is contained into a public control. Using an In Place structure on that DVR into TestCode would not work, since the class is private. So why is the DVR control problematic at that point ? Creating it do not breaks any access protection...
      Am I missing something ?
      DVR Private POC.zip
    • By IpsoFacto
      I've got some weird stuff going on with a cRIO project I'm working on wanted to get some opinions on it. The basic architecture is a set of classes that do some process. That process registers with a server. The internal data of the process is held in a DVR and the server get's access to that DVR. Clients use TCP to ask the server to do something, the server makes a call against the classes DVR and returns a response to the client.
      To simplify the issues I'm seeing I created a class that internally just increments an integer every 500ms. The client asks the server what's the current count, the server asks the Counter class and returns the answer to the client. This works perfectly fine when running the VI in the IDE. When built it connects, will get the JSON message back, but always gets a default value from the DVR call (zero, in this case). As soon as I open a remote debug panel to the cRIO, everything is working. The count is correct, the client calls work, just like normal. As soon as I right-click, close debug, it goes back to zero. Open debug works, close debug, back to zero. I know the DVR isn't getting dropped because the count continues to increment while not in debug, the process is still running happily with no issues.
      Here's a few screenshots of the code;
      Count Class process (get the count, increment, write it back to the DVR) - Counter Class process
      You can see the DVR vi's are actually vim's using a cast. I can't imagine that's the issue.
      Server Side call - Server Side calls
      All this does is get the count from the DVR (same as above) and wraps it in JSON and passes it back to the client as a JSON string.
      I also implemented an Echo class that ignores the process and DVR's, it just takes whatever string you sent form the client to the server and passes it back with a prepended "@echo". This works when running as an executable with the debug turned off so I know the client, server, and the server/class calls are all working as expected.
      Any thoughts here would be welcome, thanks.
      edit: I added the any possible errors coming from the variant cast to the JSON reply. When the debug is open there are no errors, when the debugger is closed it throws error 91, but the in-place element structure reading the DVR does not throw any errors. How can a variant not exist until a debugger is opened and than it magically exists?
      edit: the internal data dictionary is a wrapper around a variant attribute, I wired out the "found?" terminal all the way out to the JSON reply and if the debugger is open the attribute is found, but not if the debugger is closed. Anyone have issues with Variant Attributes in Real-Time?
    • By the_mitten
      The introduction of parallel, read-only access for DVRs in LabVIEW 2017 adds a great deal of flexibility to using DVRs to monitor values in parallel executions of code. Fo\The downside of this, of course, is the necessity of using the In Place Element (IPE) throughout your code simply to read the value. Having IPEs throughout your code just to read a value both takes up block diagram real estate and also takes more clicks than desirable to insert.
      Similarly, though less frequently, there are times when you only need to update the value within a DVR without actually performing any logic inside of the IPE.  This situation is less frequent, at least for me, as I am usually using arrays or classes with DVRs such that I actually need to modify the existing data rather than simply replacing it.
      A more preferable solution to the above situations would be to have Read/Get and Write/Set VIs for the DVRs to simplify the process of working with them. This way, and IPE on the block diagram would only be needed when you were actually modifying the existing data within the DVR, rather than simply overwriting or returning the current value.
      Thanks to the power of malleable VIs and the type specialization structure that is now officially released in LabVIEW 2018, a better solution is now available. I’ve created two malleable VIs, Read  DVR Value (Parallel) and Write DVR Value that allow you to perform a write and a parallel read on any DVR data type.
       Now, you can use a single VI that you can insert via Quick Drop to read or to write DVR values.  
      Download the attached ZIP file to access the two malleable VIs and example code, and please let me know your thoughts in the comments!
       


      DVR Read and Write VIs 1.0.0.zip
×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.