DVR or data copy? How to deal with large data in AOP projects

[John Lokanis]

I'm working on a large project and have run into a quandary about how to best deal with a large shared data set.

I have a set of classes that define 3 data structures.  One is for a script read from disk that the application executes.  Another is the output data from the program, generated as the script is executed.  The last is a summary of the current state of execution of the script.  The first two can be significant in size and are unbounded.

In my application, I have one actor (with sub actors) responsible for reading the script from disk, executing it and collecting the data.  It also updates the summary status data.  Let's call this the control actor.  I have another actor that takes the data and displays it to the user, allowing them to navigate through it while the script is being executed.  Lets call this the UI actor.  The last actor is responsible for communicating the summary to another application over the network.  We will call this the comm actor.

So, the control actor is generating the data and the other two are consumers of the data.  I had originally thought to have all the data stored as state data in the control actor and then as it is updated/created just message it to the other actors.  But then they would essentially have to maintain a copy of the data to do their job.  This seems inefficient.  Then I thought I could wrap the data classes in DVRs and send the DVRs to the other classes.  That way they could share the same data.  The problem with that is they would not control when their data gets updated.  And I am violating the philosophy of actors by creating what is essentially a back channel for data to be accessed outside of messages.  Also, I could block the 'read' actors when I am writing to the DVR wrapped class.  I would have to be careful when updating subsections to lock the DVR in an in-place structure to do the modify and write.  Then comes the question of how to best alert the readers to changes made to the data by the writer.  Simpy send them a message with the same DVRs in it?  Or send a data-less message and have them look at the updated values and take appropriate action?

 

So, any best practices or thoughts on how to resolve this issue?  I appreciate any feedback..

 

-John

[Chris O]

I have an application where there is a lot of data moving around and I use DVRs for the data transport and user events to signal updates to any interested processes. Some of the processes are started with the application, while others are spawned and destroyed as users select various things to view in the UI portion (separate application). I haven't really looked at the actor framework to know if this is a violation of the intent, though.

-Chris

[todd]

Just adding the cross-post link:

https://decibel.ni.com/content/message/57754#57754

 

(And watching for good ways to do that data sharing. I've done it poorly, already, including sharing a DVR with two copies of a data class - one in producer and one in consumer, each with their own index.)

[John Lokanis]

Updated the thread on the NI site.  Please reply there.