Curses Labview R&D!

[Daklu]

So I've been playing around with implementing a Memento interface, figuring it could be a fairly useful feature to have in many different kinds of objects. Sure you could use the Flatten to String prim, but what if your class uses run-time resources like queues or DVRs? Wouldn't it be cool to be able to have it automatically generate those runtime resources if needed when you restored a previous state?

So I spend a couple days designing and experimenting with it until I have something I'm happy with. As I'm wrapping up some documentation I think, "Hey, I wonder what does happen if I try to unflatten an object that uses run-time resources?" What do I discover? Flatten and Unflatten have absolutely no trouble at all handling run-time resources in classes. Silly me.

"Hey everyone, look at the cool little wagon I built. It's a little rickety and you have to kind of hold the wheel on with your foot if you start going too fast, but it's really useful if you have to go someplace! Pay no attention to that shiney sports car AQ is giving away... this is way better!"

Edited December 22, 2009 by Daklu

[Grampa_of_Oliva_n_Eden]

...

"Hey everyone, look at the cool little wagon I built. It's a little rickety and you have to kind of hold the wheel on with your foot if you start going too fast, but it's really useful if you have to go someplace! Pay no attention to that shiney sports car AQ is giving away... this is way better!"

Yes but what YOU get is knowlege and an insight thatcan serve you the future.

Did I ever tell you about coding up Control references in LV before control refs where released? Gave me a good appreciation of what had to go into control refs.

Ben

[Daklu]

Flatten and Unflatten have absolutely no trouble at all handling run-time resources in classes.

Well I'm either drunk or stupid, and since I don't drink that pretty much eliminates one possibility.

My original tests must have been invalid. For the life of me I haven't been able to recreate the situation where a class' run-time resources (queue, DVR, notifier, etc.) are automatically recreated when unflattened from a string. Reading the help leads me to believe it *shouldn't* do that, but I saw it work.

Persisting objects to disk is something I have not had to do in my own applications so I don't have a good feel for what is and is not possible. My sandboxing has been inconclusive. I know AQ(?) and others have suggested using Flatten to String to store objects, but I don't think I've seen any mention of whether or not it is supposed to create queues, etc. automatically when the object is unflattened. Does anyone with more experience in this area know?

Yes but what YOU get is knowlege and an insight thatcan serve you the future.

If I'm lucky... sometimes I just get confused.

[Norm Kirchner]

Perhaps I'm missing something, but have you simply tried the write to binary file?

I've been standing on my soap-box around here every chance I get when someone says they want to store 'cluster like' data to disk.

Now I know that this is not exactly what you're trying to do, but it shows how easy it is to save an object to disk.

<a href="http://content.screencast.com/users/NJKirchner/folders/Jing/media/dab29e2e-3762-4d8b-872d-18d564aaf030/2009-12-30_0822.png"><img'>http://content.screencast.com/users/NJKirchner/folders/Jing/media/dab29e2e-3762-4d8b-872d-18d564aaf030/2009-12-30_0822.png"><img class="embeddedObject" src="http://content.screencast.com/users/NJKirchner/folders/Jing/media/dab29e2e-3762-4d8b-872d-18d564aaf030/2009-12-30_0822.png" width="331" height="128" border="0" /></a>

So from what it sounds like you're talking about, I would not flatten to string or anything of the sort. Just write to disk.

There are a myriad of added benefits to doing this. The LV back end handles the situation of the class definition saved to disk is different than that which you are re-populating the object with. Also, you can save an array of differing objects to disk this same way. I know there are a few more benefits, but they elude me at the moment and I gotta run.

Good Luck

~,~

[Grampa_of_Oliva_n_Eden]

Well I'm either drunk or stupid, and since I don't drink that pretty much eliminates one possibility.

My original tests must have been invalid. For the life of me I haven't been able to recreate the situation where a class' run-time resources (queue, DVR, notifier, etc.) are automatically recreated when unflattened from a string. Reading the help leads me to believe it *shouldn't* do that, but I saw it work.

Persisting objects to disk is something I have not had to do in my own applications so I don't have a good feel for what is and is not possible. My sandboxing has been inconclusive. I know AQ(?) and others have suggested using Flatten to String to store objects, but I don't think I've seen any mention of whether or not it is supposed to create queues, etc. automatically when the object is unflattened. Does anyone with more experience in this area know?

If I'm lucky... sometimes I just get confused.

I have written/read class data to from file and it worked great. Yes all references to LV resources (like queues etc) need to be recreated if they went invalidi ( I imagine you could get a ref toa queue stick it in a class data write to file read back and it still works if teh queue was kept alive elsewhere).

Re: Writing to binary files

I always imagined there was a flatten to string built-in to the Binary file operations. Is there a difference?

Ben

[Norm Kirchner]

I have written/read class data to from file and it worked great. Yes all references to LV resources (like queues etc) need to be recreated if they went invalidi ( I imagine you could get a ref toa queue stick it in a class data write to file read back and it still works if teh queue was kept alive elsewhere).

Re: Writing to binary files

I always imagined there was a flatten to string built-in to the Binary file operations. Is there a difference?

Ben

As far as if the bit files are different.... it's a mix between an elephant and a rhino.

elephino

But it's worth checking out. But would there be a reason why you would choose to flatten first requiring 2 VI instead of just writing to disk directly?

Also, I don't know if the smart mutation is handled when unflattening from string.

[PaulG.]

elephino

That's a classic. I learned that as a kid and still laugh.

Except that one time my mother overheard me tell it and she washed my mouth out with soap.

[Shaun Hayward]

But would there be a reason why you would choose to flatten first...?

I know its a subtly different case, but I will often flatten to XML first so that I end up with a data store that is automatically mutating (especially useful when it comes to "preference" objects) but at the same time is human readable / editable*.

Shaun

*Manually editing XML files can lead to the file getting corrupted so I usually will give a user a UI to change all settings, but it can still sometimes be very useful when debugging an application out in the wild

[Daklu]

Perhaps I'm missing something, but have you simply tried the write to binary file?

Nope, you're not missing anything. I was. (A good dose of "how to do this.") I ended up removing the flatten to string after I posted the message.

I've been standing on my soap-box...

Wha..? You mean you have one too!? Oh, I see yours is from

I prefer mine...

Yes all references to LV resources (like queues etc) need to be recreated if they went invalid.

This is exactly the confirmation I was looking for. My original test setup must have been wrong; I think I forgot to destroy the run-time resources.

I always imagined there was a flatten to string built-in to the Binary file operations. Is there a difference?

If you flatten the class to a string you end up with the same data that is written to disk when you wire the object directly to the Write to Binary prim. However, when you write the flattened string to disk using the Write to Binary prim it prepends four bytes indicating the string length to the data. If you flatten the object to a string, save it as a binary file, and try to read from the binary file directly into the object it will fail. You have to read it as a string then unflatten it. (Or use the simpler route and skip the flatten to string altogether.)

[crelf]

The LV back end handles the situation of the class definition saved to disk is different than that which you are re-populating the object with.

What do you mean by "handles"?

[Norm Kirchner]

I know its a subtly different case, but I will often flatten to XML first so that I end up with a data store that is automatically mutating (especially useful when it comes to "preference" objects) but at the same time is human readable / editable*.

Shaun

*Manually editing XML files can lead to the file getting corrupted so I usually will give a user a UI to change all settings, but it can still sometimes be very useful when debugging an application out in the wild

And with your statement we shall all rise up against AQ and smite him down...

Why? you may all ask......

THIS IS WHY!!

The Flatten Object to XML does not flatten all of the object. ***IT ONLY FLATTENS THE PARTS THAT ARE NOT THE DEFAULT VALUE***

So although it's readable, you can have absolutely NO IDEA what is in the class if you don't know the default value.... which technically you only know IF YOU HAVE LV OPEN AND THE VERSION OF THE OBJECT IT WAS CREATED WITH!!! So having it readable gives you no value whatsoever. and it won't even translate the structure to the XML and just leave the values blank. It just simply omits them altogether. Does anyone see any value now in taking an object and flattening it to XML? I sure as hell don't.

This is a warranted discussion on another thread, but felt necessary to vent here. Sorry about the caps

</rant>

So you may want to be careful when going to XML w/ an OBJ

~,~

What do you mean by "handles"?

Check out the class mutation section of the LV Help file under "Developing LabVIEW Classes"

I have not really pushed the mutation algorithm, but have found it useful for simple additions and subtractions to the private class data structure.

[Shaun Hayward]

The Flatten Object to XML does not flatten all of the object. ***IT ONLY FLATTENS THE PARTS THAT ARE NOT THE DEFAULT VALUE***

Yep, came across that sneaky little one!

The work around I used (which while not the "nicest" approach, worked quite well) was to have a field in my class that I would always change. In the case that I'm thinking of right now it was the filename - unnecessary, for the most part, but did come with the benefit of allowing for a "save" versus "save as" type of functionality.

[crelf]

The Flatten Object to XML does not flatten all of the object. ***IT ONLY FLATTENS THE PARTS THAT ARE NOT THE DEFAULT VALUE***

Ohhhhhhhhhhh.... Of course that totally makes sense, although it's not quite what I'd expected

[Aristos Queue]

The work around I used (which while not the "nicest" approach, worked quite well) was to have a field in my class that I would always change. In the case that I'm thinking of right now it was the filename - unnecessary, for the most part, but did come with the benefit of allowing for a "save" versus "save as" type of functionality.

You need to have one of those fields at every level of inheritance -- if the child segment is non-default but the parent segment is the same as default, only the child segment will be written down.

Before the smiting begins, let me lay out "how we got to why things work the way they work today".

A) In LV 8.2, the first OO version, there was no XML support for objects. The only built-in persistence for objects was the Flatten To String primitive.

B) When we save a VI, we save the value of diagram constants and default values for controls. To save space, we don't bother to write down those values when they are the default default value of the data type.

C) LabVIEW classes are the first and only data types in LabVIEW where the default default value of the type can be changed. You can't tell LV "from now on, the default default value of an int32 is 1" but you can change the default default value of a class. This lead to a discussion of what the right decision should be about behaviors when the default default value is changed. Ultimately we settled on the behavior that a default value object should load as the default value, whatever the new default value happens to be. That decision has been criticized elsewhere on these forums, but four years later I'm comfortable that it was the right decision.

D) That raised the question of representation of the flattened string. The format of an object saved into the VI and the format of the binary string didn't have to be the same, and we could have chosen different behaviors. But once we decided that objects would include their version number in their flattened representation, it made sense to have the unflattening of binary strings and the loading of saved VIs have the same rules for mutating data. Thus the default value should remain the default value. We now have two options: To not bother saving the flat data when it is default or to save the flat data and compare against the default when we load into memory. The former saves a lot of disk space AND has significant performance advantages. Thus we decided that the Flatten To String primitive, which creates a binary string of the data, should not write down the default value.

E) When the XML prims were added in LV 8.6, we decided that the XML strings should again follow exactly the same mutation rules, and, again, the same performance trade-offs bent the discussion to not flatten the default value.

F) XML is meant to be human readable, but, as others in this thread have noted, not necessarily human editable. If you provide an editor to your users for modifying the data, that interface will allow you to change the values without caring whether the value was written down or not.

G) If you're using XML to share data between apps, you probably want to read this thread wherein I propose that you do NOT want to use the built-in LV XML primitives for XML that is going outside of LabVIEW, not because those prims are somehow flawed but rather that any built-in XML format is undesirable for data sharing (it's not a long thread if you want the arguments).

H) Using XML has allowed people to write apps that circumvent the privacy protections of the object data -- flatten to xml, modify the xml, then unflatten the data again. Hiccups such as "the default value is just missing" throw a wrench into this blatant violation of app security. In retrospect, the ability to flatten to XML (and to binary string, for that matter) probably should have been restricted to member VIs by default (author of the class could remove the restriction, just like we did with Create Data Value Reference in LV 2009) so that most classes wouldn't allow this hack. I've been talking to folks about whether or not it is too late to fix this mistake.

[Daklu]

Thanks for the explanation. Decisions that on the surface appear odd almost always have valid reasons behind them. Favoring speed and size I think was the correct decision. If a class developer needs more advanced behavior they can implement it themselves.

H) In retrospect, the ability to flatten to XML (and to binary string, for that matter) probably should have been restricted to member VIs by default (author of the class could remove the restriction, just like we did with Create Data Value Reference in LV 2009) so that most classes wouldn't allow this hack.

This is an issue I've run into while developing the memento interface. The whole point of the memento is to allow third parties (caretakers) to store an object's state without allowing the caretaker access to any of the state data. Flattening the memento to a string allows the third party to see and manipulate what's inside. I concluded that if that kind of security is required the object would need to encrypt it's state information before storing it in the memento. Not a great solution.

On the other hand, being unable to save an object to disk because the class designer didn't include SaveToDisk or FlattenToString method could be a huge obstacle when creating applications. What happens when I try to save an object that is composed of several other objects, one of which disallows flattening?

I've been talking to folks about whether or not it is too late to fix this mistake.

My first reaction was, "No, it's not too late. Please fix it." My second reaction is, "I'm not sure it's a mistake." How do other OO languages handle persisting objects to disk? Isn't manipulating the binary string or xml data essentially the same as, say, saving a c++ object to disk and manipulating the binary file?

[asbo]

D) That raised the question of representation of the flattened string. The format of an object saved into the VI and the format of the binary string didn't have to be the same, and we could have chosen different behaviors. But once we decided that objects would include their version number in their flattened representation, it made sense to have the unflattening of binary strings and the loading of saved VIs have the same rules for mutating data. Thus the default value should remain the default value. We now have two options: To not bother saving the flat data when it is default or to save the flat data and compare against the default when we load into memory. The former saves a lot of disk space AND has significant performance advantages. Thus we decided that the Flatten To String primitive, which creates a binary string of the data, should not write down the default value.

Just out of curiosity, was there ever consideration given to the primitive having a context-menu option to offer access to either behavior?

[Aristos Queue]

How do other OO languages handle persisting objects to disk?

JAVA has a built-in serialization mechanism that can be privitized if desired. C# has a built-in "to string" but not a "from string". C++ does not have a built-in to or from string. I find that the C# system is the most common among other languages.

Just out of curiosity, was there ever consideration given to the primitive having a context-menu option to offer access to either behavior?

No. That would defeat the entire purpose. It has to be a class-level decision, in the hands of whoever is deciding how much security this particular class needs to have.

[Shaun Hayward]

Just a thought... but could there be any possiblility to allow "special VIs" for classes kind of like "Ability" VIs for XControls. That way, a developer could specify how a class should behave when it gets serialized, etc?

...oh... I've just realized whilst typing this... this would also open a whole other fun bag of worms - Operator Overloading

[Daklu]

C# has a built-in "to string" but not a "from string".

I don't think the .Net ToString() method is meant to be used to save objects to disk. ToString() can be overridden in classes to output whatever the class designer wants. On top of that it returns culture-sensitive strings, such as 1,00 instead of 1.00. For persisting data you would use serialization. I read through the help a bit and C# requires an object to have the [serializable] attribute in order to use the language's built-in data persistence methods. I guess that's a vote in favor of restricting the flatten prims when used with classes.

C++ does not have a built-in to or from string.

It's been almost 20 years since I've done anything in c++, but I thought there was some way to pipe arbitrary data to disk? Most likely I'm just confused...

[Aristos Queue]

Just a thought... but could there be any possiblility to allow "special VIs" for classes kind of like "Ability" VIs for XControls. That way, a developer could specify how a class should behave when it gets serialized, etc?

Yeah... I've got a whole set of specs for this, including VIs that would define the class's behavior when wired to the ? terminal of a case structure, defining coercion dots for classes to non-class types, etc. I wanted those back in 8.2, but other things keep getting in the way. There is some limited work being done in this area today that you should see either 2010 or 2011.

[Shaun Hayward]

Yeah... I've got a whole set of specs for this, including VIs that would define the class's behavior when wired to the ? terminal of a case structure, defining coercion dots for classes to non-class types, etc. I wanted those back in 8.2, but other things keep getting in the way. There is some limited work being done in this area today that you should see either 2010 or 2011.

Awesome!