Thoughts on large amount of config data

[dblk22vball]

I am looking to pick the brains of the LAVA members on how they deal with config files and/or large initialization files.

I have an application where I need to load in 20+ values to initialize the GUI and set up the programs test parameters. I am sure that with future applications, the number of items will increase, so I am trying to get some ideas ahead of time.

What methods would you recommend for large init files or config files?

I have done the single config file, and then you get the number of keys in that section, use a FOR loop to read all of the values and insert into array or cluster. This seems ok, but I have run into issues "elegantly" putting the data in a cluster (since there are different datatypes). You either have to read each key seperatly and put it into the cluster one at a time (can create a huge block diagram) or you use a FOR loop with a case structure and depending on the loop count, insert the value into the cluster, but this can get tedious for adding or removing data from your cluster.

The other way I have done it is to just save everything as a string (config file or just a text document) and read into an array and then index the array when you need the value. This seems to have a lot of unnessecary string to X conversions, and index array actions.

I have been trying to apply some OOP to this, but not sure if this is the right place, and since I am new to OOP I did not get that far yet.

Is there another method that you have used?

Thanks.

[jgcode]

QUOTE (dblk22vball @ May 5 2009, 08:11 PM)

I am looking to pick the brains of the LAVA members on how they deal with config files and/or large initialization files.

Is there another method that you have used?

Thanks.

By far the easiest way is MGI or Open G solutions.

The hard work is already done for you!

When working with classes I currently like to do this (props to MGI).

This way makes a lot of sense to me.

Both sets of VIs are elegant solutions for any size application - Get VIPM to install both!

The only time I do what you have done is on a Windows CE (last time I used it on a TPC config VIs do not work) and when I want to write my own style or format of file.

I recommend this book: Oldie but a goodie. There is some interesting ideas on config management.

A Software Engineering Approach to LabVIEW (National Instruments Virtual Instrumentation Series) (Paperback) by Jon Conway (Author), Steve Watts (Author)

But unless it is a requirement then why bother - use the above VIs!

I currently write to multiple files for config data.

One thing I am currently looking at is protecting the config file if I want multiple classes to read and write from the same file.

Obviously using different section names - but I mean the problem is config files can be opened and manipulated by simultaneous read/writes!

I am still playing with this but thought I'd mentioned it and am looking forward to what others say!

[Mark Smith]

QUOTE (dblk22vball @ May 5 2009, 06:11 AM)

I am looking to pick the brains of the LAVA members on how they deal with config files and/or large initialization files.

I have an application where I need to load in 20+ values to initialize the GUI and set up the programs test parameters. I am sure that with future applications, the number of items will increase, so I am trying to get some ideas ahead of time.

What methods would you recommend for large init files or config files?

I have done the single config file, and then you get the number of keys in that section, use a FOR loop to read all of the values and insert into array or cluster. This seems ok, but I have run into issues "elegantly" putting the data in a cluster (since there are different datatypes). You either have to read each key seperatly and put it into the cluster one at a time (can create a huge block diagram) or you use a FOR loop with a case structure and depending on the loop count, insert the value into the cluster, but this can get tedious for adding or removing data from your cluster.

The other way I have done it is to just save everything as a string (config file or just a text document) and read into an array and then index the array when you need the value. This seems to have a lot of unnessecary string to X conversions, and index array actions.

I have been trying to apply some OOP to this, but not sure if this is the right place, and since I am new to OOP I did not get that far yet.

Is there another method that you have used?

Thanks.

I manage large config files two different ways, but they both start the same way - build a cluster (always typedef this cluster - it makes changes so much easier!) that contains all of your configuration fields and then use one of the tools available to serialize the cluster contents to text and then to file.

Method 1 - use the Flatten to XML then Write to XML File to create an XML tagged text file and use Read from XML File and Unflatten from XML to recover the file contents. Attached is a simple example.

Advantages -

- easy to use - just put whatever you need for config info in the cluster and then flatten to xml and write to file

- All native LabVIEW code

Disadvantage

- harder to read and edit manually than a .ini style file

- you won't be able to read old configuration data if you change the cluster definition

Method 2 - use David Moore's Write Anything and Read Anything files

http://www.mooregoodideas.com/File/index.html

Just wire your configuration cluster to the input of Write Anything and it will automatically create a .ini style text file - use Read Anything to read the file and repopulate the config cluster

Advantages -

- creates easy to read .ini style files

- quick! The performance is more than adequate even on config files with hundreds of entries

- if the cluster definition changes, the Read Anything will still read the old config file - it just provides its default data for any fields not found

Disadvantages

- Mostly open code - there's one password protected VI in the bunch

- not native LabVIEW

I use both but for most simple configuration tasks I use method 2. OpenG also has a set of config file VIs that operate much like David Moore's, but the last time I used them (which has been awhile) I found it easier to use Moore's VIs and the performance to be better on larger config files.

Mark

Download File:post-1322-1241527217.zip

OK - I saw JGCode beat me to the punch but I'll post this any way!

[dblk22vball]

thank you both for the help, I will definitely look at the MGI tools.

[Grampa_of_Oliva_n_Eden]

QUOTE (dblk22vball @ May 5 2009, 08:56 AM)

thank you both for the help, I will definitely look at the MGI tools.

http://forums.ni.com/ni/board/message?board.id=170&message.id=267659&query.id=74453#M267659' target="_blank">Here is another approach that produces ini files that are easy to edit. Maybe not as robust as the others but it gives you another approach.

Ben

[jgcode]

QUOTE (mesmith @ May 5 2009, 08:41 PM)

OK - I saw JGCode beat me to the punch but I'll post this any way!

Good summary tho!

Also MGI VIs can take any data - it does not have to be cluster.

We use XML configs a bit e.g. for RT when the user does not have to see the contents of the file.

I.e. like in the cRED.

Maybe this toolkit would be good for you to clean your XML so it is human readable?

If there is no constraint for the file to be human-readable (which was not mentioned) then you could also use datalog files.

But I preferred the auto-conversion features of MGI and OpenG VIs for most apps when the typedef changes.

However, another thing I am currently playing with is config/file versioning.

I.e. setting data to default is ok sometimes, but other times it might be more appropriate to insert a known value and alert the user that the defaults have changed.

The idea being that on an upgrade (i.e. new release to client) these files can be read back in and handled appropriately if the file structure changes to the new version.

The old file or section then gets re-written in the new format.

Plus old keys are cleaned up.

This is used only if it is appropriate for the application.

All this happens within a class so its nice and encapsulated.

The only thing to do is save out the typedef and version it on Release in my project so that is does not ever change.

I kind of set it up like how an XControl reads its Library Version and acts appropriately on running init ability.

[Mark Yedinak]

I basically rolled my own solution that appears to be very similar to the Open G and MGI variants. The main difference that I have is that mine are specific to the section being read. They are driven by typedefs but they mine do require modification if the typedef changes. I may look at converting mine over to use these other methods for the ease of updates. However with thta said I would recommend some form of text based ini file with typedefed clusters for the parameters. You should be able to reuse sections of your ini file in different applications (for example network configuration data) when you have common data. Over time you build up a set of specific sections in your files and your users get a consistent looking configuration file across your applications. If you are only writing one or two applications this may not be as important but it becomes more desireable when you are maintaining 20+ tools for your users.

[PaulG.]

I think the simplest approach for your application would be using OpenG configuration files and a single cluster. I think you want to stick with a single cluster to keep data encapsulated and keep it simple. And the OpenG vi's have already been worked out for you. I use it all the time. It's just too convenient.

[asbo]

QUOTE (PaulG. @ May 6 2009, 09:06 AM)

I think the simplest approach for your application would be using OpenG configuration files and a single cluster. I think you want to stick with a single cluster to keep data encapsulated and keep it simple. And the OpenG vi's have already been worked out for you. I use it all the time. It's just too convenient.

I've used these before and I think they're fantastic - with a few short comings.

A) There's no way to "clean" an INI file against a cluster. If your typedef changes and you're using an older INI file, you'll have deprecated entries floating around that have to be removed by hand.

B) It's not easy to get a list of entries missing from the INI but are present in the cluster - one of the methods will throw an error if the two don't line up and I ran into this.

I toyed with solutions for these for a bit, but have bigger priorities so I never really came up with anything solid.