Application configuration data storege

[pawhan11]

Hello Guys,

How do You store configuration data for applications?

I have tried ini files, xml, binary over the years. Now I started to think that having one SQLite file for the application would be nice + some interface to configure all that data.

What is Your experience on that aspect for small and big scale applications? What worked best, what kind of problems did You have? 

 

[crossrulz]

I have been using ini files for all of my applications.  But I recently ran into needing of a hierarchy for a few of my settings.  So I am transitioning to XML.  I am currently in the in-between stage of using both an ini and a XML file for my settings (a lot of libraries to update to change to full XML, but no time to do it).  I have been using XML Notepad for editing my XML files, making them super easy to make.  Ultra Edit also has some nice features to edit XML files, most notable for me has been the column editing.

SQLite might be a bit overkill because I want to be able to update settings with a text editor.  Otherwise, I would need to create a database editor for each and every application.  That would not be good for my plugins.  I am sure there are ways, but probably more complicated than it is worth.

[pawhan11]

@crossrulz

Are You using DOM parsing or toolkits like JKI XML where you cast to predefined structure in LV?

 

 

[crossrulz]

I am using the DOM parsing, especially since I need to interface with other places that generate XML and my structure can change based on several factors.  So it is easier to maintain the 1 library.  I do want to find some time to investigate the JKI XML toolkit to see if it will simplify some things, but I am swamped through at least July.

[pawhan11]

@crossrulz

Thanks for the reply, Are You using default parser integrated in LV or something better? Have You tried to use it on RT systems as well?

I have looked some time ago and it seems there was no native DOM parser for LV RT. That was the main reason I had to stick with ini files on RT.

[drjdpowell]

I use JSON, and have a library for JSON, which has recently been added to the LabVIEW Tools Network.  I don't modify the config files with a text editor, though; all config editing is done in the Application itself.  I also do a bit of JSON in SQLite.

[crossrulz]

I have been using the default with a few additional abstractions I wrote.  And, yes, that does have a DLL call in the end, so won't work with RT.  All of my applications lately are on PC, so no issues there.  If going RT, I would look into the JKI XML and the GXML libraries.  From my quick looking, those are completely written in LabVIEW.

[Tim_S]

INI files were the way to go years ago, but they have the challenges when handling complex data types and arrays. I used XML (DCOM) in the last revision of my core application (designed for medium to large system). It worked well in small scale, but was a significant impact (30+ seconds) when my configuration editor and application tried reading in the file of a full system. I switched to JSON using this package which has greatly improved performance. Each plugin can have its own section for any configuration, so additions are easy. The files aren't intended to be edited directly, but I do install Notepad++ on the PCs to make it easy to go in and take a quick look (useful for repairs should anything bad happen).

[Cat]

I use ini files.  If I have complex data types, or large amounts of data required, the ini points to files where that data is stored.  Also, LabVIEW annoyingly creates an ini file for every executable, so I figure I might as well use it.  

 

Cat

[hooovahh]

I have some nice configuration managing code.  Where I define the VI names which are like sections in an INI file, and then the control values themselves are what is read and written to the INI file where each control is a key.  There are also methods to tell the VI values have been changed, because maybe other things like tables or trees need to be updated based on the change.  

This works real well for basic things and adding a checkbox or a debug setting that is then a value that can be read globally in the application is a snap.  But as you noticed complicated structured, or large structures cause issues.  I use the OpenG Variant INI functions which flattens larger structures into somewhat readable sections in the INI but adds many lines to the file and slows down reading and writing a lot.  I added an ability to read or write a key as a binary blob so it will look like Debug=89FJD82JKSKFK2I.  Which isn't as useful and thought about re-writing using the MGI INI functions.  I've been meaning to transition to SQLite, or possibly TDMS since our test data is in TDMS, and being able to embed the tester configuration, along with the test sequence, and test data into a single file is an attractive idea but just haven't had the time at the moment.

[smithd]

18 hours ago, drjdpowell said:

I use JSON, and have a library for JSON, which has recently been added to the LabVIEW Tools Network.  I don't modify the config files with a text editor, though; all config editing is done in the Application itself.  I also do a bit of JSON in SQLite.

same, although I have some header data stored in ascii\r\n format which includes a payload data version and who wrote the file, when.

I manage the ini equivalent of 'sections' by creating a folder hierarchy, and maintain a history of all edits performed on a given file. my little library looks for the newest file (or the file at some specific time) at the specified subpath. 

For centrally driven config management I use a sql db (sqlite so far) which has basically the same structure of key (=folder hierarchy path segment), value (json string or blob), and activation time, with the addition of the hostname of the device that owns that configuration data. This tool also lets me tag a set of values under a specific ID regardless of their timestamp, so I can redeploy that named configuration.

while I do use Json for most everything, I think the most important tool at my disposal is this set of helper libraries...the actual file contents don't matter so much.

Edited February 6, 2018 by smithd

[drjdpowell]

1 hour ago, smithd said:

although I have some header data stored in ascii\r\n format which includes a payload data version and who wrote the file, when

I started doing that, but switched to an "outer" JSON object with the sections "Rev" (data version number), "Settings", and "Other Info" (which records timestamp, User, computer, software versions, CRC-32, etc.).  

[Gribo]

I use JSON, and to prevent the end user from editing, I add an MD5 checksum. 

[smithd]

On 2/6/2018 at 2:17 AM, drjdpowell said:

I started doing that, but switched to an "outer" JSON object with the sections "Rev" (data version number), "Settings", and "Other Info" (which records timestamp, User, computer, software versions, CRC-32, etc.).  

Thats probably a better plan if you're always sure to use json. Conditionally wrapping json or xml or whatever means handling escape possibilities and when I first made the format I wanted to be open to other payload types.

22 hours ago, Gribo said:

I use JSON, and to prevent the end user from editing, I add an MD5 checksum. 

I dont really understand this (or the crc). why use a human editable format if you dont want people to edit it? And more generally, why not let people edit the files? you have to validate the contents anyway, right?

[Gribo]

Because there are programmers, there are techs and there are operators. You don't want your operators messing with your data (test limits, etc), but you do want them to be able to tell you on the phone, what data there is there, in case you can't remote into the machine.

[pawhan11]

I remember doing checksum in ini file as one of the params and do actions in SW based on it. Now in SQLite idea is to have one DB with all params password-protected, depending on user privileges they can edit or only view. At the end, people can find way to alter everything if they really want, but then it is no longer my problem  

What approach do You use when exchanging data between other platforms written in c#, java or others? I have used XML since all had some parser libraries, now I face that problem with NI RT systems since there is no parser and writing one is beyond my skills.

[drjdpowell]

8 hours ago, smithd said:

Thats probably a better plan if you're always sure to use json. Conditionally wrapping json or xml or whatever means handling escape possibilities and when I first made the format I wanted to be open to other payload types.

JSON is properly escaped, so you can put any binary data in a JSON string.  However, I do just standardise on JSON.

[drjdpowell]

8 hours ago, smithd said:

why use a human editable format if you dont want people to edit it? And more generally, why not let people edit the files?

In my case, the CRC is because the Setting file lives in flash memory on the hardware, and is downloaded to the computer, so we are guarding against corruption.  Customers (operators) never see this file, but internal people (techs) need to read them.  They could edit them, if they wanted to do that instead of using the configuration tools.

[drjdpowell]

What approach do You use when exchanging data between other platforms written in c#, java or others? I have used XML since all had some parser libraries, now I face that problem with NI RT systems since there is no parser and writing one is beyond my skills

JSON is starting to overtake XML for data exchange.  JSONtext should work on RT, though I have yet to try it.  

[smithd]

3 hours ago, drjdpowell said:

In my case, the CRC is because the Setting file lives in flash memory on the hardware, and is downloaded to the computer, so we are guarding against corruption. 

6 hours ago, Gribo said:

Because there are programmers, there are techs and there are operators. You don't want your operators messing with your data (test limits, etc), but you do want them to be able to tell you on the phone, what data there is there, in case you can't remote into the machine.

Makes sense.

3 hours ago, drjdpowell said:

JSON is starting to overtake XML for data exchange.  JSONtext should work on RT, though I have yet to try it.  

Definitely works on RT*. You'd hear from me right away if that stopped being true . Its definitely slower than using the primitives, but its much easier to use and 90% of my data is so small it doesn't matter. The other 10% is so big (ie images, gigantic arrays) I just use binary.

*with the obvious exception of that "make this string control have pretty colors" function and any similar