John Lokanis

WooHoo!  The two PRAGMA settings seem to fix it.  No Disk I/O errors and no speed issues (so far).  I'll have to test a bit more of course.  Thanks everyone!  LAVA FTW again!

Jack: I am using the .net interface for SQLite.  How do I set these flags?  I have poked around but not found a way to do it yet.

I'll give those a try!

Multiple Clients might be using the Server at the same time.  I want each of them to have their own data file for their work.  If I go with a database server solution then all their data gets intermingled.  And I don't want to spinup a database server for each user on the fly.  Oh, and I don't have a network resource to run the DB server, unless I go to battle with IT.

I wish there was a better way.  My understanding is the issue has to do with making the SQLite file transaction safe in case of power loss.  I wish there was a way to just turn that off.

In my case, I am never accessing it at the same time from the Client and the Server.  The Client does it's work and then closes the connection.  The Server then connections later and does it's writes.

I could replace this with a flat file of some type (.csv, etc) but I like the ability to query the data since I can reuse some of my SQL code I use with the regular DB server.  This is just sort of an off-line mode I am implementing for experimenting with new ideas where we dont want to pollute the central database but still want to collect data for analysis.

Sucks that SQLite can't do this since it is such a good solution otherwise.  I hate to have to switch to MSAccess but it is the only other file based DB solution I know of.

thanks for the feedback.

-John

I am using file based SQLite in my LabVIEW application as a temporary data storage file when not using the main database server.  My application is a client-server system so I need the ability to access the SQLite DB File from two machines at a minimum.  The Client app will create the DB if it does not exist and read existing data from the SQLite DB.  When it hands off to the Server application, the Server will write to the DB as it collects information.  Later, I want the user to be able to access the data with their preferred analysis tool.

My original plan was to place the SQLite file on a network drive that both the Client machine and the Server machine could access.  But, when I started testing this, I ran into a number of issues:

1. SQLite does not like UNC paths.  I got around this by detecting the UNC path and adding an extra to the start of the path.

2. When accessing a SQLite file using a UNC path, I get lots of random Disk I/O errors and my DB file is missing data.

3. I tried mapping a drive letter to the network path and this seemed to eliminate the Disk I/O errors but the speed was so slow that operations took several minutes or never completed at all.

So, after much googling, I found some comments that said SQLite does not work well when the file in on a network drive.  This sucks as it makes my application nearly impossible to do.  I don't want to have a server for the database since these were supposed to be one-off temporary data files for storing data when not using a real DB server.  I require an atomic file based solution.

So, has anyone found a way around this?  Is there any clever hacks or settings or other ways to make this work reliably?

thanks for any ideas,

-John

(using LV2013.0.1.f2 and SQLite Interop DLL version 1.0.90.0 based on SQLite3)

Looking for a LabVIEW Dev with minimum 5 years experience and CLD certification in the Seattle area for a 6 month contract.

This position would be working on a team doing code development, unit test generation and documentation.

Experience with LVOOP, .NET, Actor orientated programming and SQL a plus.

CLA certification also a plus.

If you are interested, send me your info via PM and we can go from there.

thanks,

-John

I am not sure I understand your solution.  But, I have already created the classes that store the data and do not want to modify them.  They are used throughout the application already and changes would be far reaching.

I could add a method to their parent that extracts the name-value pairs of the class or gets the value of the element by name.

I have a need to access the private data values and control names at runtime.  I have come up with a quick solution but was wondering if anyone had a better way to solve this problem.

Background:  I want to execute some SQL statements that insert data into a DB.  These statements are in a config file.  Each section of the config file has a name and a series of statements to execute.  These statements have tags for the data values that I want to replace with real data.  These tags correspond to the names of elements in a class's private data.

There are several classes that contain data I want to write.

There are several different sets of SQL statements, each with different set of tags within them.

I want to be able to modify the SQL statements outside of the application without changing the code.  This needs to work as long as the object containing the data has elements that correspond to all the tags in the SQL statements.

So, what I am looking for is a generic solution.  I feed in an object containing the data, a name of the set of SQL statements in the config file to use and the code takes the data out of the object, using the element names in the private data to match to the tag names in the SQL statements and then replaces the tags with the values of the data in the object.  I can then execute the statements and my data is in the database.

I have attached the VI I came up with to take the tag name and return the object data.  This has some limits I am willing to live with.  First, the elements in the object must have unique names.  If they are not unique, the tag is going to get replaced with the first match only.  Second, this mainly works for simple objects that do not use composition or arrays of other objects, since we would always match to the first element in the array.

Why not just create a VI that unbundles the data in the object and wires it into a VI to do the replacement and execution of the SQL?  Well, given the number of elements that need to be replaced, these VIs would be porcupines at best.  And I would have to update them every time I added elements to the class.

Why not simply bundle the class data into a cluster and pass that in, then use VI Server calls to get the cluster element names and do the tag replacement?  Again, I would have to maintain two versions of the class data structure and update both when changes were made.

And finally, why not build an array of name-value pairs from the object data and use that for tag replacement?  Same issue of maintenance.  I would have to add a new name constant each time I updated the class.

And with all of these solutions, I would have to build the custom version of the tag data extraction for every SQL command set case.

I know some of you have come up with some interesting ways of accessing class private data.  I am looking for a stable approach that is likely to continue to work in the future and is simple and fast.  Please share any solutions you have.

Here is my first attempt:

Get Class Private Data Value by Name.vi

-John

If you have VI Server on both sides, you can avoid polling, at least in your code.  I do not know if you consider VI Server waiting for incoming connections as polling.

I will look into web sockets.  For some reason, the links in you message just take me back to this LAVA post.  Weird.

In my current design, VI Server is my listener, always waiting for a connection.  The caller simply runs a VI in the receiver and injects a message onto the receiver's message bus.  So, both sides are callers and receivers.  It is basically a peer to peer messages system.

If I were to give up on VI Server, I would need to find a way to send my messages (LVOOP objects) over the alternative transport (TCP?).  I could flatten them to XML but not JSON as the built-in flatten to JSON functions don't work with objects.

But as you point out, the client would have to open a connection to the server, send it what ever commands it wants to send, then sit and listen for responses in a loop.

Since both applications are based on an Actor design (not AF) they need to not be blocked while waiting for messages.  This will mean coming up with some way to keep the listener in a separate thread that can hear messages, while at the same time still being able to send message to the server on the same channel.  And be able to abort the listener when the app shuts down.

Oh, and the server must allow N clients to connect to it at the same time.  And it must be able to reply to any of them or all of them at any time.  With little if any latency.

It just kills me that I get all this with VI Sever and now I can't use it...

Bummer.  That is what I thought.  Thanks for the feedback...

Unfortunately, that won't fix it.  The problem is not having app1 (the server) discover app2 (the client).  The problem is our network security blocks all connections from a server to a client.  So, it is fine for a client to initiate a connection to a server, but the server can never connect back.  Apparently this would lead to an insecure network so it is not allowed.  (personally, I think this is more due to insecure IT people, but either way it is out of my control).

So, I am trying to think of a way for the client to open a connection, leave it open without polling on it and allow the server to use that connection to send a message to the client at any time.  I would love to still use VI Server because it eliminates the need to worry about flattening data, but if that won't work, what is the next best thing? 

I think I am going to be out of luck solving this one, but can anyone think of a way for a LabVIEW EXE (call it app1) to call a VI on remote machine running a different LabVIEW EXE (call it app2) without app1 knowing the IP or name of the machine running app2?

What I need to do is allow VI Server to call a VI where the application with the VI being called has opened the connection to the caller.

So, the client needs to open a VI Server connection to itself and then allow the server to call VIs in the client using that connection.

I know it sounds impossible, but I was hoping there was some hack or some unique way of achieving this through some other means.  I do not want the client to poll.  The server needs to push the data to the client when the server is ready.  The client needs to be able to accept the data at any time.  And the data needs to be a LVOOP object.

-John

Yes, I thought of using plain TCP but I did not want to invent my own wheel if possible.  And I could not think of a way to avoid polling.

The connection must be robust so it can deal with dropped connections and not lock up.

I also need to serialize my message objects in order to send them.  I am concerned with the overhead this might entail.  I have the VI Server solution working well right now with decent responsiveness.

Is there a good LabVIEW messaging library out there that uses TCP and would support sending objects?

I need to find a transport for message objects that allows two way communication without polling but is limited to server side connections only.  So, the client can connect to the server but the server cannot connect to the client.

First some context:  My application communicates over the network using VI Server.  My client app (the UI) opens a ref to a VI in my server app (the engine) and sends a message object containing the client app’s machine name and VI server port.  The server app then opens a ref to a VI in my client and sends a message object with the reply data.  I now have a two way communication channel via VI server and can pass any message object back and forth without polling.

I learned today that our IT department plans to block all incoming connections to all non-server machines in the future.  So, my client would still be able to connect to the server app within the network, but the server would not be able to connect to the client app because of this rule.  This will completely break my networked messaging system.  I do not know a way for LabVIEW to setup VI Server so only one end can connect to the other but allow two way communication without polling.

Does anyone use a message system that would work in my situation?  I would prefer to continue to use VI Server but I am willing to look at other solutions, as long as they were very robust and had low latency.

thanks in advance for you help.

-John

'All VIs in Memory' is blocked for all remote connections.

"Returns a list of VIs in memory for the specified application instance. This property returns an error if you wire a remote application reference to the reference input."

I did finally get this working.  The problem was a mistake in the port number due to a default value in a VI I forgot to override.  So, it turns out I was doing something stupid...

thanks for the replies...

I am running into a problem running a remote VI using VI Server.  I am sure I am just doing something stupid but I can't seem to put my finger on it.

The system is intended to work both in Development (running in the IDE) and compiled mode (running under the RTE).

When I want to control the remote application from the local one, I open an application reference to the machine/port, then open a VI reference to the target VI.  The difference is, if the app.kind is Development System then I get the project list, search the project names for the one with my VI in it, then get that project's app reference.  I then open the VI ref using this app ref.  I use the VI name, not path to open the VI since is should already be in memory in the target application instance.  I am using the fully qualified name (.lvlib and .lvclass included in the name).

This works fine if both the 'local' and 'remote' projects are open on the same machine.  But if I put the remote project on a difference machine, then I get error 1004, the VI is not in memory.  I get this error even if the target VI has it's front panel open on the remote machine.  I have also tried using the main VI Server app ref instead of getting the app reference of the project from the main app reference but that gave the same result.

I can confirm that I have the correct application ref since it returns the correct lvproj name.  And if I open the VI using it's full path, that works fine.  It just appears that I cannot open a ref to the VI that should clearly be in memory.

So, is there some subtle difference in what is allowed with an application ref if it is on the local machine vs a remote machine?  Can I simply not get a ref to a VI in memory if the machine is remote?  I thought anything you could do between two application instances would work in all cases, regardless of if they were on the same machine or different ones.

thanks for any thoughts...

-John

Ok, I managed to fix this one.  I edited the class icon on the source machine, applied the change to all members of the class, then saved changes.  I then copied the code to the new machine, cleared the object cache again (just to be sure).  Restarted LabVIEW and opened the class.  It now looks correct.  Still no idea why.

But, when I try to close the class or the project, I get a save request for 100's of VIs.  The change description seems to be one of these two reasons:

"The master copy of a Type Definition (or Strict Type Definition) used by this VI was modified."

"The VI's name was changed outside of LabVIEW, or one of its subVIs was found in a different directory."

Since the disk layout has not changed, what else could be causing this?  Is this a known bug?

Nope.  Tried that.  And this is the first time opening the code on the new machine so I started with an empty cache.  But thanks for the suggestion.

Something is amiss.  I wish there was some tool to reset a class and clear away the mutation history so it does not have all that baggage.  I suspect many bugs lurk in there.

I have been over the steps I took to move the code to the new machine and I cannot figure out why it should be different.

Same OS

Same version of LabVIEW

Same versions of VI Packages

Same file layout on disk

It seems to me that LabVIEW code is becoming non-transportable as they add more features.  Especially LVOOP features.  I have done this type of move in the past with non-LVOOP code and had no issues.

Any other ideas to try?

I am running into a very strange issue.  I have a second dev machine that I want to use.  I took all my code and zipped it up, then copied it to the new machine and unzipped it.

Both machines have the same version of LabVIEW (2013) installed from the same DVDs.

Both machines have the same VI packages installed.

All files are in the same position on the drive on both machines.  (EXACTLY the same).

I *should* be able to open the project on the new machine and get no dirty dots.  I assume it will want to compile all the code once since I have separated source from BD.

Instead, one of the projects opens but want to save 108 VIs because it claims a typedef moved on disk  (it did not).

The other project opens and then as soon as the project window finally appears, LabVIEW crashes.  Every time.

So, I started opening individual classes one by one to see if I could find the issue.  On just the second class I opened, I found that some of the accessor VIs were missing.  They were on the disk and the .lvclass file was identical to the one on my main machine (I double checked) but when it was opened, LabVIEW claimed it had no knowledge of the accessors.  The control was in the class, just not the accessors.  And since that was missing, everywhere this class was used in a property node, it claimed that I had no element by that name to access.

I have never seen this before.  I assumed that all the information about what was in a class was in the lvclass file.  And if the files were the same, and the required VIs where still there at the exact same paths, then everything would work.  But it does not.

Has anyone run across this before?  Is there a fix?  I am guessing this might be due to class mutation data (a feature I could really do without) but I am not sure.

thanks for any tips that lead to a solution...

-John

Thanks for the input.  I decided to create a object cache FGV.  This seems to alleviate the speed issues.  There is still a blocking issue since it is a singleton implementation but that is unavoidable.  I suppose I could store the cache in a DVR or SEQ and 'peek' it to check for a hit, but I am not sure it is worth the added complexity.  Has anyone timed these options for performance?

Anyways, attached is my simple object cache VI if you are interested.

-John

Class Object Cache.vi

ps. I prefer this implementation to a 'load at startup' option because it allows the cache to change based on user interaction or by data driven actions.  In my case, the user loads a script that defines what object classes will be needed.  This can change at any time.  I only incur the load penalty once per class that is actually needed, instead of loading every plugin class possible at startup.  This also allows for new classes to be added while the application is running without a restart.

Of course, if your application is time critical on every run, pre-loading will avoid the first run penalty.

I have an application where I am loading plugins from disk and executing them.  Each plugin is a child class of my plugin class.  I am using the factory pattern.

During development, I started out by having a static plugin class object for testing purposes and then later switched to loading the plugin class default value from disk.

My architecture works by dynamically launching a generic plugin handler that then loads the required plugin class and dynamically launches a method in that class.  So, the handler and the plugin are disconnected from the main application.  They communicate using messages.

I am launching many of these plugins at the same time.  A common use case would be in the 100's (this is a test system)

When I switched from the static object to loading the classes, I noticed a significant slowdown, especially with higher #s of plugins (100+) loading at once.  I did an experiment to make all the plugins being loaded be the same so it should only incur the disk load of the class a single time.  When I compare this to the static plugin version, there is a 4x reduction in execution speed.

So, it seems that the function that gets the default value of a class is much slower and more resource intensive than using a static object, even if the class is already loaded into memory.

I also suspect that this function runs in the root loop, causing blocking issues.

Does anyone know of a way to speed this up or mitigate the slowdown?  In the past I used to cache refs of dynamically loaded plugins (before LVOOP) so I would not incur the load penalty.  There does not seem to be a way to do this here that I am seeing.

thanks for any ideas.

-John

Solved it with a variant.  I changed the system class DVR input in the handler to a variant and then cast it to the proper type within the handler.  Now the system class is happy to have the strictly typed vi ref of the handler in its private data.

I also added a test to the handler launcher that checks the stored ref and creates a new one if it is invalid.  I wrapped this in an IPE structure to prevent multiple callers from updating this ref at the same time.  I suppose that is not absolutely necessary since the ref is to the same VI type, but this prevents me from leaking references if two get opened at the same time and I only store (and clean up) the last one.

I see that AF does something similar except there the launcher is non-reentrant and the ref is stored in an uninitialized shift register.  Personally, I am not a big fan of non-reentrant code.  I only use that setting in specific cases (like FGVs) and tend to make all my other VIs shared clones.  I wish I could set that to be the default when creating a new VI...

Now, for each concrete instance of my system class that uses dynamic process (actor) launching, I get a separate VI ref to the handler.  I wonder if this creates a separate pool of shared clones?  And if so, I wonder if this improves or degrades performance?

My handler VI is not in a class, but it is in a library.

I have already made some child classes of my system that enable additional features, like networked messaging.  I suppose I could add a new child for dynamic launching, but that would mean that the networked enabled system class would need to inherit from the dynamic launching system class.  Should would be nice to have multiple inheritance for this situation.

Also, I consider the dynamic launching feature to be a core feature of the system so I would prefer to have it in the top parent.  I don't like adding complexity if I can avoid it.  But, this might be the only solution.  So far, it does not look like I can cast a generic DVR ref to a specific DVR ref.