John Lokanis

QUOTE (Mark Yedinak @ Mar 24 2009, 02:13 PM)

I would ask though if you find yourself only peeking at a small part regularly and only accessing the large data occasionally wouldn't it improve the design to separate the elements and their storage to improve performance?

Yes yes yes. That is very true. And that is exactly how I fixed it. My mistake was not realizing I was doing this in the first place. It wasn't untill I started using a large data set in the structure that the problem became more evident. But, after some work, liberal use of the inplace structure and addition of some shift registers, I was able to corden off the large data set and eliminate all reoccuring copies.

The concept of a tree of variants stored in a single element queue as an efficient way of sharing data between parallel threads seemed like such a good idea. But, like every programming construct, it does have pitfalls. Maybe I should put together an NI Week session on the do's and don'ts of massive parallelism in LV programming. I think I have had to climb out of almost every pitfall there is over the last few years...

QUOTE (MJE @ Mar 24 2009, 02:23 PM)

That may be true, but if you preview the queue, you make a copy. I suppose if you dequeue it, the element you get is just a pointer to the same block of memory that the queue was holding on to. But even that may not be true. After all, if you dequeue all elements in a queue, the queue does not free the memory it used to store those elements. A queue hangs on to all the memory it has ever allocated. So, if each element takes up 1k and you had 100 elements in the queue at one time, then the queue is still consuming 100k, even if you flush it.

Now, I do know that when I previewed the queue, even if I was just referencing an attribute of the variant that was a scalar, I did incur a memory copy of all the attributes of the variant. You can even see this if you turn on 'show buffer allocations'.

Getting the large structure out of the variant tree and into it's own queue that I did not access often saved the day for my app.

No network drives. Local storage only.

I am connected to version control (perforce).

I do have a lot of .NET calls and I do use VISA in some places.

QUOTE (Neville D @ Mar 24 2009, 12:42 PM)

So is the copy made because of the preview Q or is it because its an attribute of the variant?

Yes.

The answer really is both. When you preview, it makes a copy. When you then get the attribute value, it makes another copy. And, I am pretty sure when you cast the variant value of the attribute to its LV datatype, it makes yet another copy.

So, the worst case is when you try to access the large data structure in the queue element. But even if you are just looking as something small that is stored as part of the variant 'tree' you still incure one copy of everything.

QUOTE (Aristos Queue @ Mar 12 2009, 07:25 PM)

In that case, there is allocation regardless of the preallocate vs. shared clone setup. The cache of clones are only (to the best of my knowledge) shared among the subVI calls, not among the Open VI Reference calls.

In the end, the problem was not the clone setting but rather a very hard to find memory allocation issue.

Quick tip: don't place a large data structure in an attribute of a variant that you store in a single element queue and access from several points in your app. If you do, then everytime you preview the queue element to read it, you will make a copy of the whole shabang, even if you are only interested in another attribute of the variant structure that is a simple scalar. This will eventually blow up on you. Took over a year to discover that one!

So, now we are back to shared clones and all is well again... Oh, and the app runs 100x faster. Maybe I will have time to come to NI Week after all!

Am I the only one with this problem? Now I really feel alone...

QUOTE (Aristos Queue @ Mar 16 2009, 12:59 PM)

Check your print settings. Is 8.6 searching for a non-existent remote printer?

I have noticed that if the defaul printer is 'far away' (I'm on the VPN instead of at the office) then the VI Properties dialog takes forever to open. If I change the printer to something local (like the MS XPS fake printer) then it opens fast.

How can I permenantly change LV to point to the XPS printer? It seems to lose this setting everytime I close LV. Since I never print from LV, I don't care if it is pointing to a real printer.

Here is something that has bugged me for awhile:

If I open a project and then open a VI that has a lot of sub-vis, it takes a very long time to open (many minutes). I can see the 'VI loading' dialog and it is iterating through the sub-vis at a rate of about 1 VI per second.

Once the VI finally loads, I can close it and then reopen it and it opens in about 2 seconds.

So, at first I thought it was cached on the second load and that is why it was fast.

But, if I close the project and then reopen it, then load a simple VI with very few sub-VIs first, that VI will open fairly quickly. If I then try to open my big VI, it will also open quickly (in seconds). In fact, it opens so fast that I never see the 'VI loading' dialog at all.

So, does anyone know what is going on here? This only happens if I load the VIs from a project. If I load them directly they open instantly.

-John

QUOTE (Aristos Queue @ Mar 12 2009, 12:24 PM)

In preallocate mode, the memory is all preallocated at the moment you start your program running. There is clone allocation after that point, so that shouldn't be responsible for pegging the UI thread.

Yes, but if the app is continuously launching reentrant VIs dynamically, then every time it does this, it needs to allocate space for that VI and all it's sub-vis 'on the fly'. So, there is continuous memory allocation going on in this case.

QUOTE (Mark Yedinak @ Mar 12 2009, 11:04 AM)

Are you saying you need to have multiple UIs in your system or that coordinating your data for multiple applications on multiple machines to a single UI is the problem? If it is the latter you could use network queues to send data back and forth from the controlling UIs to the slave applications. If you want to have bidirectional communication I would have a master UI queue that all of your individual applications can post to to inform the UI that an update is required. As part of your message you would need an application ID. Each application would have its own queue for receiving updates from the UI. The master UI would need to manage the application queues. As processes start up they register with the UI an give it the connection specifics for its queue. The connection specifics would need to be static for the UI code (fixed server address and port number) so all of the remote processes know how to register with the UI. If the UI wanted to broadcast an update to all remote applications it could simply iterate over all of its remote queues and post the event.

Those are all good ideas for rearchitecting my system but it would be a lot of work at this point. Currently, my top level VI spawns sub-vis (engines) that run tests on a DUT. My top level VI can display the FP of any of these running engines inside a sub-panel on the main VI's FP. From this sub-panel, I can interact with the UI of the engine. The problem is I cannot use this technique if the engine is running in a different app instance.

So, I would have to use your techniques above and separate my engine code into a UI component (running in the app instance of the main VI) and a functional component, running on a different machine (and app instance).

Also, I would need to maintain a pool of machines (like a server farm) and be able to ping them for their current load so I can load balance tests across them.

A very interesting project, but first I would need to get my boss's approval to spend a few months on it. In the short run, buying a faster machine is easier.

I will be sure to post some benchmarks on the Nehalem machine vs the Penryn machine when I get it.

QUOTE (mesmith @ Mar 12 2009, 09:38 AM)

Here's an interesting article from the home team

Great link! Thanks!

I have considered load balancing my test system across multiple machines but it does make it a nightmare to do UI interaction. LabVIEW just does not offer an easy way to embed UIs from other machines. If only I could have a VI in a sub panel that was actually running on a different computer over the network!

But, for now, I will be looking at a Nehalem based system to get me over the hump. It is supposed to address the memory access and multicore bottlenecks and yield at least 30% improvement over Penryn based cores.

-John

Found some really interesting behaviors when trying out shared clone mode vs preallocate clone mode:

I have a massively parallel application that does a lot of dynamic VI execution in many (200+) threads. I have been trying to figure out if I should set all my reentrant VIs (almost everything is reentrant) to the 'shared' mode or the 'preallocate' mode.

I am running this app on an 8 core machine.

In 'shared' mode, the CPU load on all cores seems to be fairly balanced but the whole system bogs down as I cross the ~100 thread level. Here is what the CPU load looks like:

In 'preallocate' mode, the CPU load on one core seems to get 'pegged' a lot but the overall load seems lower. I can reach the ~150 thread level before it bogs down too much and can get to ~200 threads, albeit slowly. Here is what the CPU load looks like in this mode:

From talking with some sources at NI, it seems that the UI thread is used for the memory manager as well as all VI server calls. So, in preallocate mode, there is a lot more memory management going on (allocating space for all those clones) and that is swamping out the core that is running the UI thread and the VI server calls, thus causing the sluggishness. In shared clone mode, this does not happen, but the system has to do a lot more overhead (spread across all cores) to manage the shared clones and their execution data spaces.

Does anyone have any ideas on what the optimal solution/configuration would be in cases like this? Any 'best practices' to apply here? It seems to me if we are going to be doing a lot of multi-core programming in the future, LabVIEW needs to be tweaked to spread the load out a bit more effectively.

-John

QUOTE (Kal @ Mar 5 2009, 05:32 AM)

Hi John,

Thanks for posting this example.

I tried installing the web service using 'with RTE' installer in PCs not having LabVIEW. In some PCs the installed web service failed to work while in some other PCs it is working perfectly. Is there any way to check why it may not be working in some PC? I have enabled all the ports mentioned in the PDF document and even tried disabling the firewall. Still no luck.

Thanks

I would first test it with the 'echo' service built into the web server. If that does not work, then you have some sort of networking issue. Call your IT guys. There might be a hardware firewall in the path between your server and client.

If the echo test does work, then I would investigate the VI server ports on the EXE running on your server target. You need to have the EXE's ini file set to enable VI Server on the default port if you want to grab the front panel of a VI. To grab a screenshot, the screen must be rendered (someone must be logged in and looking at the screen). And the screensaver cannot be running.

Another thing to look at is the Distributed System Manager on the server machine. This will let you peek at the status of your web server to see if it is running. Call NI for support on this tool. It should have been installed with the RTE files.

Hope this helps. Good luck.

I will try to get around to posting a new version of this with some enhancements soon. One thing I added was the ability to point to different EXEs on the target server, as long as each of them uses a unique VI Server port. Keep watching this space...

Here is a link to the KB article that resulted from finding this bug:

Avoiding the window focus bug

QUOTE (Jim Kring @ Feb 12 2009, 02:03 PM)

> "It's really complicated!"

Unfortunatly this is true. But, I managed to get it to work. Check out this post:

http://forums.lavag.org/Screenshot-Web-Service-t13285.html' target="_blank">Screenshot Web Service

-John

Not sure if this will help but you can check out my web service project here. Some of the code show how to call into other LV apps from a web service and how to call a web service from within LV.

One thing I wonder about is how to enumberate all the LV EXEs running on a target. I don't think the method of finding all open projects would work in that case since these are separate compiled EXEs.

Anyway, here is the link:

Screenshot Web Service

QUOTE (Ton @ Feb 18 2009, 09:58 AM)

could you consider uploading this to the http://forums.lavag.org/downloads.html' target="_blank">LAVA CR?

Sure, but doesn't it need to pass some sort of review before it can be posted there? I thought this forum was where we posted things that might qualify and then the 'reviewer' can give it their approval before we submit it.

What is the proper process for getting someting into the CR? I'm happy to follow it.

-John

QUOTE (Jim Kring @ Feb 17 2009, 04:20 PM)

Thanks for posting this. This is very cool in several different dimensions! :thumbup:

Your very welcome. Glad to give something back to the community the has helped me so much over the years!

I have created a project that builds and deploys a web service to a target machine that does not have the LV dev suite installed.

I used some information on the NI site and some support from NI people to make this work.

The web service not only demonstrates how to do this but also has some useful features:

1. Takes screenshots on demand of target machine.

2. Can get VI panel images of any VI running on the target machine.

3. Shows how to call a LV web service from another LV app.

I hope you find this project helpful for learning about LV web services and as a tool for remotely monitoring your applications.

(I use this to check the status of my test system from my iPod Touch!)

Please let me know if you find any bugs or have any suggestions for improvement.

-John

Project:

Download File:post-2411-1234913566.zip

Readme:

Download File:post-2411-1234913620.pdf

QUOTE (PaulG. @ Feb 9 2009, 12:00 PM)

II

Sorry. some random noise must have gotten on the line while trying to write the original post. Please see the now fully operational battlest..er..post...

I

QUOTE (Aristos Queue @ Feb 4 2009, 08:12 AM)

Status: This bug was not isolated in time to be included in the 8.6.1 bug fix release. It will be fixed in the next release of LabVIEW.

Yes, I have been working with NI support on this for what seems like a year now. It is unfortunate we were not able to find the root cause in time for the 8.6.1 release. I am just glad they finally have confirmed the exisitance of this bug that has plagued my code for so long. I was beginning to fell like Hurley on Lost. Was I just crazy or was there really something wrong in the LabVIEW libraries?

Too bad it couldn't have been called CAR 4815162342.

One benefit of this has been to review my entire architecture and change it to no longer share queue references between threads. Instead every queue in a VIT gets namespaced and then every sub thread gets and destroys it's own reference to that queue. This eliminates the need to force destroy a queue to clean things up.

-John

QUOTE (jdunham @ Jan 21 2009, 03:45 PM)

If that doesn't work, verify that Windows firewall is open on port 3363

*$*#@!! Firewall. :headbang:

Thanks for the help!

:worship: :beer: :beer:

QUOTE (jdunham @ Jan 21 2009, 11:55 AM)

It should work fine, even over the network. First I would telnet to that machine on port 3363 (unless you changed VI Server's port) to make sure it is accepting connections. If that works, then the error out from each VI server call should tell you what's not working.

Can't telnet in. So, the server does not seem to be running.

Here is the contents of the ini file:

prefDlgTestData=1234

appFont=""0" 13"

dialogFont=""2" 13"

systemFont=""1" 13"

HideRootWindow=True

blinkBG=000000FF

DebugServerWaitOnLaunch=TRUE

server.tcp.access="+127.0.0.1"

WebServer.Enabled=True

server.tcp.acl="290000000A000000010000001D00000003000000010000002A10000000030000000000010000000000"

server.tcp.enabled=True

What am I missing?

(BTW: I am running 8.6)

I want to connect to another LV app on a different machine, open a reference to a specific VI in that app (the main GUI) and then use FP.GetImage to get a snap of the front panel.

I can't seem to get the right ini settings in my remote app to make this work. Has anyone else done this? If so, what are all the steps nessesary to make this work? Will it work over the network or must the caller and callee app both be on the same machine?

Can this work at all?

thanks for the help!

-John

I am trying to build a new runtime installer for LabVIEW under 8.6. I want to include support for the new web service feature. Looking at the additional installers page in the build installer screen, it is not clear what components are needed. I only want to include the parts that are needed to support this on a deployment machine where EXEs will be running.

Here is a partial list of the installers to choose from:

LV Web Services

NI Distributed System Manager

NI LV Web Services Runtime

NI LabVIEW Run-Time Engine Web Server

NI LabVIEW Web Server

The help is no help (as is often the case in LabVIEW) and a search with google and on the NI site have yielded no information. Also, I called the support line and spoke with an apps support engineer and they said they would have to get back to me because they also had no documentation on this.

So, LAVA is my last hope. Anyone else sort this out already?

thanks,

-John