Major LabVIEW Slowdown (Several 100 time slower)

[PJM_labview]

I have been experiencing this bug since LabVIEW 8.0. I have not report it until now because it is VERY hard to reproduce. Nevertheless I have characterized it.

Note:

1. I (and other coworkers) have encountered this bug on several machines running XP and LV 8.x or 8.20.
   
2. This bug is present both in dev mode and in runtime mode.
   

Background:

In some instances, some LV code suddenly slow down so much that in most case it gave the impression that the LV prog has hanged.

Typically this seem to happen once or twice a week (I am using LabVIEW everyday) and only on some VIs.

Clicking, and shaking the hanged program title bar "unstuck" the prog temporary.

Restarting LV (or if in runtime environement, your program) fix the problem until the next time.

Characterization:

In one of these instance when the bug occurred, I spend several hours characterizing the issues.

I have first characterized this by comparing run time of a particular VI both in LV 7.1 and LV 8.0.

Note:

1) The slow down was happening on two VIs, one with a for loop another with a while loop. I just picked the one with the while loop to compute the stats.

2) The VI was run both on LV 7.1 and LV 8.01 for the same test set.

3) The VI was running continuously when the screenshot were taken.

Below are screenshot and benchmark data of that particular VI.

This is on LabVIEW 7.1. Total run time (for 91 iterations) is about 10 ms with an average per iteration of about 0.11 ms (processor usage is 50%).

This is on LabVIEW 8.0.1. Total run time (for 91 iterations) is about 1616 ms with an average iteration of about 17.8 ms (actually the value are all over the place, from 12 ms to 25 ms). Processor usage in this case is very low (about 4%, in the noise level of idle)

Please Note: 1616/10=161. This VI is running 161 time slower on LV 8.0.1.

This first experiment let me to believe than this might be some optimization that went very bad and that it seem to happen only on VI manipulating LV Object References.

Further Investigation:

In another instance when my LV got in that slowdown state, I further narrowed the issue to the following:

This particular slowdown appears to happend only when operating on LV Object Reference in a For Loop or in a While Loop.

Below is a screenshot of the VI BD I used in both LV 7.1 and 8.0.1 to get to this conclusion.

Once the statistical code is exclude, this VI is very simple. It just open a reference to itself, get all the Object Reference on its panel, then close the array of reference in a for loop (this last operation is benchmarked). Then for comparison sake, another identical for loop (doing pretty much nothing) is run for the same number of iterations and benchmarked.

This VI was run both on LV 7.1 and on LV 8.0.1 (when in the "slow down state").

Below are screenshots of the results. The Left side statistical data of the Panel is about the for loop that is indexing the array of Object reference. The right side statistical data of the panel is about the No Op for loop. On each side, the first plot is the delta t of the for loop run time and the second does average the delta t over each iteration of the continous run.

This is the VI running on LV 7.1: This is fast. Typically the indexing loop run in less that 0.2 ms

This is the VI running on LV 8.0.1: This is very slow. Typically the indexing loop run in more than 44 ms

Note: 44/0.2=220 This VI is running 240 time slower on LV 8.0.1.

Note: The No Op loop does also appear to be slower on LV 8.0.1 but the value are so low, I am not sure this is significant enough.

In any case, this is a very serious issue that I hope will be fixed.

PJM

[PJM_labview]

See this post on the LV 8.20 bug list.

PJM

[tnt]

Hi,

I noticed that since LV7.1 the behaviour of the continuous-button had changed.

e.g. the first call is always TRUE when in LV 7.1 it was only TRUE the first loop.

I'm curious if you could repeat your tests while running your vi conyinuously

using a while/for-loop and not using the continuous-button...

Keep us updated...

TNT

[molsen]

Hi PJM

I have experienced a similar problem that occurs from time to time. After this has happend a few times I tried to figure out what was happening and by coincidense I noticed that after a minimize followed by maximize of the LabVIEW front panel the problem dissapeared!?? And for the record the problem is even there when the VI has stopped and I am working in the diagram. (Here you will notice a slow reaction to mouse clicks).

Looking in the task manager showed that the LV process was using all the CPU power. I have also opened the Process Explorer and tried to find out what LV was dooing but I could not find anything. It all looked normal (port access, disk access, memory usage, TCP comm ...).

I think these problems are related to the LabVIEW <-> Windows interface and as one of the guys on info-labview pointed out it seems to be related to which window has the focus in the Windows operating system and the associated priority assigned by Windows. What NI should look into is why this did not happend in LV 7.1.1. on the same operating system.

Regards

Henrik Molsen

[robijn]

Judging by the name "get all front panel controls", I wrote VI's doing similar things and because that was implemented with property nodes, I set them to run in the UI thread, preventing thread switches. Because it was in 7.1.1 I did not have your slowdown problem, but running in the UI thread gave an enormous speedup already. Maybe the slowdown is caused by a bug in the thread switching. Maybe you can try running this VI in the UI thread (if not already).

Joris

[LAVA 1.0 Content]

PJM could you post VI that has the problem here to the forum?

[Gary Rubin]

I think these problems are related to the LabVIEW <-> Windows interface and as one of the guys on info-labview pointed out it seems to be related to which window has the focus in the Windows operating system and the associated priority assigned by Windows. What NI should look into is why this did not happend in LV 7.1.1. on the same operating system.

I'm not convinced that such behavior did not exist in LV7.1.1, although I do agree that it probably has something to do with the way Labview and Windows play together.

I have a data processing tool that I have written (see screenshot). It essentially preallocates an array (on the order of 200 x 3000 elements) for the big intensity plot, then reads a chunk out of a file and does a bunch of number-crunching to obtain the next column of the intensity plot. There are also some property nodes related to cursors and making controls visible/invisible.

When I was running this on LV711 on Win2K, I found that if I clicked and held, but did not release, the Windows Minimize button, this program would run much faster. My system has since been upgraded(?) to WinXP (still running LV711), and I am unable to duplicate that behavior.

Edit: Adding screenshot.

[PJM_labview]

Hi,

I noticed that since LV7.1 the behaviour of the continuous-button had changed.

e.g. the first call is always TRUE when in LV 7.1 it was only TRUE the first loop.

I'm curious if you could repeat your tests while running your vi conyinuously

using a while/for-loop and not using the continuous-button...

Keep us updated...

TNT

I am pretty convinced that this issue is not linked to the run continuously button. I used the run countinously button to characterize and benchmark the problem, but in regular applications (not running continuously), the problem is still there.

So far I have isolated the problem on VI operating on LV object reference in a for loop or a while loop.

Hi PJM

I have experienced a similar problem that occurs from time to time. After this has happend a few times I tried to figure out what was happening and by coincidense I noticed that after a minimize followed by maximize of the LabVIEW front panel the problem dissapeared!?? And for the record the problem is even there when the VI has stopped and I am working in the diagram. (Here you will notice a slow reaction to mouse clicks).

Looking in the task manager showed that the LV process was using all the CPU power. I have also opened the Process Explorer and tried to find out what LV was dooing but I could not find anything. It all looked normal (port access, disk access, memory usage, TCP comm ...).

I think these problems are related to the LabVIEW <-> Windows interface and as one of the guys on info-labview pointed out it seems to be related to which window has the focus in the Windows operating system and the associated priority assigned by Windows. What NI should look into is why this did not happend in LV 7.1.1. on the same operating system.

Regards

Henrik Molsen

I am not 100% positive, but I don't remember that minimizing and maximizing does improve the performance at all. Next time I am in that situation, I will try to see if this has any impact.

And for the record the problem is even there when the VI has stopped and I am working in the diagram

I have also noticed this (since LV 8.x) but I think this is another issue (more like an editor issue). I dont like this, but I can leave with this since it does not affect my buid app. The other issue I mentioned affect build app as well as app running in the IDE. For example we had report from some user of VIPM that sometime it start/run very slow (I have also experience this as well). This, I definitively linked it to the same issue.

Judging by the name "get all front panel controls", I wrote VI's doing similar things and because that was implemented with property nodes, I set them to run in the UI thread, preventing thread switches. Because it was in 7.1.1 I did not have your slowdown problem, but running in the UI thread gave an enormous speedup already. Maybe the slowdown is caused by a bug in the thread switching. Maybe you can try running this VI in the UI thread (if not already).

Joris

Typically, I hardly ever change thread of my VI, and when I do so, I don't put anything in the UI thread (since there are already so much stuff running in there). The top level caller of the first particular VI I used to characterize the pb (Get all FP Controls) is a UI running in the UI thread and since I left the default thread setting to same as caller, I *think* this VI was running in the UI thread. But, next time I experience the slow down, I will try changing the threading explicitly.

PJM could you post VI that has the problem here to the forum?

Jimi,

Here you go, but please note, this issue happens in many VIs (like the second one in my initial post that I wrote on the spot to characterize the problem).

Download File:post-121-1166801608.vi

I'm not convinced that such behavior did not exist in LV7.1.1, although I do agree that it probably has something to do with the way Labview and Windows play together.

I have a data processing tool that I have written (see screenshot). It essentially preallocates an array (on the order of 200 x 3000 elements) for the big intensity plot, then reads a chunk out of a file and does a bunch of number-crunching to obtain the next column of the intensity plot. There are also some property nodes related to cursors and making controls visible/invisible.

When I was running this on LV711 on Win2K, I found that if I clicked and held, but did not release, the Windows Minimize button, this program would run much faster. My system has since been upgraded(?) to WinXP (still running LV711), and I am unable to duplicate that behavior.

Edit: Adding screenshot.

I never experienced anything even remotely comparable to this slowdown in LV 7.1.1 (of course it does not mean that it is not there).

PJM

[Gary Rubin]

I never experienced anything even remotely comparable to this slowdown in LV 7.1.1 (of course it does not mean that it is not there).

PJM

Sorry, I did not intend to imply that your current problem and what I described in 7.1.1 are comparable. I was commenting on molsen's post and adding that I think I've seen some unexpected behavior associated with Labview and Windows going back before LV8.x.

Gary

[AdamRofer]

I think these problems are related to the LabVIEW <-> Windows interface and as one of the guys on info-labview pointed out it seems to be related to which window has the focus in the Windows operating system and the associated priority assigned by Windows. What NI should look into is why this did not happend in LV 7.1.1. on the same operating system.

I have not experienced the problems people are describing here first hand, yet.

I suggest (a) up the priority (which one shouldn't have to do) or (b) set the background processing equal (which I do almost all the time anyways):

For (a), go to Task Manager (ctrl-alt-del or rt-click on taskbar), rt-click on LabVIEW.exe, select Set Priority and try "AboveNormal" or "High"

For (b), see below.

Anyone experiencing this problem can try to set the windows thread quantum to be equal for all processes (default on Windows Server machines):

(Windows XP)

Go to Control Panel, System

Click on the Advanced tab

Click the Performance --> Settings button

Click on the Advanced tab

Select Background Services

Reboot your machine

(Windows 2K is similar but not exactly the same to get this -- Advanced and then Performance Options)

What (non-server) Windows machines do by default is set the foreground window's thread's quantum (amount of time allowed to execute before trying to switch the processor to another thread) to three times that of other threads with the same priority (whew! say that three times fast).

This means if any threads are running in the background they get to process about 1/4 of your processing power if threads are competing at the same priority level.

Setting the setting I describe above instead forces Windows to calculate all threads that are at the same priority with the same quantum value, ie the same amount of net calculation time.

I have personally found that in any large development system (especially those favoring speed increases) that involves any background processing that is time-critical, enabling this setting can make a large difference. A good example would be any application that you have that runs more than one VI at a time. Try it out (even if you don't have this 8.0 problem) and see if this works better for your applications!

I first discovered this when I created a VI that was calculation-intense and that also displayed the current calculation results. It would speed along happily until I placed any other window in the foreground, when I discovered very quickly that it would chug at a much slower pace.

For a lot more detailed information on this, check this Microsoft link. For slightly less in-depth but easier to read descriptions, try this, this, and this.

As for LabVIEW 8.0's problem here:

- Maybe LabVIEW 8.0 splits computation into other processes that do not hold the foreground thread, thereby suffering calculation time.

- Maybe LabVIEW 8.0 does some fancy threading that counteracts Windows' foreground thread priority

- Maybe the default threading of LabVIEW 8.0 differs slightly from LabVIEW 7

- Maybe (and this is likely) none of these are the culprit and it is something else entirely.

Sorry this is such a long post! Maybe this can help shed some light (or, if not, help someone squeeze some bonus time from their apps!)

[PJM_labview]

Adam,

Thanks for your post. The information you provide are very interesting for some applications. I can already see a couple of system I am working on that could benefit from this thread balancing setting change. Unfortunately, in the case of applications where one does not have access to the end user computer or when the computer changes, asking the end user to go through all these loops so it can run a piece of software (just becasue it is written using LV) is not a realistic option. This need to be fixed.

Thanks

PJM

[AdamRofer]

Adam,

Thanks for your post. The information you provide are very interesting for some applications. I can already see a couple of system I am working on that could benefit from this thread balancing setting change. Unfortunately, in the case of applications where one does not have access to the end user computer or when the computer changes, asking the end user to go through all these loops so it can run a piece of software (just becasue it is written using LV) is not a realistic option. This need to be fixed.

Thanks

PJM

If you're feeling risky, you can try to change the registry key automatically from your programs (as referenced in the first link)... :ninja:

On a more serious note, you're right...my suggestion is most useful for applications on machines that you yourself have administrative access to. Even then, some applications (on an overall windows client machine level) benefit slightly from the default setting.

I merely provide this information here to suggest that this might be a part of what is causing the problems, since it seems to be in the same ballpark. If you change this setting and the problems reduce or disappear, then that might tell us more about what the problem itself might be.

[LAVA 1.0 Content]

I have been experiencing this bug since LabVIEW 8.0. I have not report it until now because it is VERY hard to reproduce.

I reported this to NI Support. This was reported to R&D (#41OF7H00) for further investigation. NI also replied me with an example VI below. To reproduce the problem with the attached VI NI engineer used the following steps:

"Basically, to reproduce the problem: save that example VI (I actually notice it much more prominently using LabVIEW 8.0.1), restart LabVIEW, and open and run the VI continuously (make sure not to resize the window). After some brief 'settling' time, the time should level off."

"Then, I double-click the title bar to maximize the window, and then double click it again to restore the window. The VI keeps running, but it's time to complete the operations about doubles."

"This effect remains until LabVIEW is restarted (not just the VI itself). The problem has something to do with how LabVIEW is posting messages. You should notice that things like setting the processor affinity of LabVIEW.exe (basically mimicking a single-processor machine), or activating some of the menus (like File, Edit, etc...) seem to speed the operations back up."

Download File:post-4014-1167819835.vi

[PJM_labview]

Jimi,

This is great news! :thumbup: Now that there is a fool proof way of reproducing this, it should be fairly easy for NI to fix this.

PJM

[LAVA 1.0 Content]

Jimi,

This is great news! :thumbup: Now that there is a fool proof way of reproducing this, it should be fairly easy for NI to fix this.

PJM

Perhaps you should still check if the problem occurring in the VI sent to me by NI and attached at my above post really matches the problem that you have been facing. I only saw a small decline in speed, about 10-20% which is far from the slow down reduction numbers you got.

[PJM_labview]

Perhaps you should still check if the problem occurring in the VI sent to me by NI and attached at my above post really matches the problem that you have been facing. I only saw a small decline in speed, about 10-20% which is far from the slow down reduction numbers you got.

I did check before I post. I notice a significant slowdown (not in the 100 magnitude, but in the 10 magnitude). Here are screenshot of the before and after runned in LV 8.01.

Before

After

The after is about 10 time slower.

I dont know for sure that this is the same issue, but it look very similar.

Also, recently I seem to have experience the slowdown (in LV 8.01) with VI that do not appear to operate on references (I have not yet characterize this for sure Edit: such as the OpenG "List Directory Recursive__ogtk.vi"). I have not figured out the trigger mechanism (the one you describe for the reference in for loop or while loop, do not trigger it).

PJM

[Dan Press]

I thought I noticed something very strange. I now have two cases of the creepy slowdown.

In once case, there was an OpenG Read INI Cluster happening while a digital I/O task with change detection was running. I actually theorized that it was something to do with the change detect that was causing the slow down. I switched to a polling scheme and it actually helped!

Today, one of our developers saw the slow down during Read INI Cluster without any DAQ tasks running. Increasing the priority of the calling VI seems to help for now.

This is a nasty one!

[Grampa_of_Oliva_n_Eden]

This may not apply but I should point it out anyway.

There seems to be a bug in how LV handles re-using buffers and the "bundle" function with an un-wired input (in LV 8.2).

LV will copy the data associated with the un-used bundle input when to should leave it alone. If the un-used bundle element is an array, this can really impact performance.

See this thread on the NI forum

http://forums.ni.com/ni/board/message?boar...=232654#M232654

to see more details.

Ben