Occasional loop hangup in executable - how to find the source?

[george seifert]

I have an executable that seems to be hanging up in a timed loop occasionally (1 sec loop time). There are several things happening in the loop that could cause this, but I'm not quite sure how to pinpoint it while it's running as an executable. I'm reading up to 4 serial devices and the problem is definitely worse when I'm reading more than 1 device. I'm also storing data to a TDMS file during the loop. The amount of data stored depends on how many serial devices I'm reading. So there's not a good way to separate the two effects. I suppose one thing I can do is disable the data saving to see if that makes a difference. Any other ideas on how to pinpoint a problem like this in an executable?

Oh, one other thing I noticed when this happens is that the PC appears to hang (I know when it happens because I'm graphing the data). I can't move windows around or scroll things. Then suddenly everything frees up and is fine. I watched the Task Manager when it happens because I figured I'd see the CPU usage shoot way up, but it didn't.

The VI is kind of way too big to post and nobody's going to want to try to figure it out anyway. So any general hints would be welcome.

George

[Grampa_of_Oliva_n_Eden]

You already have the first, get rid of the file writes in the loop.

Second, shut down viris checking. Those criters lock-up every PC I have touched when ever they think they should be doing something.

Ben

[Tomi Maila]

In general code hangs because of race conditions or because of LabVIEW or driver bug. The race condition may happen also within external code. But as always the nobody is able to help you without the code. So unless you cannot post failing code, at least I'm incapable of helping you.

Tomi

[Neville D]

QUOTE(george seifert @ Jul 30 2007, 01:03 PM)

I have an executable that seems to be hanging up in a timed loop occasionally (1 sec loop time).

The VI is kind of way too big to post and nobody's going to want to try to figure it out anyway. So any general hints would be welcome.

George

How about enabling debugging for executables, and trying to track it down using that ? I am assuming you are using LV >8.x

Neville.

[george seifert]

QUOTE(Ben @ Jul 30 2007, 03:08 PM)

You already have the first, get rid of the file writes in the loop.

Second, shut down viris checking. Those criters lock-up every PC I have touched when ever they think they should be doing something.

Ben

I can't really get rid of the writes in the loop. It should be able to handle it. I mean that's why the TDMS file structure was implemented. I need to make sure I have data saved along the way in case of a system crash.

The virus checker is a good idea though. IT just recently gave us a new Symantec Antivirus. Things were working fine before that came along. Now I just have to figure out how to disable the thing.

George

[MikaelH]

George

I've seen timed loops in some cases just stop for a while and then get going after a while.

(that's why I try to avoid them ;-)

I don't know if this could be your case.

But you could try to use the old fashion while-loop, and measure the loop time for every loop

and see how long time every loop take.

Mikael

[eaolson]

QUOTE(george seifert @ Jul 30 2007, 03:03 PM)

Any other ideas on how to pinpoint a problem like this in an executable?

It would probably require a lot of rearrangement to your code, but you could write a string to a log file between each step. "Read COM1, Read COM2, Write TDMS, Done" and so forth. At least it might tell you where it was getting stuck. It's not clear if you mean that one loop iteration takes longer than previous ones or if the whole thing just stops.

Are you building arrays as you go? I've had loops slow down if they kept building larger and larger arrays. How much data are you collecting? If a large chunk of memory has to be swapped to disk that can slow things down, depending on your disk.

[Guillaume Lessard]

QUOTE(george seifert @ Jul 30 2007, 02:45 PM)

I can't really get rid of the writes in the loop. It should be able to handle it. I mean that's why the TDMS file structure was implemented. I need to make sure I have data saved along the way in case of a system crash.

You can send your data along to another, lower-priority loop using a queue. File operations are some of the first things to send to another thread if you ever have problems with code that hiccups...

[george seifert]

QUOTE(eaolson @ Jul 30 2007, 05:40 PM)

Are you building arrays as you go? I've had loops slow down if they kept building larger and larger arrays.

I was a good boy and allocated my arrays before starting the loop.

I tried disabling disk writes but the problem persists.

The PC definitely hangs when this happens. CPU usage shoots to 100% for up to 10 seconds. Unfortunately I can't use the Windows Task Manager to see which process jumps in CPU usage because no updates happen anywhere during the pause. It happens in irregular intervals too. Sometimes up to 100 seconds apart, sometimes 10 seconds. Is there any way to tell if my program is causing this or is it some other Windows service?

George

[Tomi Maila]

QUOTE(george seifert @ Jul 31 2007, 04:22 PM)

I was a good boy and allocated my arrays before starting the loop.

I tried disabling disk writes but the problem persists.

The PC definitely hangs when this happens. CPU usage shoots to 100% for up to 10 seconds. Unfortunately I can't use the Windows Task Manager to see which process jumps in CPU usage because no updates happen anywhere during the pause. It happens in irregular intervals too. Sometimes up to 100 seconds apart, sometimes 10 seconds. Is there any way to tell if my program is causing this or is it some other Windows service?

George

Sounds like a driver issue. Only kernel level processes can hang the whole operating system. In windows most non-windows kernel level processes are device drivers. So I presume there is an issue with one of the device drivers you either directly or indirectly use in your project. I don't know if timed loop uses some sort of kernel level process for the timing, that may very well be. Someone from NI can maybe say something about this.

[Ton Plomp]

Have a look here

Perfmon can store this info. Select the processes you suspect and hope you find it!

Ton

[Grampa_of_Oliva_n_Eden]

I could write on this topic for days and still not cover all of the bases.

1) In the Task Manager you can set your update interval to something small (5 seconds?).

2) Still task manager, you can select more collumns to view in the "process" tab. Look at everything (click on header to sort on that column).

3) Show kernal will let you see if the kernal is using CPU durring your interuptions.

No one has aked "How is your memory?" Running with more memory than you have physical memory will cuaswe interuptions durring page-faults.

Here are some notes on how I got the hardware timed timed loop to run without missing a beat.

"

After some experiments and punching a lot of buttons, it appears that multiple timed loops can run under Windows with HARDWARE timing.

This does not seem to be possible without adjusting the environment to help this happen.

The following is a list of things that can help you "adjust" Windoze do its job.

1) Set Windows to optimize background services.

Windows by default will attempt to optimize its scheduling to make foreground processes perform well. For single threaded applications this is fine but for LabVIEW, some of its background threads can suffer.

A) Start >>> Control Panel

B) Open "System" and choose the "Advanced" tab

C) In the "Performance" section click "Settings"

D) On the "Advanced" tab in the "Processor Scheduling" section select "Background Services"

E) Save or apply everything.

2) Stop Indexing services.

Windows will by default do file indexing. File indexing is functionality that tells windows to maintain in memory a cache of files on a disk or in a folder. This makes files show up in the Explorer faster and also makes searching for files faster.

A) Start >>> Accessories >>> Windows Explorer

B) Right-click C: drive and select "Properties"

C) On the "General" tab un-check "Allow Indexing Service to Index This Drive for Fast File Searching"

D) Save and apply everything

3) Set LabVIEW priority in Task manager. Windows will by default treat all processes that are ready to run as peers and will share the CPU evenly.

A) Open the Task Manager and select the "Processes" tab.

B) Locate "LabVIEW.exe" in the list of processes.

C) Right -click "LabVIEW.exe" and choose Set Priority >>> High

Note: Setting LV to run at higher than "High" will put it on equal footing with interrupt service routines etc. This could result in the machine "locking up" because LV is using all of the CPU and there is no opportunity to respond to interrupts from your mouse moving.

4) Shutdown Virus Checking. Virus checking gets its hooks into everything!

5) Disconnect Ethernet cable. Ethernet traffic requires intervention by the OS. No cable, no traffic, no distractions.

6) Set the VI properties of the sub-VI that will run the Timed Loop to "Priority = Time Critical" and "Execution Thread = Data Acquisition".

7) Configure hardware Timing source to run at 2000 Hz.

8) Set "period" of timed loop = 2 (i.e. 2000 Hz clock / period of 2 = 1000 Hz)

Now wasn't that easy?

I have tested the above using the instructor machine with LV8 running three timed loops at 1000Hz for 20 minutes with no missed iterations and no "Finished Late" indications where the loop failed to start on time.

"

Please report back on your findings so we can do a "group learn".

Have fun!

Ben

[george seifert]

QUOTE(tcplomp @ Jul 31 2007, 08:37 AM)

Have a look http://forums.lavag.org/Monitor-CPU-load-t8438.html' target="_blank">here

Perfmon can store this info. Select the processes you suspect and hope you find it!

Ton

Thanks. Do you have any hints as to which process might be the best to monitor?

George

[Gary Rubin]

QUOTE(Ben @ Jul 31 2007, 09:58 AM)

1) Set Windows to optimize background services.

Windows by default will attempt to optimize its scheduling to make foreground processes perform well. For single threaded applications this is fine but for LabVIEW, some of its background threads can suffer.

Ben,

Thank you for reminding me of this. I wonder if this explains why my processing seems to bog down, yet my Core 2 processor never shows more than about 60% usage... Certainly something to try.

Gary

[george seifert]

QUOTE(Ben @ Jul 31 2007, 08:58 AM)

Please report back on your findings so we can do a "group learn".

Thanks for all the help everyone. After disabling various bits of code I finally tried replacing my timed loop with a regular loop. Bingo! I don't know why I keep using those things. They've bitten me a few times - especially when you've got two running in parallel.

George

[Grampa_of_Oliva_n_Eden]

QUOTE(george seifert @ Jul 31 2007, 03:03 PM)

...finally tried replacing my timed loop with a regular loop. Bingo! ...

George

Timed loop eh?

I was using hardware timing in my case.

Am I correct in assuming you were not using hardware timing?

Ben

[Tomi Maila]

QUOTE(george seifert @ Jul 31 2007, 10:03 PM)

Thanks for all the help everyone. After disabling various bits of code I finally tried replacing my timed loop with a regular loop. Bingo! I don't know why I keep using those things. They've bitten me a few times - especially when you've got two running in parallel.

Would you be so kind that you would submit a bug report to NI. This can very well be LabVIEW related issue and it's better that they have a code that can reproduce the issue so that they can investigate the issue and possibly even fix it.

[george seifert]

QUOTE(Ben @ Jul 31 2007, 02:13 PM)

Am I correct in assuming you were not using hardware timing?

Ben

You are correct sir.

[Grampa_of_Oliva_n_Eden]

QUOTE(Gary Rubin @ Jul 31 2007, 10:15 AM)

Ben,

Thank you for reminding me of this. I wonder if this explains why my processing seems to bog down, yet my Core 2 processor never shows more than about 60% usage... Certainly something to try.

Gary

Again with the virus checking already! :headbang:

I do not understand all of the details but virus checkers have their hooks into everything and when they kick in, they seem to go into kernal mode and do not allow other threads to run until they have handled this "emergency situation", like opening a file in Notepad or browsing to a new folder.

I swear I used to be able to get my PC performance to flat-line priopior to performance profiling, but now, since virus checking, I have been stymied to get the PC to just do nothing.

:headbang:

Ben

[Gary Rubin]

QUOTE(Ben @ Jul 31 2007, 07:29 PM)

Again with the virus checking already! :headbang:

I do not understand all of the details but virus checkers have their hooks into everything and when they kick in, they seem to go into kernal mode and do not allow other threads to run until they have handled this "emergency situation", like opening a file in Notepad or browsing to a new folder.

I swear I used to be able to get my PC performance to flat-line priopior to performance profiling, but now, since virus checking, I have been stymied to get the PC to just do nothing.

:headbang:

Ben

No virus checking on this machine - it's a stand-alone system with no external connections, so we didn't install any antivirus...

[Yair]

QUOTE(Ben @ Aug 1 2007, 02:29 AM)

since virus checking, I have been stymied to get the PC to just do nothing.

That's easy Ben. There's a big round button on the front of the computer. Pushing it should disable the anti virus program.

[Grampa_of_Oliva_n_Eden]

QUOTE(yen @ Aug 1 2007, 01:26 PM)

That's easy Ben. There's a big round button on the front of the computer. Pushing it should disable the anti virus program.

Are you talking about the one labeled "X2" ?

Ben

[Yair]

It should have http://forums.lavag.org/index.php?act=attach&type=post&id=6512 on it. That's the international symbol for anti-virus disabling.