labpython Problem with labpython and numpy

[Gombo]

I was requested to provide Labview bindings for an app written in mostly in python and I've found LabPython tobe a blessing. However, I run into a nasty problem and I can't really figure who is to blame.

 

In a nutshell: LabPython does not unload python server dll and dll of imported modules (python27.dll and *.pyd) after "PYTHON Close Session.vi" have been executed and VI stopped . For most cases it causes no problem, but numpy module causes LabView to hang when "import numpy" is executed on the second run.

 

I'll greatly appreciate any help with the matter, because I can't even tell who is to blame - labpython, numpy, or LabView

 

In details:

1. Here is a test case

2. When the VI is loaded into memory, only labpython.dll have been loaded - OK.

 

3. When the VI have been run the python27.dll and a lot of numpy-related dll's are loaded and remain in memory when the VI stops.

 

4. If I quit the VI, labpython.dll is unloaded, but all other dll's remains - this definitely seems to be wrong.

 

5.  The second run generates an error and hangs on closing

Some handles are open at this moment:

 

6. If I do not quit the VI before second run (skip step 4), there is no error and it hangs on the first labpython VI after "import numpy" execution.

 

 

 

I tested this behavior with

1) Labview 2012, labpython 4.0.0.4, python 2.7, numpy 1.7.1 and 1.6.1

2) Labview 2012, labpython 4.0.0.4, python 2.4, numpy 1.4.1

3) Labview 8.5, labpython 1.2, python 2.7, numpy 1.6.1

 

Test VI is attached to the post.

 

 

test.vi

[Darin]

The short story is that numpy is not compatible with multiple interpreters being launched in the same process (say LV IDE).  I have not seen the code for LabPython but I will wager dollars to donuts that there are calls to Py_NewInterpreter() in there.  This would explain the issue.  A quick test would be to run the same code from two different VIs.  I bet that the second VI will hang in the same way that the first one would on its second run.  This points to a single global session having multiple interpreters.

 

You should be able to get around it by building executables, or disabling threads in numpy (yuk).

 

As for blame, there is a little for everyone, but I tend to scale by cost, so I am not very hard on things which are free.  Python made rules for extension modules and numpy followed those rules.  At the same time, Python supports a method for launching multiple interpreters in the same session which LabPython takes advantage of apparently. 

[Gombo]

Thank you for prompt response, it got me started.

I've peeked into labpython code and, sure, you get your bet - Py_NewInterpreter() calls are there. Labpython seems to be specifically designed to allow several concurrent sessions (interpreters). On the other hand,  "PYTHON Close Session.vi" does call Py_EndInterpreter(), so on the surface of things there should be no old interpreter in the second run. Moreover, Labview should clean things up when the VI is unloaded from memory, shouldn't it?

Anyway, I'm going to try to make a call to Py_Finalize() at the end of the vi run. Would it help, what do you think?

[Gombo]

Well, approach suggested above didn't work. I have no idea why, but python dlls still did not unload  (I called PyCloseHost() function from lvpython.c, which includes Py_Finalize()) and second run would not work. The only improvement was that instead of hanging up the second attempt caused an exception, which was handled by Labview, but I still needed to close it after each run.

 

Anyway, I found a workaround, which suits me: it turned out that it is generally OK to leave out call to "PYTHON Close Session.vi". You can reconnect to the same interpreter on the second run using state machine like this:

 

 

(you get all your variables back), or even simply create a New Session each time, but that would lead to a memory leak.

 

(BTW, it turned out that the source would not compile because of a typo, so simple but difficult to locate for a person unfamiliar with LabView develoment that it seems to be deliberate.  )

 

 

 

[Darin]

I was going to suggest the single interpreter approach, did not know if the variable persistence would be a feature or a bug.  I figured it would be ok in most situations since you should be initializing values anyway.

[Gombo]

OK, things turned out to be even stranger.  The real perpetrator turned out to be "PYTHON Set Server Path.vi".  It attempts to restart server and, evidently, that messes things up somehow. If it is not used (it is really have to be run once per installation to write the path into config), then even with "PYTHON Close Session.vi" subsequent runs go fine and you can have concurrent session with numpy (at least, with "import numpy" - I have not tested any further). The memory leak persists, though, so single interpreter setup is probably recommended.

 

It would be nice to update default Python version from 2.4 to 2.7, I think.

[Rolf Kalbermatter]

Well, approach suggested above didn't work. I have no idea why, but python dlls still did not unload  (I called PyCloseHost() function from lvpython.c, which includes Py_Finalize()) and second run would not work. The only improvement was that instead of hanging up the second attempt caused an exception, which was handled by Labview, but I still needed to close it after each run.

 

Anyway, I found a workaround, which suits me: it turned out that it is generally OK to leave out call to "PYTHON Close Session.vi". You can reconnect to the same interpreter on the second run using state machine like this:

 

test4.PNG

 

(you get all your variables back), or even simply create a New Session each time, but that would lead to a memory leak.

 

(BTW, it turned out that the source would not compile because of a typo, so simple but difficult to locate for a person unfamiliar with LabView develoment that it seems to be deliberate.  )

 

I can assure you that I have not put in any deliberate typo in the source code, both in the C or LabVIEW part. I can't guarantee that something might be messed up somehow because of some settings on my machine at that time, that would not work out of the box on a standard installation, but there was no intention to make LabPython not compile for others. Would be also pretty useless, given that the ENTIRE source code is actually openly accessible.

 

As to why LabPython is the way it is now, that has a long story. The first idea was simply to have a LabVIEW Python Script server so that you could execute Python scripts in the LabVIEW formula script node. While this was a fun idea, the practical usefullness of this solution turned out to be rather limited, since the script text was in fact compiled into the VI and therefore not possible to change at runtime.

 

Looking at it a bit more it seemed quite trivial to simply export some more of the DLL functions and write LabVIEW VIs to access them, to have a truly dynamic script execution. And the dynamic linking to the Python server was a further afterthought to the whole story, since Python had regular updates and each version used a different DLL name.

 

But working with the Python community turned out a bit complicated at that time. While Python both supports embedding other environments into it as well as embedding Python into other systems (like here in LabVIEW) support for the latter was considered highly unimportant and various discussions about improving that interface where shutdown as undesirable. So once I got it working in LabVIEW I sort of lost every drive to do anything more with it.

 

The fact that you seem to be the first noticing a typo (after more than 10 years) only points out that nobody apparently has bothered in all that time to even take a serious look beyond the Python VIs, despite that the entire source code is openly available. Also that typo might be rather something that was 12 years ago not a typo, with Visual C 6.0 and LabVIEW 6.0 cintools.

[ShaunR]

embedding Python into other systems (like here in LabVIEW) support for the latter was considered highly unimportant and various discussions about improving that interface where shutdown as undesirable.

That has been my experience with the entire Linux community. When you get the answer "install linux" in answer to any windows question - You just wonder what they do as a real job.

[Gombo]

2roflk:  I'm sorry for suggesting that the error was deliberate, please accept my apologies. I was a bit bugged because I am unfamiliar with cintools and it took me a while figuring out the problem (which was most probably caused by a change in cintools API) and I took a wrong way a phrase from README about why would you bother compiling the code. Sorry again.

 

Th problem is caused by line 105 in lvsapi.h. It reads

 

PrivateP(lvsnInstance);

 

but should be

 

NIPrivatePtr(lvsnInstance);

 

for my LabView 2012 version of cintools.

 

May be you can correct it and, by the way, may I suggest changing default python version to 2.7?

 

Anyway, thank you for your work, it is a great help.

[Yair]

(BTW, it turned out that the source would not compile because of a typo, so simple but difficult to locate for a person unfamiliar with LabView develoment that it seems to be deliberate.  )

 

It might actually be much harder for someone who's familiar mainly with LabVIEW, as LV doesn't have the concept of typos breaking the compilation. Essentially, it's impossible to create such a problem in LV, so you would have to be aware of other languages to realize that could be a problem.

[Rolf Kalbermatter]

2roflk:  I'm sorry for suggesting that the error was deliberate, please accept my apologies. I was a bit bugged because I am unfamiliar with cintools and it took me a while figuring out the problem (which was most probably caused by a change in cintools API) and I took a wrong way a phrase from README about why would you bother compiling the code. Sorry again.

 

Th problem is caused by line 105 in lvsapi.h. It reads

 

PrivateP(lvsnInstance);

 

but should be

 

NIPrivatePtr(lvsnInstance);

 

for my LabView 2012 version of cintools.

 

May be you can correct it and, by the way, may I suggest changing default python version to 2.7?

 

Anyway, thank you for your work, it is a great help.

 

The NIPrivatePtr() macro definitely is a LabVIEW 2012 invention. That didn't exist before. In LabVIEW 8.5 until 2011 this was called LV_PRIVATE_POINTER() and before that PrivatP(). So as you can see the bad guys here are the NI developers, devilish changing macro definitions over the course of decades.

 

The change from PrivateP() to LV_PRIVATE_POINTER() is most likely due to supporting a development environment (like maybe a new MacOSX SDK) that introduced exactly this macro for its own use and then NI has to change its own macros to prevent a name collision. Them forcing parties like Apple or Microsoft to change their macro definitions because of having longer rights on the naming is not an option. 

 

The last change looks a bit bogus. Most likely the macro in 8.5 was added by a Microsoft inspired hungarian notion loving guy or gal in the LabVIEW team, while someone in 2012 decided that this notion is evil and changed it to the more Apple like Upper/Lowercase notion, that is used throughout the LabVIEW source in most places (since LabVIEW was originally written on the Mac and then ported to other platforms later).

 

 

Changing LabPython such that it can compile with all the different LabVIEW cintools header versions is a nice exercise in preprocessor macro magic, and one I honestly do not feel any inclinations to do at this moment. If (and the emphasis is here on IF) I ever would need to recompile LabPython for any reason, I would still need the old definitions since I generally use cintools headers that are based mostly on the LabVIEW 7.1 definitions, with a few minor edits to support 64 bit compilation.