OpenG Package: LabPython weirdness

[gb119]

I'm seeing a slightly weird (and inconvenient) bug with LabPython...

I've a Windows 7 64bit machine with 32bit LabVIEW 2012 and 32 bit LabVIEW2013 and both 64bit and 32bit Python 2.7 installed (the former I use standalone, the latter is just for linking to some LabVIEW code that makes use of LabPython). My  32bit python27.dll therefore lives in C:\Windows\SysWOW64\ and I point LabPython to it (for both versions of LabVIEW) and can see in both labview.ini files that the correct key has been added.

 

On LabVIEW 2012, everything work s absolutely fine.

On LabVIEW 2013, if I load a VI containing the Python script node, it complains that it can't load lvpython.dll via a debug window message. This is uually followed by a fatal exception, which is not a massive surprise ! If I use the LabVIEW vi API to the LabPython, then I don't have load errors and LabVIEW crashes about as much as it normally does.

 

The only other LabVIEW 2013 setup that I've played with are 32bit Windows 7 machines running 32bit LabVIEW 2013 only, and they seem all fine.

 

My current workaround is to load the VI that creates a session in the LabVIEW vi API mode of LabPython operation, thus forcing the dll to load, which then seems to let me load and use the script node (which sort of makes sense as Windows will use the already loaded dll rather than loading it again) - but it's a little messy and doesn't explain why LabVIEW 2012 works just fine....

 

Any thoughts anyone as to what's going on ?

[LogMAN]

Some time ago I tried to launch the On-Screen keyboard that ships with Windows on a 64bit system from 32bit LabVIEW (starting "osk.exe" via shell command or console in LabVIEW). That didn't work even though I explicitly pointed to %windir%\SysWOW64. The issue is Windows redirecting calls to the system32 and SysWOW64 folder depending on your application bitness. This is called File System Redirection. Disabling File System Redirection allows you to call files in the SysWOW64 folder instead of being redirected to the system32 folder.

 

Call this function and try it again: http://msdn.microsoft.com/en-us/library/windows/desktop/aa365743(v=vs.85).aspx
Don't forget to revert redirection once your call is finished: http://msdn.microsoft.com/en-us/library/windows/desktop/aa365745(v=vs.85).aspx

 

Read more here: http://msdn.microsoft.com/en-us/library/windows/desktop/aa384187(v=vs.85).aspx

 

Oh yeah, be sure to execute your VI as well as the DLL calls in one specific thread, or else these functions will have no effect!

Edited September 23, 2014 by LogMAN

[X___]

That might qualify as thread-resurrection-and-hijack, but is there any chance to get a 64-bit compatible version of labpython?

Currently, after correcting the erroneous path of lvpython.dll in the "PYTHON New Session" VI, loading fails with this pretty explicit message:

 

[Rolf Kalbermatter]

It might be possible but it is far from just a recompilation of the code. The code was written at a time where nobody was thinking about 64 bit CPUs and no standards existed how one should prepare for that eventualiity. Also it is possible that the script interface in LabVIEW has been cleansed of support for older API standards for the 64 bit version. LabPython uses the first version of that API, but if the 64 bit version of LabVIEW doesn't support that anymore, then the new version documentation would need to be gotten from NI. This is not an official public API.

[X___]

I'll take that as a no...

[Rolf Kalbermatter]

I'll take that as a no...

 

That's pretty harsh!   The source code is on sourceforge and there is nobody preventing other people from accessing it and attempt to port it to 64 Bit LabVIEW. It won't be easy but given enough determination it is certainly doable.

[X___]

Including comments such as "/* Oh well this can get messy."?

You've probably graduated from the Concise-Better-Yet-No-Comments School, like I did...

I'll leave it to the experts as I haven't the slightest clue which API you are talking about.

[Argold]

On 6/30/2015 at 0:32 AM, rolfk said:

 

The source code is on sourceforge and there is nobody preventing other people from accessing it and attempt to port it to 64 Bit LabVIEW. It won't be easy but given enough determination it is certainly doable.

I'm confused.  There is also a 4.0.0.4 version on Github.   Which is the one that is available on the LabView tools network via VIPM?

Edited January 8, 2017 by Argold

[Rolf Kalbermatter]

I have not  moved anything to github and am very unlikely to do. Besides that I find git not very easy to use I have come across way to many projects that were taken from somewhere, put on github and then abandoned.

The 4.0.0.4 version of LabPython is on the old CVS repository for the LabPython project on sourceforge, but I did add the LabPython project with some initial improvements to the shared library to the newer SVN repository of the OpenG Toolkit project on sourceforge. That is as far as I'm concerned the current canonical version of LabPython, although no new release package has been created for a few reasons:

-The changes I did to the C code are only a few minimal imrovements to make LabPython compile with the Python 2.7 headers. Only very brief testing has been done with that. More changes to the C code and a lot more testing would be needed to make LabPython compatible with Python 3.x.

- More changes need to be made to the code to allow it to properly work in a 64 bit environment. Currently the pointer to the LabPython private management structure which also maintains the interpreter state is directly passed to LabVIEW and then treated as a typed log file refnum. LabVIEW refnums however are 32 bit integers, so a 64 bit pointer will not fit into that. The quick and dirty fix is to change the refnum to a 64 bit integer and configure all CLNs to pass it as a pointer sized variable to the shared library. But that will only work fro LabVIEW 2009 on onwards which probably isn't a big issue anymore. The bigger issue is that a simple integer will not prevent a newby user to wire just about anything to the control and cause the shared library to crash hard when it tries to access the invalid pointer.

-There is currently serious problem when trying to use non-thread safe Python modules like numpy and similar from within LabPython. These modules assume that its functions are always executed from within the same OS thread and context. LabPython doesn't enforce that and LabVIEW happily will call it from multiple threads if possible, which makes those modules simply fail to work. LabPython tries to use the interpreter lock that the Python API does provide, but either that is not enough or they changed something between Python 2.3/2.4 and later versions in this respect that makes LabPython not correctly use this lock. Getting this part debugged will be a major investment. Documentation about the interpreter lock and thread safety of the Python interpreter are scarce and inconsistent.

Edited January 8, 2017 by rolfk

[JKSH]

4 hours ago, rolfk said:

- More changes need to be made to the code to allow it to properly work in a 64 bit environment. Currently the pointer to the LabPython private management structure which also maintains the interpreter state is directly passed to LabVIEW and then treated as a typed log file refnum. LabVIEW refnums however are 32 bit integers, so a 64 bit pointer will not fit into that. The quick and dirty fix is to change the refnum to a 64 bit integer and configure all CLNs to pass it as a pointer sized variable to the shared library. But that will only work fro LabVIEW 2009 on onwards which probably isn't a big issue anymore. The bigger issue is that a simple integer will not prevent a newby user to wire just about anything to the control and cause the shared library to crash hard when it tries to access the invalid pointer..

Would you consider encapsulating the 64-bit "refnum" in an LVClass for the user can pass around?

[ShaunR]

8 hours ago, JKSH said:

Would you consider encapsulating the 64-bit "refnum" in an LVClass for the user can pass around?

It's not as straightforward as that. The issues are fairly straight forward to resolve for any one version but since the Linux community don't see any problem with breaking ABIs at the drop of a hat and that backwards compatibility is a fairy tale; Rolf has already lost the battle for keeping it current,  requiring significant effort with almost every release. If I was in his shoes, I would make it 64 bit compliant, make all the CLFNs orange nodes and tell the users they have to use a specific version of Python or patch it themselves (it's open source after all). Then freeze it for others to champion. Why should Rolf shoulder the problems caused by the intransigence of a completely different language that is nowhere near as disciplined as he or NI are?

Unless Rolf has a business need in one of his projects, this is just a millstone caused by open source altruism. We can hope he finds the time and energy to update it but I suspect he would have done so already if that was the case. This is Rolfs very polite way of saying "here are all the problems I know about, fix it yourself if you are good enough".

[JKSH]

8 hours ago, ShaunR said:

It's not as straightforward as that. The issues are fairly straight forward to resolve for any one version

This small part is straightforward enough, no? The process of making LabPython 64-bit friendly is orthogonal to process of dealing with compatibility breaks in 3rd party components. I myself have not used LabPython (or even Python), so I wasn't commenting (and won't comment) on how the project should/shouldn't be carried forward.

Rolf suggested a quick and dirty fix for a specific issue (i.e. 64-bit support), which exposes a large gun that a user could shoot themselves in the foot with. I suggested an alternative quick and dirty fix that hides this gun, and asked what he (and anyone else reading this thread) thought of the idea.

 

8 hours ago, ShaunR said:

Rolf has already lost the battle for keeping it current,  requiring significant effort with almost every release.... Why should Rolf shoulder the problems caused by the intransigence of a completely different language that is nowhere near as disciplined as he or NI are?

Unless Rolf has a business need in one of his projects, this is just a millstone caused by open source altruism. We can hope he finds the time and energy to update it but I suspect he would have done so already if that was the case. This is Rolfs very polite way of saying "here are all the problems I know about, fix it yourself if you are good enough".

Understood.

OK, it's now recorded here for posterity: 2 different suggestions on how to achieve 64-bit support, for anyone who's happy to work on LabPython.

 

8 hours ago, ShaunR said:

If I was in his shoes, I would make it 64 bit compliant, make all the CLFNs orange nodes and tell the users they have to use a specific version of Python or patch it themselves (it's open source after all). Then freeze it for others to champion.

That sounds sensible.

 

8 hours ago, ShaunR said:

since the Linux community don't see any problem with breaking ABIs at the drop of a hat and that backwards compatibility is a fairy tale;

Such blanket statements are a bit unfair...

• https://www.python.org/dev/peps/pep-0384/
• https://community.kde.org/Policies/Binary_Compatibility_Issues_With_C%2B%2B
• https://en.wikipedia.org/wiki/Linux_Standard_Base
• http://forums.ni.com/t5/Machine-Vision/IMAQdx-14-5-breaks-compatibility-with-IMAQdx-14-0/m-p/3168479

Granted, what's now in place in the Python and Linux worlds probably isn't enough to smooth things out for LabPython, but snarky remarks help nobody.

[Rolf Kalbermatter]

On 1/10/2017 at 7:33 AM, JKSH said:

Such blanket statements are a bit unfair...

• https://www.python.org/dev/peps/pep-0384/
• https://community.kde.org/Policies/Binary_Compatibility_Issues_With_C%2B%2B
• https://en.wikipedia.org/wiki/Linux_Standard_Base
• http://forums.ni.com/t5/Machine-Vision/IMAQdx-14-5-breaks-compatibility-with-IMAQdx-14-0/m-p/3168479

Granted, what's now in place in the Python and Linux worlds probably isn't enough to smooth things out for LabPython, but snarky remarks help nobody.

While I think that the remark in itself wasn't helpful I do understand where it comes from. In many open source projects trying to interface to them from another software is like trying to continuously keep a moving target in focus. Granted, maintaining backwards compatiblity can be a painful process and there is something to say about starting with a clean slate at some point. And of course often the open source programmer is dedicating his own free time to the cause, so it is really his decision if he rather spends it to keep the software compatible or develop new exciting features and change whatever is needed to change during that without considering the possible consequences.

Still I think a bit more discipline wouldn't normally hurt. It's sometimes the difference between a cool but for many applications pretty unusable solution and a really helpful and useful piece of software.

Another thing are changes made on purpose for the sake of disallowing their use from certain types of clients. That I have a pretty ambivalent feeling about. It seldom prevents what they try to block, but it causes lots of mischief for the users.

The IMAQdx link you provided refers to a forward compatibility issue. That is something that is very difficult to provide. There are techniques to help with that somewhat but they more often than not tend to take up more code and complexity than the entire rest of the library, so in short basically never worth the effort. Working in regulated industries might be an exception here.

Edited January 11, 2017 by rolfk

[ShaunR]

1 hour ago, rolfk said:

While I think that the remark in itself wasn't helpful I do understand where it comes from.

If they don't listen to their guru, Linus Torvalds, they deserve "snarky" comments as far as I'm concerned. Let's be clear, here. It is a community problem that Python is just the tip of the iceberg and perpetuates because of its ubiquitous use. I have many Linux friends that complain like hell that games are often not ported to Linux and when I tell them why, they get all defensive and demand cross-platform developers yield and point blank refuse to improve the platform (even though they have the skills to do so). When seeking support, after being told to read the manual, I am always told to patch/port it myself, send it to them and they "will think about it". So ...... what is good for the goose is good for the gander, right? (Not the guys/gals who maintain CMake. They are a drop of water in the dessert)

As you will probably be aware, most of my toolkits are backwards compatible to 2009 and work on Windows, Mac and Linux (mainly due to LabVIEWs guarantee) but I removed support for Linux with those that use external libraries for exactly these reasons.

I wouldn't blame you in the slightest for putting LabPython back into your private toolbox because it is more hassle than it is worth and they (the Linux community) aren't prepared to give an inch to improve the platform.

Of course. They could ask you or your employer for a quote to upgrade to a certain Python version on a particular distribution version. Perhaps start a Kickstarter campaign?

 

[smithd]

1 hour ago, ShaunR said:

I have many Linux friends that complain like hell that games are often not ported to Linux and when I tell them why, they get all defensive and demand cross-platform developers yield and point blank refuse to improve the platform (even though they have the skills to do so).

I saw a hissy fit recently between the AMD folks and linux core folks. As I understand it amd said basically 'we're making our driver this way because you keep changing the function interface for video drivers' and the kernel folks said 'if you do that we won't accept it'. Then AMD made it their way so they could finally have a proper functioning gpu driver on linux while not sinking a ridiculous amount of money into a platform that such a small number of people use. Then the kernel folks said 'nopenopenopenope' and rejected it. And then I lol'd.

[Rolf Kalbermatter]

On 1/11/2017 at 5:10 PM, smithd said:

I saw a hissy fit recently between the AMD folks and linux core folks. As I understand it amd said basically 'we're making our driver this way because you keep changing the function interface for video drivers' and the kernel folks said 'if you do that we won't accept it'. Then AMD made it their way so they could finally have a proper functioning gpu driver on linux while not sinking a ridiculous amount of money into a platform that such a small number of people use. Then the kernel folks said 'nopenopenopenope' and rejected it. And then I lol'd.

Reading a bit further on the technical mailing list it seems there was an initial clash of some sorts between a few people who were on the two opposite ends of wanting to get code into the kernel and wanting to maintain a clean kernel source code base. Both points are pretty understandable and both sides have sort of resolved to some name calling in the initial phase. Then they sat down together and actually started working through it in a pretty constructive manner. None of the latter seems to have been picked up by the mainstream slashdotted media, and the quick and often snarky comments of more or very often less knowledgeable people, concentrated mostly on that initial fallout.

It's very understandable that the kernel maintainer didn't want to commit 100k lines of code into the kernel code just like that. Apparently the AMD guys didn't expect that to happen anyways and actually were more of proposing the code as it was as a first RFC style submission, after having worked a bit to long in the shadows on the huge code base. They didn't however make this clear enough when submitting the code. And the maintainer was a bit quick and short in his answer.

In hindsight the way this was handled from both sides isn't necessarily optimal, but there is no way this code would have been committed if Linus himself would still control the kernel sources. Yes developing for the Linux kernel is a very painful process if you are used to other device driver development such as on Windows. There you develop against a rigidly defined (albeit also frequently changing) kernel device interface. In Windows 3.1 and Windows 95 days you absolutely had to write assembly code to be able to write a device driver (VxD), in Windows 98 and 2000 they introduced the WDM model which replaced both the VxD and NT driver model completely. With Windows Vista the WDF framework was introduced, which is supposed to take away many of the shortcomings that the WDM model had for the ever increasing complexity of interactions between hardware drivers, such as power saving operations, suspend, or also IO cancelation or pure user mode drivers.

In the Linux kernel, a driver is normally an inherent part of the kernel sources. As such they are very tightly coupled with the specific kernel interfaces, which is not static at all but changes as needed and all the drivers in the kernel source then have to be modified to adhere to these changes. It's obviously a very different development model than what you see in closed source OSes but there is something to be said about not involving several layers of complex intermediate abstraction that are generally difficult to debug and even more difficult to keep in sync with modifications on both the upper and lower boundary of each layer. It's not a very good model in terms of scaling when adding many different device drivers, as a single change in the kernel interface will easily require to change every single device driver in the source tree too, but that's what the kernel developers decided in. The alternative is to not only strictly specify the interface between the kernel and user space, which the Linux kernel does, although often with a twist in that they seem to prefer to do it in a different way to BSD and other Unix variants, seemingly for the sake of being different, but also define a stringent and static device driver API inside the kernel that will never change except maybe between major kernel versions. And even-though that might seem like a good idea for device driver developers as it would allow closed source drivers that won't need to be recompiled with every kernel upgrade, it's also a model that requires enormous architectural work upfront, before any driver can be written, only to find out that at the time the interface has been defined and the necessary infrastructure has been developed, that it is already obsolete. 

Another factor that might play in here is that the only way a device driver is actually easy to maintain in such a development model, is by actually open sourcing it, which is of course one of the main motivations of GNU in general and the Linux kernel especially. Unfortunately this leaves users of hardware that the manufacturer doesn't want to document openly, such as most NI hardware too, pretty much in the cold when they want to use Linux. One can get angry at NI or the Linux kernel guys that they each maintain their position, but that doesn't help and in the end both sides have the right to deal in this as they wish and currently do. Linux is not going to have a static device driver interface and while that is a pita for anyone not wanting to donate their device driver source with lots of their own support and sweat into the kernel mainline, it's how the Linux world is going to work for as long as there are people who want to work on Linux.

It seems that many open source developers favor this model also outside the rather confined albeit extensive kernel development, but they forget that if you write a library you are not operating in a closed environment such as the kernel sources, but in a world where others will actually want to interface to that library and then arbitrarily changing the contract of a library API, simply because it is convenient to do so, should not be something that is considered without the uttermost care

Edited February 23, 2017 by rolfk
correct some typos

[ShaunR]

3 hours ago, rolfk said:

Reading a bit further on the technical mailing list it seems.........

I think I actually heard an audible sigh at the end of that. Nice.

3 hours ago, rolfk said:

and then arbitrarily changing the contract of a library API, simply because it is convenient to do so

And learning the meaning of "deprecate" wouldn't hurt either.

Edited February 23, 2017 by ShaunR

[Rolf Kalbermatter]

19 minutes ago, ShaunR said:

And learning the meaning of "deprecate" wouldn't hurt either.

Except that from the languages I do know, only Java, C# and C++14 support a deprecated keyword. Yes you can do it with gcc with the __attribute((deprecated)) and with MSVC with the __declspec(deprecated) keyword too, but that is a compiler toolchain specific extension which is not portable.

So for many languages it either ends up as a custom decorator of a specific library (Python or Lua can do that) or it's just a comment that nobody will read anyhow! 

Edited February 23, 2017 by rolfk

[ShaunR]

3 hours ago, rolfk said:

Except that from the languages I do know, only Java, C# and C++14 support a deprecated keyword. Yes you can do it with gcc with the __attribute((deprecated)) and with MSVC with the __declspec(deprecated) keyword too, but that is a compiler toolchain specific extension which is not portable.

So for many languages it either ends up as a custom decorator of a specific library (Python or Lua can do that) or it's just a comment that nobody will read anyhow! 

But we have internet nowadays so you can do stuff like this-it's not rocket science.

As for LabVIEW. Since I use polymorphic VIs for APIs and always make the menu visible, I use the instance menu and add (deprecated) after the instance (and menu) name. From that release it will be available for a minimum of 2 years to allow migration but generally it will not be removed at all from the library, only removed from the polymorphic list after that.

Edited February 23, 2017 by ShaunR