Request for collaboration:Open Tensorflow Toolkit

[Neil Pate]

Hi everyone, 

I have recently been playing around with the Tensorflow support built into the Vision Development Module. It seems to work fine and I have the basics up and running. There are a couple of problems though that I would like to solve. 

Firstly, it seems to not use GPU acceleration at all which mostly makes it useless for real time processing at any sensible frame rate. 

Secondly, it is locked into the VDM which is not cheap and also totally closed source. Under the hood the regular Tensorflow DLLs are called, but via a layer of the Vision toolkit. 

Third, for this closed source reason we are locked into whey ever version of Tensorflow NI chooses. I totally get that from their business point of view, but conversely I suspect that there is very little push inside NI to update this regularly due to all the hurdles that come with it.

My proposal is to implement some kind of community edition of Tensorflow API that wraps the Tensorflow DLL directly in LabVIEW and exposes hardware acceleration capabilities.

Anybody interested in collaboration? I know a little bit about Tensorflow, but not enough to be productive, so my first step would just be to mimic the API provided by NI which is deliberately quite simple. 

Edited October 12, 2019 by Neil Pate

[Neil Pate]

Although my intended application is vision based I suppose in principle a Tensorflow Toolkit should support other types of ML applications. As I said I don't really know enough about tensorflow to know how big an undertaking it would be to create a wrapper for the whole DLL. 

[JKSH]

I can't commit to direct collaboration, but I'm happy to share my thoughts, experiences, and code from a similar project.

https://www.tensorflow.org/api_docs/ comes with a warning:

Quote

A word of caution: the APIs in languages other than Python are not yet covered by the API stability promises.

This means new versions can break backwards compatibility which would be a pain for a non-Python based wrapper/binding project. This is probably a reason why NI might not stick to the latest and greatest version of TensorFlow.

I can see 2 possible approaches:

1. Wrap the TensorFlow Python API, and integrate into LabVIEW using the Python Node.
2. Flatten the Tensor C++ API into a C API, wrap the C API, and integrate into LabVIEW using the Call Library Function Node.

 

#1 guarantees that the wrapper won't be broken by a TensorFlow update, but restricts the wrapper to Windows and LabVIEW >= 2018 only. I also don't have experience with the Python Node and don't know what limitations might exist.

#2 allows the wrapper to be used on Linux RT and on older versions of LabVIEW, but there's always a risk of compatibility breaks between TensorFlow versions. Also, given the large number of classes and functions (see https://www.tensorflow.org/api_docs/cc ), it might make sense to write a code generator (like VI scripting) that produces the actual wrapper code, instead of writing the wrapper by hand.

I've written a code generator to (partially) wrap the C++ API of the Qt toolkit (see https://github.com/JKSH/LQ-CodeGen ). This generator makes many assumptions based on Qt's features and coding conventions so it's not usable for generic C++ libraries, but its source code might provide some ideas and hints. The final wrapper library is at https://github.com/JKSH/LQ-Bindings

[Neil Pate]

Thanks JKSH 

I echo your sentiment. Versioning is always going to be a problem, but just recnelty Tensorflow hit 2.0 so that was what I was planning on supporting as the minimum version.

Going down the python route is also interesting but a bit fraught. Probably due to my inexperience with python and it's tool chain  I spent the better part of a month just trying to follow myriad tutorials online to get python and tensorflow working properly on my PC, with very little to show for my results. 

I do think it would be reasonable to say that a toolkit such as this requires LV 2018 or greater. 

[Rolf Kalbermatter]

13 hours ago, Neil Pate said:

I do think it would be reasonable to say that a toolkit such as this requires LV 2018 or greater. 

If you want to take the Python route then of course. As far as the Call Library Node is concerned there is virtually no difference since at least LabVIEW 2009 and even before that the only real difference from 8.0 on onwards to 2009 is the automatic support for 32bit and 64 bit DLL interfacing at least if it is about pointers being passed directly as parameters. Once you deal with pointers inside structures you have to either create a wrapper DLL anyhow or deal with conditional code compilation on the LabVIEW diagram for the different bitnesses. 

Edited October 13, 2019 by Rolf Kalbermatter

[Neil Pate]

@Rolf Kalbermatter my brain was clearly running badly when I typed that we needed LV2018, what I actually meant to say was the 64 bit LV was required. 🤐

I say this as it is my understanding the Tensorflow DLL is strictly 64 bit. 

Do you suppose it would be possible to use 32 bit LV if going via the Python route?

 

Edited October 13, 2019 by Neil Pate

[Rolf Kalbermatter]

3 hours ago, Neil Pate said:

@Rolf Kalbermatter my brain was clearly running badly when I typed that we needed LV2018, what I actually meant to say was the 64 bit LV was required. 🤐

I say this as it is my understanding the Tensorflow DLL is strictly 64 bit. 

Do you suppose it would be possible to use 32 bit LV if going via the Python route?

 

Not out of the box. 32 bit LabVIEW interfaces to 32 bit Python. So you would need to have some  32 bit Python to 64 bit Tensor Flow remoting interface. If 64 bit LabVIEW is a requirement the lowest possible version is 2009 on Windows and around 2014 on the Mac and Linux platforms.

Edited October 13, 2019 by Rolf Kalbermatter

[Antoine Chalons]

Do you really need/want to support 32 bit? I mean... if it's for vision, you probably use a large size of memory, no?

I wish I could offer my collaboration but calling external code is not really my cup of tea, so I'd probably be mostly useless anyway.

If you are interested in getting in touch with tensorflow users to have their feedback, get in touch with me via PM.

[Neil Pate]

I don't actually need the large memory support at this point, but feel it is getting close to calling time on 32 bit applications. I have been dabbling with LV2019 64 and so far it does what I need but admittedly I have not needed to interface with any hardware yet apart from a GigE camera. 

[Rolf Kalbermatter]

On 10/14/2019 at 3:34 PM, Neil Pate said:

I don't actually need the large memory support at this point, but feel it is getting close to calling time on 32 bit applications. I have been dabbling with LV2019 64 and so far it does what I need but admittedly I have not needed to interface with any hardware yet apart from a GigE camera. 

At this point, if you only use NI hardware you are fairly safe. It's either supported with 64 bit drivers or discontinued anyways. If you use other 3rd party drivers the situation is a lot more checkered. Some have already abandoned 32 bit software and only deliver 64 bit anymore. Others have not made the step and many might never as their hardware and software offering is in a sort of maintenance state.

"Look it works!" 

"Hands off and don't touch it anymore! It was hard enough to get it not to crash constantly!" 😆

Edited October 15, 2019 by Rolf Kalbermatter