OpenG LabVIEW Zip 5.0.0-1 - stuck at the readme

[ShaunR]

On 11/27/2024 at 2:36 PM, hooovahh said:

I'm also in Windows only land, and renaming the DLLs in a Post Install has worked well enough so far.  If we in a 64 bit LabVIEW for instance the post install looks for if there are DLLs named [X].dll but there exists an [X]64.dll.  If so it will delete the [X].dll and rename [X]64.dll to [X].dll.  This renaming bit also works in reverse looking for [X]32.dll if we are on a 32 bit version of LabVIEW.  Development then is done in one bitness statically picking the correct DLL. I have no clue how difficult or complicated this gets supporting Linux.

I think we are talking past each other.

I don't have a problem with naming. That is just what the *.* stuff does. I'm interested in what Rolf is doing to put the *.* into the node paths. I have a problem with linking-the VI search popping up during installation, VIPM not compiling everything and asking the user to save after use. This is caused by VIPM setting in concrete, the full DLL path in the nodes when it builds and not compiling certain VI's after installation.

[Rolf Kalbermatter]

48 minutes ago, ShaunR said:

I think we are talking past each other.

I don't have a problem with naming. That is just what the *.* stuff does. I'm interested in what Rolf is doing to put the *.* into the node paths. I have a problem with linking-the VI search popping up during installation, VIPM not compiling everything and asking the user to save after use. This is caused by VIPM setting in concrete, the full DLL path in the nodes when it builds and not compiling certain VI's after installation.

Yes, we kind of talk past each other here. Mostly because it is an involved low level topic.

And VIPM is in this specific case technically pretty innocent. It uses the private Read Linker Info and Write Linker Info methods in LabVIEW to retrieve the hierarchy of VIs, relocating them on disk to a new hierarchy, which is quite a feat to do properly, patching up the linker info to correctly point to the new locations and writing it back. The problem is that the paths returned by Read Linker Info are nicely de-wildcarded, which is a good thing if you want to access the actual files on disk (the standard file functions know nothing about shared library wildcard names), but a bad thing when writing back the linker info. The Read Linker Info itself does not deliver the original wildcard name (it could theoretically do so in the "delimited name" string for shared libraries as that name is otherwise not very useful for shared libraries, but doesn't do so, which is in fact a missed opportunity as the fully resolved name is already returned correctly in the path).

The only way to fix that consistently across multiple LabVIEW versions is to go through the array of linker infos after doing the Read Linker Info, recursively determine the callers that reference a specific shared library, enumerate their block diagram for any Call Library nodes, retrieve their library name and replace the "delimited name" in the linker info with the file name portion of that! Then before writing back the Linker Info, replace the file name in the path, respectively in newer LabVIEW versions you have the additional "original Name" string array when writing back the Linker Info, so might be possible to pass that name in there. Quite an involved modification to the existing OpenG Builder library. 

[hooovahh]

On 11/29/2024 at 4:34 AM, ShaunR said:

I think we are talking past each other.

I don't have a problem with naming. That is just what the *.* stuff does. I'm interested in what Rolf is doing to put the *.* into the node paths. I have a problem with linking-the VI search popping up during installation, VIPM not compiling everything and asking the user to save after use. This is caused by VIPM setting in concrete, the full DLL path in the nodes when it builds and not compiling certain VI's after installation.

Yes the *.* stuff would take care of a lot of this.  But it seems VIPM does weird things to path during the build.  That's why I did the post install renaming work. VIPM has the linking to a full path to the DLL, so renaming that linked path on install seemed like an easy way to make it work the way I wanted.  A post install fixing the paths work as well, and in fact Jim posted a VI that can fix the paths on post install.  My issue with this is it is time consuming for the post install to open a reference to every VI, find all call library nodes on the VI, and then update the path on them, and then save the VI.  Especially if you are installing a ton of VIs.  Renaming the DLL is much fasters.

[ShaunR]

On 12/2/2024 at 4:04 PM, hooovahh said:

VIPM has the linking to a full path to the DLL, so renaming that linked path on install seemed like an easy way to make it work the way I wanted. 

Yes. This is why we have the post install VI's. There are other edge-case issues too, that cannot be resolved just by renaming.

On 12/2/2024 at 4:04 PM, hooovahh said:

A post install fixing the paths work as well, and in fact Jim posted a VI that can fix the paths on post install. 

Yes. That is not a viable solution.

However, what we are talking about is modifying the og libraries that VIPM uses to build (they are distributed as source), and fixing the paths BEFORE VIPM adds it to the the install so we don't need the Post Install at all.

[Rolf Kalbermatter]

The most proper solution will be to go through all the files retrieved through the Read Linker Info function in the Generate Resource Copy Info.vi, find all shared libraries and from there their callers, enumerate for Call Library Nodes in the diagram and retrieve the library name and patch up the "delimited name" in the linker info for the shared library with the file name portion of that library name. In the Copy Resource Files and Relink VIs.vi, simply replace the file name portion of the path with the patched up "delimited name" value! The finding of the callers is already done in the Generate Resource Copy Info.vi, so we "just" have to go through the Resources Copy Info array and get the shared library entries and then check their Referrers.

This assumes that all the library paths are properly wildcard formatted prior to starting the build process as it will maintain the original library name in the source VIs.

Possible gotcha's still needed to find out and resolve:

- Is the library name returned by the scripting functions at this point still the wildcard name or already patched with the resolved name while loading the VI?

- Are all the scripting functions to enumerate the Call Library nodes and retrieve the library name also available in the runtime or only in the full development runtime? Although the Build Application.vi function is most likely executed over VI server in the target LabVIEW version and not as part of the VIPM executable runtime?

 

 

 

 

 

[ShaunR]

1 hour ago, Rolf Kalbermatter said:

This assumes that all the library paths are properly wildcard formatted prior to starting the build process as it will maintain the original library name in the source VIs.

That may be a problem and I don't really see why it needs to be so. Is this just to resolve files that are named like user32.dll?

The entry point for what you suggest seems to be here:

[Rolf Kalbermatter]

Actually I'm more inclined to hack things here:

I don't feel like traversing the entire linker info array recursively if that is done anyhow in the next VI in the execution flow! But the current implementation is a bit naive, did some refactoring on that too.

Edited December 4 by Rolf Kalbermatter

[hooovahh]

Has anyone talked to JKI about this? I'm sure they make VIPM do this during the build for a reason and they would probably be interested in the changes you are doing.

Quote

 There are other edge-case issues too, that cannot be resolved just by renaming.

Like what? Are you referring to the earlier discussion, where you may have "32" in the DLL name when it is a 64-bit binary?  I handle that by only performing the rename if there exists the correctly named DLL for the other bitness, on disk. Otherwise it leaves it.  User32.dll wouldn't be renamed for multiple reasons. But even if we installed "MyAgeIs32.DLL" it wouldn't rename it unless there exists a "MyAgeIs64.DLL" and we are in 64 bit LabVIEW.  Not perfect but I'd prefer that over editing the VIPM build environment.

[Rolf Kalbermatter]

26 minutes ago, hooovahh said:

Has anyone talked to JKI about this? I'm sure they make VIPM do this during the build for a reason and they would probably be interested in the changes you are doing.

We are also developers, not just whiners! 😀

Honestly I did try in the past to get some feedback. It was more specific about supporting more fine grained target selections than just Windows, Linux and Mac in the package builder, specifically to also support 32-bit or 64-bit and not just for the entire package, as VIPM eventually supported, but rather for individual file groups in the package. Never received any reaction on that. Maybe it was because I wanted to use it with my own OpenG Builder, I just needed the according support in the spec file for the VIPM package installer to properly honor things. Didn't feel like distributing my own version of the package installer too.

That format allows specifying the supported targets, LabVIEW version and type on a per file group base. VIPM package builder is a bit more limited for the purpose of more easy package configuration when building packages. I'm using that spec file feature through my own Package Builder to build a package that contains all shared libraries for all supported platforms and then install the correct shared libraries for Linux or Windows in the OpenG ZIP (and Lua for LabVIEW) package, depending on what platform things are installed. Unfortunately the specification for the platform and LabVIEW version does not seem to support 32-bit and 64-bit distinctions. So I have to install both and then either fixup things to the common name in the Post Install, or use the filename*.* library name scheme.

Edited December 4 by Rolf Kalbermatter

[hooovahh]

24 minutes ago, Rolf Kalbermatter said:

We are also developers, not just whiners! 😀

I wasn't suggesting you were whiners.  If I ask for support from NI I don't think that makes me a whiner.  I was suggesting the changes you are proposing are useful for everyone, and that if you have useful changes, all VIPM users can benefit from them by having JKI use what you've developed.  Because you are a developer.  I also asked that question because I wanted to know incite into why VIPM does what it does so I can be a better developer.

[Rolf Kalbermatter]

1 hour ago, hooovahh said:

I wasn't suggesting you were whiners.  If I ask for support from NI I don't think that makes me a whiner.  I was suggesting the changes you are proposing are useful for everyone, and that if you have useful changes, all VIPM users can benefit from them by having JKI use what you've developed.  Because you are a developer.  I also asked that question because I wanted to know incite into why VIPM does what it does so I can be a better developer.

It's my intention to throw the VI, once it seems to work for me, over the fence into JKI's garden and see what they will do with it. 😀 More seriously I plan to inform them of the problem and my solutions, through whatever support channels I can find. It does feel a bit like a black hole though. The reason for working like it does, is most likely that it is simply how the Read Linker Info in LabVIEW works. They use that and the information it returns and that is not including the original wildcard name. Also most of that is the original OpenG package builder library with some selected fixes since, for newer LabVIEW features. I also don't get the feeling that anyone really dug into those VIs very much to improve them beyond some bug fixes and support for new LabVIEW features since the library was originally published as OpenG library (mainly developed by Konstantin Shifershteyn back then, with some modifications by Jim Kring and Ton Plomp until around 2008).

Edited December 4 by Rolf Kalbermatter

[ShaunR]

37 minutes ago, hooovahh said:

I wasn't suggesting you were whiners.  If I ask for support from NI I don't think that makes me a whiner.  I was suggesting the changes you are proposing are useful for everyone, and that if you have useful changes, all VIPM users can benefit from them by having JKI use what you've developed.  Because you are a developer.  I also asked that question because I wanted to know incite into why VIPM does what it does so I can be a better developer.

I think it was just tongue-in-cheek whimsy.

 

2 hours ago, Rolf Kalbermatter said:

Actually I'm more inclined to hack things here:

I'm still not convinced it needs fixing around there at all. As far as I can tell, it only needs to be fixed at the original VI you proffered. The only issue you seemed to have is when a binary that isn't part of the developers distribution has a 32 or 64 on it (like user32.dll). I'd be more inclined to think of your initial suggestion of comparing paths to circumvent that though.

[Rolf Kalbermatter]

42 minutes ago, ShaunR said:

I'm still not convinced it needs fixing around there at all. As far as I can tell, it only needs to be fixed at the original VI you proffered. The only issue you seemed to have is when a binary that isn't part of the developers distribution has a 32 or 64 on it (like user32.dll). I'd be more inclined to think of your initial suggestion of comparing paths to circumvent that though.

Yes, my concern is this. It's ok for your own solution, where you know what shared libraries your projects use and how such a fix might have unwanted effects. In the worst case you just shoot in your own foot. 😀

But it is not a fix that could ever be incorporated in VIPM proper as there is not only no way to know for what package builds it will be used, but an almost 100% chance that an affected user will simply have no idea why things go wrong, and how to fix it. If it is for your own private VIPM installation, go ahead, do whatever you like. It's your system after all. 😀

[ShaunR]

16 hours ago, Rolf Kalbermatter said:

Yes, my concern is this. It's ok for your own solution, where you know what shared libraries your projects use and how such a fix might have unwanted effects. In the worst case you just shoot in your own foot. 😀

But it is not a fix that could ever be incorporated in VIPM proper as there is not only no way to know for what package builds it will be used, but an almost 100% chance that an affected user will simply have no idea why things go wrong, and how to fix it. If it is for your own private VIPM installation, go ahead, do whatever you like. It's your system after all. 😀

That is a given. The only issue I would have there is when VIPM is updated.

However, you still have not explained why the "Extract Resources Info From Linker Info.vi"  needs modification if all modifications can be achieved in the "Copy Resource Files and Relink VIs__ogb.vi"

[Rolf Kalbermatter]

51 minutes ago, ShaunR said:

That is a given. The only issue I would have there is when VIPM is updated.

However, you still have not explained why the "Extract Resources Info From Linker Info.vi"  needs modification if all modifications can be achieved in the "Copy Resource Files and Relink VIs__ogb.vi"

It's pretty simple. You want to get the real information from the Call Library Node, not some massaged and assumed file name down the path. Assume starts with ass and that is where it sooner or later bites you! 😀

The Call Library Node knows what the library name is that the developer configured. And that is the name that LabVIEW uses when you do your unit tests. So you want to make sure to use that name and not some guesstimated one.

The Read Linker Info unfortunately only returns the recompiled path and file name after the whole VI hierarchy has been loaded and relinked, possibly with relocating VIs and DLLs from anywhere in the search path if the original path could not be found. That is as far as the path goes very useful, but it looses the wildcard name for shared libraries. The delimited name would have been a prime candidate to use to return that information. This name exists to return the fully names spaced name of VIs in classes and libraries. It serves no useful purpose for shared libraries though and could have been used to return the original Call Library Node name. Unfortunately it wasn't.

So my idea is to patch that up in the Linker Info on reading that info and later reuse it to update the linker info when writing it back to the relocated and relinked VI hierarchy.

Edited December 5 by Rolf Kalbermatter

[ShaunR]

3 minutes ago, Rolf Kalbermatter said:

It's pretty simple. You want to get the real information from the Call Library Node, not some massaged and assumed file name down the path. Assume starts with ass and that is where it sooner or later bites you! 😀

The Call Library Node knows what the library name is that the developer configured. And that is the name that LabVIEW uses when you do your unit tests. So you want to make sure to use that name and not some guesstimated one.

That doesn't help you with user32.dll.

[Rolf Kalbermatter]

4 minutes ago, ShaunR said:

That doesn't help you with user32.dll.

It does. The Call Library Node explicitly has to be configured as user32.dll or maybe user32.*. Otherwise, if the developer enters user*.*, it won't work properly when loading the VI in 64-bit LabVIEW.

Edited December 5 by Rolf Kalbermatter

[ShaunR]

4 minutes ago, Rolf Kalbermatter said:

It does. The Call Library Node explicitly has to be configured as user32.dll or maybe user32.*. Otherwise, if the developer enters user*.*, it won't work properly when loading the VI in 64-bit LabVIEW.

This is the same as naming a DLL x32 or x64 with extra steps. You are now adding a naming convention to a read linker and a modify linker. It's getting worse, not better.

Edited December 5 by ShaunR

[Rolf Kalbermatter]

9 minutes ago, ShaunR said:

This is the same as naming a DLL x32 or x64 with extra steps. You are now adding a naming convention to a read linker and a modify linker. It's getting worse, not better.

It depends what your aim is. If your aim is to get something working most of the time with as little modification as possible to the OpenG OGB library, then yes you want to guesstimate the most likely wildcard name, with some exception handling for specially known DLL names during writing back the Linker Info.

If your aim is to use the information that LabVIEW is using when it loaded the VIs into memory, you want to get the actual Call Library Name. And you want to get that as early as possible, preferably before the OGB tool resaved and post- and/or prefixed the name of all the VIs to a new location. All these things have the potential to overwrite the actual Call Library Node library name with the new name and destroy the wildcard name. So while you could try to query the library name in the Write Linker Info step, it may be to late to retrieve that information correctly. That means modifying the OGB library in two places. Is that really that much more bad than modifying it in one place with an only most of the time correct name?

Edited December 5 by Rolf Kalbermatter

[ShaunR]

32 minutes ago, Rolf Kalbermatter said:

It depends what your aim is. If your aim is to get something working most of the time with as little modification as possible to the OpenG OGB library, then yes you want to guesstimate the most likely wildcard name, with some exception handling for specially known DLL names during writing back the Linker Info.

Yes. This is exactly what is required. The User32 problem can be resolved either with file path comparison (which you stated earlier) or a list of known DLL names. I could think of a few more ways to make it automatic but I would lean to the latter as the developer could add to the list in unforeseen edge cases. The former might just break the build with no recourse.

37 minutes ago, Rolf Kalbermatter said:

If your aim is to use the information that LabVIEW is using when it loaded the VIs into memory, you want to get the actual Call Library Name.

You seem to have added that for no apparent reason, from what I can tell.

[Rolf Kalbermatter]

2 hours ago, ShaunR said:

You seem to have added that for no apparent reason, from what I can tell.

Let's agree to disagree. 😀

However, some testing seems to indicate that it is not possible with the publicly available functions to get the real wildcard library name at all. It's there in the VI file and the Call Library Node can display it, but trying to read it through scripting will always return the fully qualified name as it was resolved for the current platform and current bitness. No joy in trying to get the wildcard name. That's rather frustrating!

Edited December 5 by Rolf Kalbermatter

[ShaunR]

On 12/5/2024 at 12:47 PM, Rolf Kalbermatter said:

No joy in trying to get the wildcard name. That's rather frustrating!

Well. You could get it from the file. It's under the  block diagram heap (BDHP) which is a very old structure. I personally wouldn't bother for the reasons I stated earlier but it would be much faster than using scripting.

[Rolf Kalbermatter]

1 hour ago, ShaunR said:

Well. You could get it from the file. It's under the  block diagram heap (BDHP) which is a very old structure. I personally wouldn't bother for the reasons I stated earlier but it would be much faster than using scripting.

Faster is a bit debatable. 😁 The scripting method took me less than half an hour to find out that it doesn't return what I want. Should have done that from the start before getting into a debate with you about the merits of doing that. 😴

The file is of course an option, except that there are a number of pitfalls. Some newer versions of LabVIEW seem to Deflate many of the resource streams and giving them a slightly different name. Not a huge problem but still a bit of a hassle. And the structure of the actual resources is not always quite casted in stone. NI did change some of that over the course of time. But pylabview definitely has most of the information needed for doing that. 😁

One advantage, it would not be necessary to do that anywhere else in the OGB library. The original file structure should still be guaranteed to be intact at the time the file has been copied to its new location! 

But it goes clearly beyond the SuperSecretPrivateSpecialStuff, that OpenG and VIPM have dared to endeavor into so far. Considering that I'm currently trying to figure out how to do LabVIEW Variants from the C side of things, as properly as possibly without serious hacks, it's however not something that would scare me much. 😁

Edited December 7 by Rolf Kalbermatter

[ShaunR]

10 minutes ago, Rolf Kalbermatter said:

Faster is a bit debatable. 😁 The scripting method took me less than half an hour to find out that it doesn't return what I want.

Faster execution than scripting. Scripting is incredibly slow.

[Rolf Kalbermatter]

3 minutes ago, ShaunR said:

Faster execution than scripting. Scripting is incredibly slow.

Ahh, ok, that I would wholeheartedly agree to. It's probably mostly going through the UI thread and possibly a lot through the root loop even. It touches many global resources and that's the most easy way to protect things. Adding mutexes over the whole place makes things nasty fast and before you know it you are in the land of mutual exclusion if you are not VERY careful.

Edited December 7 by Rolf Kalbermatter