45 replies to this topic

[Black Pearl]

Well, check the enternal 'being' of a *.vipc

Felix

[jgcode]

Well, check the enternal 'being' of a *.vipc

I have - I wrote reuse VIs (for VIPM 3.0) that I use in my build scripts for OGPB packages that parse the VIPC for dependencies (i.e. create list) so I could programmatically update the OGPB file's dependencies etc... So by joke I am guessing you mean simple then?

[Daklu]

vipc format is a joke.

Personally I'm quite glad JKI uses well-establish file formats. It allows us to use other tools to manage these packages instead of being completely dependent on JKI for functionality.

[asbo]

Personally I'm quite glad JKI uses well-establish file formats. It allows us to use other tools to manage these packages instead of being completely dependent on JKI for functionality.

Agreed - every now and then I'll have something I need to swipe something from and I'm on a machine that doesn't have VIPM installed. It's very convenient to just rename the file and extract.

[jgcode]

Agreed - every now and then I'll have something I need to swipe something from and I'm on a machine that doesn't have VIPM installed. It's very convenient to just rename the file and extract.

If you don't have VIPM installed then what are you doing with the extracted packages?

[vugie]

Agreed - every now and then I'll have something I need to swipe something from and I'm on a machine that doesn't have VIPM installed. It's very convenient to just rename the file and extract.

Or just Ctrl+PgDn in Total Commander

If you don't have VIPM installed then what are you doing with the extracted packages?

Quick look into spec file gives you sufficient information what to do Ok, I saw the emoticon...

[jgcode]

Quick look into spec file gives you sufficient information what to do Ok, I saw the emoticon...

Gotcha!

[rolfk]

But still: vipc format is a joke.

Not sure why you say that. Just because it is not all obfuscated or even binarised doesn't mean it is a joke. The OGP format was the first format devised and Jim, me and I think someone else whose name I can't come up right now, came up with it after I had looked at ZLIB and according ZIP library and decided to create a LabVIEW library to deal with ZIP formats.
The idea for the spec file format I did in fact derive from the old application builder configuration format and it proved flexible, yet simple enough to deal with the task. Jim came with the idea to put the spec file inside the archive under a specific name, similar to how some Linux package managers did it back then. The rest was mostly plumbing and making it work and it is still in principle the same format as in the beginning. The VIPC is just another similar format to track down packages more easily.

JKI goes to great lengths to obfuscate the software in the VIPM but they are to be applauded to not have gone the path of obfuscating the actual file formats.

[jgcode]

Not sure why you say that. Just because it is not all obfuscated or even binarised doesn't mean it is a joke. The OGP format was the first format devised and Jim, me and I think someone else whose name I can't come up right now, came up with it after I had looked at ZLIB and according ZIP library and decided to create a LabVIEW library to deal with ZIP formats.
The idea for the spec file format I did in fact derive from the old application builder configuration format and it proved flexible, yet simple enough to deal with the task. Jim came with the idea to put the spec file inside the archive under a specific name, similar to how some Linux package managers did it back then. The rest was mostly plumbing and making it work and it is still in principle the same format as in the beginning. The VIPC is just another similar format to track down packages more easily.

JKI goes to great lengths to obfuscate the software in the VIPM but they are to be applauded to not have gone the path of obfuscating the actual file formats.

Great post.

someone else whose name I can't come up right now,

Is it Konstantin Shifershteyn?

[Black Pearl]

The original ogp format is brilliant.

My rant was only targetting the vipc format, 'cause it doesn't add a lot of new information and does this in a 'bad' way.
-> Information: The only new things I found was a second icon and inside the config.xml the ID. I guess the ID is used for the features of VIPM enterprise.
-> 'bad' way: In the original ogp the spec file is a config file, hence extendible. So it would have been easy to add the extra information to that as new entries. Then the spec file is duplicated outside the ogp container. All of this is zipped, hence basically a zip around a zip, which isn't really efficient.

Going back to my original work, I get these questions/topics/tasks/ideas:
* Namespacing via OpenG builder? So my reuse project would have under SCC the original code (and the build/package scripts). Then non-SCC a folder for the namespaced vi's and a folder for the created packages. As far as I know, VIPM allows you to do both in a single step (namespacing + packageing).
* Propagating the revision from the repository inside the package. My idea was to add this as extra info (sections) to the spec file or to autogenerate a file that goes into the package and is installed alongside.
* Make the package format transparent to SCC/diff. I found that beyond compare is able to handle .zip and it might be that others diff tools have hooks to unzip the files for them.

Any comments?

Felix

[jgcode]

The original ogp format is brilliant.

My rant was only targetting the vipc format, 'cause it doesn't add a lot of new information and does this in a 'bad' way.
-> Information: The only new things I found was a second icon and inside the config.xml the ID. I guess the ID is used for the features of VIPM enterprise.
-> 'bad' way: In the original ogp the spec file is a config file, hence extendible. So it would have been easy to add the extra information to that as new entries. Then the spec file is duplicated outside the ogp container. All of this is zipped, hence basically a zip around a zip, which isn't really efficient.

FWIW - VIPM uses a VIP format for packages (which is the 'new' OGP format if you like).
VIPC is a file that contains packages (VIP or OGP files).
Seeing that the VIP format is an extension of the OGP I would say is better - i.e. in VIPM 2010 supports non-symbolic installation locations using sub-packages.

* Make the package format transparent to SCC/diff. I found that beyond compare is able to handle .zip and it might be that others diff tools have hooks to unzip the files for them.

I just downloaded Beyond Compare as it can be configured to handle diff's of packages, but it seems you have to pay for it

[Black Pearl]

FWIW - VIPM uses a VIP format for packages (which is the 'new' OGP format if you like).
VIPC is a file that contains packages (VIP or OGP files).
Seeing that the VIP format is an extension of the OGP I would say is better - i.e. in VIPM 2010 supports non-symbolic installation locations using sub-packages.

That now makes sense to me. So forget about the rants...

I just downloaded Beyond Compare as it can be configured to handle diff's of packages, but it seems you have to pay for it

I also look for a non-commercial way, preferable OS, as this allows to better adjust it to the specific needs of LV.
For mercurial there is an option to store zip files uncompressed (should me more efficient):
http://www.selenic.c...l#decode-encode
I'll see if I get that working.

Felix

[jgcode]

That now makes sense to me. So forget about the rants...

Yer, it all makes sense now.

I also look for a non-commercial way, preferable OS, as this allows to better adjust it to the specific needs of LV.
For mercurial there is an option to store zip files uncompressed (should me more efficient):
http://www.selenic.c...l#decode-encode
I'll see if I get that working.

Well, apparently Beyond Compare works well, but my trial is running out and its prompting me to buy a key.
Its 30-50 bucks, so its not a ball-breaker, but I look forward to seeing if you come up with an alternative.

[Black Pearl]

For mercurial there is an option to store zip files uncompressed (should me more efficient):
http://www.selenic.c...l#decode-encode
I'll see if I get that working.

This doesn't work. I contacted the mercurial support and they told me that it's only working if a single file is zipped, not for a complete archive.

Well, apparently Beyond Compare works well, but my trial is running out and its prompting me to buy a key.
Its 30-50 bucks, so its not a ball-breaker, but I look forward to seeing if you come up with an alternative.

Free & OS alternative is WinMerge
Make sure you also download the 7-zip-Plugin.
I'd like too know how it's compared (using user-diff ) to BeyondCompare.

Felix

[Justin Goeres]

Felix,

I have a few thoughts on your new points, just to give you my perspective as someone who uses VIPM all the time.

* Propagating the revision from the repository inside the package. My idea was to add this as extra info (sections) to the spec file or to autogenerate a file that goes into the package and is installed alongside.

Maybe I'm sort of misunderstanding your goals, but in my experience it's not necessary to capture the repository revision inside the package itself. What we typically do at JKI is add a tag, or at least a commit comment, to the source repository (we use SVN) when we build a package. The package itself is versioned, of course (1.0, 1.1, 2.0, etc.) and if we want to know what SVN rev it came from we can look that up in the SVN repository itself. We find that's the easiest way to capture the information, since it's basically one of the core features of any SCC system and doesn't require us to write software. This keeps the package format itself lightweight.

* Make the package format transparent to SCC/diff. I found that beyond compare is able to handle .zip and it might be that others diff tools have hooks to unzip the files for them.

I guess I'm not sure what need is solved by having the package format be transparent to SCC/diff. Again, the change history of the source for the package is stored in SCC (plus the release notes of the package). If you find yourself needing to actually diff revs of your libraries it seems to me you've got a change control problem that needs to be solved upstream, when you're developing the libraries in the first place. Or am I misunderstanding?

[Black Pearl]

Maybe I'm sort of misunderstanding your goals, but in my experience it's not necessary to capture the repository revision inside the package itself. What we typically do at JKI is add a tag, or at least a commit comment, to the source repository (we use SVN) when we build a package. The package itself is versioned, of course (1.0, 1.1, 2.0, etc.) and if we want to know what SVN rev it came from we can look that up in the SVN repository itself. We find that's the easiest way to capture the information, since it's basically one of the core features of any SCC system and doesn't require us to write software. This keeps the package format itself lightweight.

Justin, thank you for the feedback.

My goals aren't really clear, that's why I'm posting here.
(a) I'm still working on the overall architecture. I think I should try to get an electronic drawing of it and post it.
(b) I'm learning/evaluating a lot of new stuff (using mercurial, testing diff-tools, using packages).

But your post has brought something to my attention. If I created a newer version of a package, the file name would change (containing the version).
This is useful when I have a share, so for each package all the versions can reside in a single directory and it's easy for the user to get the correct version.
On the other hand, when I place the packages in the project folder, two issues might arise:
(a) conflicting versions, there are 2 packages of the same library with different versions. This is pretty easy to get by accident, as just deleting the file wouldn't remove it from the repository, so on an update it would be restored.
(b) automatic tracking by the SCC would fail. So I need to use a rename command so it's tracked at what stage the newer package was applied.

Felix

[Justin Goeres]

* Propagating the revision from the repository inside the package. My idea was to add this as extra info (sections) to the spec file or to autogenerate a file that goes into the package and is installed alongside.

Another quick note on this. Depending on what you want to do with the information (see my post above where I don't quite understand what that is ), you might be able to do exactly what you want with VIPM's Custom Actions. You can create special VIs that run at pre-/post-build, pre-/post-install, and pre-/post-uninstall time for each package. This gives you a tremendous amount of flexibility in terms of mangling your VIs or various support files exactly the way you want, in an automated way.

Install Actions and Uninstall Actions are available in VIPM Community.
Build Actions are available in VIPM Professional and Enterprise.

On the other hand, when I place the packages in the project folder, two issues might arise:
(a) conflicting versions, there are 2 packages of the same library with different versions. This is pretty easy to get by accident, as just deleting the file wouldn't remove it from the repository, so on an update it would be restored.

(b) automatic tracking by the SCC would fail. So I need to use a rename command so it's tracked at what stage the newer package was applied.

Correct. I'm not a hardcore mercurial user so I can't comment intelligently on exactly how that would happen, but in SVN, if you had a directory of packages under source code control you'd have to delete the old package and then Commit that change.

However, this is another advantage of VIPC files. The VIPC file contains a list of packages (and their versions!) so you can never have the duplicate package problem you're describing. You'd handle your package upgrades this way:

• Build the new package.
• Open your existing VIPC file for a given project and drag the new package into it (requires VIPM Pro).
• Save the VIPC.
• Commit the modified VIPC file to SCC. Use a commit comment like "Upgraded blackpearl_lib_kraken_tracker.vip to 2.0.1."

This way you're not "adding" or "removing" files from SCC -- you're just updating the package configuration (VIPC file) that you're already tracking.

[Ton Plomp]

...
Correct. I'm not a hardcore mercurial user so I can't comment intelligently on exactly how that would happen, but in SVN, if you had a directory of packages under source code control you'd have to delete the old package and then Commit that change.

In my book Built files do not belong in SCC, for mercurial I excluded the following items from scc:

• .ogp$
• /built*$
• built
• /build*$
• .lvlps$
• .aliases$
• .zip$

This prevents OGP and zip files as wel as built/d folders to get automatically added to SCC, I can add them manually (hg add ......)

Ton

[Black Pearl]

In the meantime I did bring my scetch to an electronic form: EDIT: See at the end of post.

Another quick note on this. Depending on what you want to do with the information (see my post above where I don't quite understand what that is ), you might be able to do exactly what you want with VIPM's Custom Actions. You can create special VIs that run at pre-/post-build, pre-/post-install, and pre-/post-uninstall time for each package. This gives you a tremendous amount of flexibility in terms of mangling your VIs or various support files exactly the way you want, in an automated way.

Install Actions and Uninstall Actions are available in VIPM Community.
Build Actions are available in VIPM Professional and Enterprise.

Yes, I was thinking about using these hooks to pass the revision number from the SCC to the package.

Correct. I'm not a hardcore mercurial user so I can't comment intelligently on exactly how that would happen, but in SVN, if you had a directory of packages under source code control you'd have to delete the old package and then Commit that change.

SCC is a bit troubling when changing file names or locations. The only problems I had with SVN were related to this (saving a vi in a new location and deleting the old, forced update before commit, merge conflicts ) and I think a fellow developer stopped using SVN for this reason. I found hg to be much more robust in this case (automatically find renames, much nicer to resolve merge conflicts on a file basis).

However, this is another advantage of VIPC files. The VIPC file contains a list of packages (and their versions!) so you can never have the duplicate package problem you're describing. You'd handle your package upgrades this way:

• Build the new package.
• Open your existing VIPC file for a given project and drag the new package into it (requires VIPM Pro).
• Save the VIPC.
• Commit the modified VIPC file to SCC. Use a commit comment like "Upgraded blackpearl_lib_kraken_tracker.vip to 2.0.1."

This way you're not "adding" or "removing" files from SCC -- you're just updating the package configuration (VIPC file) that you're already tracking.

Ok, now I see another usage of the VIPC format.
Well actually you are just making the replacement of the file intransparent to the SCC.

Back to my older posts. Drawing the distribution scetch again braught back why I was looking at propagating the Changeset ID into the package.
I was mainly thinking about how to know if the project has the same packages that are currently installed in the user.lib. If my analysis is right, this is managed by the VIPM having a config file where it stores the currently installed packages with their version number.
So if I apply the vipc file, it can check this against it's own data base and manage all versioning conflicts.

Felix

In my book Built files do not belong in SCC, for mercurial I excluded the following items from scc:

While above statement is true, the point of view towards the packages changes. For the reuse developer, the packages are 'built files' that should not be placed under SCC.
However the user of the reuse lib (packages) will see them as part of his source, hence place them under SCC.

Felix

Attached Thumbnails

• 

Edited by Black Pearl, 15 December 2010 - 04:46 PM.

[Justin Goeres]

In the meantime I did bring my scetch to an electronic form:

Thanks for the diagram, that makes some things clearer!

I hacked up a diagram similar to yours, but showing where VIPM might fit into the system. See attached.

Notable differences:

• Package Building is simpler because VIPM takes care of the hard parts of the OGB/OGPB hassle.
• Instead of a Network Share, built packages are stored in a VI Package Repository on the network. Note that this is exactly like NI's LabVIEW Tools Network repository, or the VI Package Network repository used by OpenG, but it contains your packages.
• Instead of manually getting packages from the Network Share, each project developer uses VIPM to download packages from the Package Repository, and creates a VIPC file for each project. This VIPC file specifies exactly which packages are in use on the project. When switching between projects, the developer just "applies" the new project's VIPC file and he/she knows the correct packages will be installed.

EDIT: To clarify my 3rd point above... The project-specific VIPC doesn't just allow a single developer to switch projects easily; it also makes it easy for one developer to hand a project off to another developer because in ensures everyone's environment has the correct packages when they start work.