Reuse, Packages and SCC

[vugie]

Agreed - every now and then I'll have something I need to swipe something from and I'm on a machine that doesn't have VIPM installed. It's very convenient to just rename the file and extract.

Or just Ctrl+PgDn in Total Commander

If you don't have VIPM installed then what are you doing with the extracted packages?

Quick look into spec file gives you sufficient information what to do Ok, I saw the emoticon...

[jgcode]

Quick look into spec file gives you sufficient information what to do Ok, I saw the emoticon...

Gotcha!

[Rolf Kalbermatter]

But still: vipc format is a joke.

Not sure why you say that. Just because it is not all obfuscated or even binarised doesn't mean it is a joke. The OGP format was the first format devised and Jim, me and I think someone else whose name I can't come up right now, came up with it after I had looked at ZLIB and according ZIP library and decided to create a LabVIEW library to deal with ZIP formats.

The idea for the spec file format I did in fact derive from the old application builder configuration format and it proved flexible, yet simple enough to deal with the task. Jim came with the idea to put the spec file inside the archive under a specific name, similar to how some Linux package managers did it back then. The rest was mostly plumbing and making it work and it is still in principle the same format as in the beginning. The VIPC is just another similar format to track down packages more easily.

JKI goes to great lengths to obfuscate the software in the VIPM but they are to be applauded to not have gone the path of obfuscating the actual file formats.

[jgcode]

Not sure why you say that. Just because it is not all obfuscated or even binarised doesn't mean it is a joke. The OGP format was the first format devised and Jim, me and I think someone else whose name I can't come up right now, came up with it after I had looked at ZLIB and according ZIP library and decided to create a LabVIEW library to deal with ZIP formats.

The idea for the spec file format I did in fact derive from the old application builder configuration format and it proved flexible, yet simple enough to deal with the task. Jim came with the idea to put the spec file inside the archive under a specific name, similar to how some Linux package managers did it back then. The rest was mostly plumbing and making it work and it is still in principle the same format as in the beginning. The VIPC is just another similar format to track down packages more easily.

JKI goes to great lengths to obfuscate the software in the VIPM but they are to be applauded to not have gone the path of obfuscating the actual file formats.

Great post.

someone else whose name I can't come up right now,

Is it Konstantin Shifershteyn?

[Black Pearl]

The original ogp format is brilliant.

My rant was only targetting the vipc format, 'cause it doesn't add a lot of new information and does this in a 'bad' way.

-> Information: The only new things I found was a second icon and inside the config.xml the ID. I guess the ID is used for the features of VIPM enterprise.

-> 'bad' way: In the original ogp the spec file is a config file, hence extendible. So it would have been easy to add the extra information to that as new entries. Then the spec file is duplicated outside the ogp container. All of this is zipped, hence basically a zip around a zip, which isn't really efficient.

Going back to my original work, I get these questions/topics/tasks/ideas:

* Namespacing via OpenG builder? So my reuse project would have under SCC the original code (and the build/package scripts). Then non-SCC a folder for the namespaced vi's and a folder for the created packages. As far as I know, VIPM allows you to do both in a single step (namespacing + packageing).

* Propagating the revision from the repository inside the package. My idea was to add this as extra info (sections) to the spec file or to autogenerate a file that goes into the package and is installed alongside.

* Make the package format transparent to SCC/diff. I found that beyond compare is able to handle .zip and it might be that others diff tools have hooks to unzip the files for them.

Any comments?

Felix

[jgcode]

The original ogp format is brilliant.

My rant was only targetting the vipc format, 'cause it doesn't add a lot of new information and does this in a 'bad' way.

-> Information: The only new things I found was a second icon and inside the config.xml the ID. I guess the ID is used for the features of VIPM enterprise.

-> 'bad' way: In the original ogp the spec file is a config file, hence extendible. So it would have been easy to add the extra information to that as new entries. Then the spec file is duplicated outside the ogp container. All of this is zipped, hence basically a zip around a zip, which isn't really efficient.

FWIW - VIPM uses a VIP format for packages (which is the 'new' OGP format if you like).

VIPC is a file that contains packages (VIP or OGP files).

Seeing that the VIP format is an extension of the OGP I would say is better - i.e. in VIPM 2010 supports non-symbolic installation locations using sub-packages.

* Make the package format transparent to SCC/diff. I found that beyond compare is able to handle .zip and it might be that others diff tools have hooks to unzip the files for them.

I just downloaded Beyond Compare as it can be configured to handle diff's of packages, but it seems you have to pay for it

[Black Pearl]

FWIW - VIPM uses a VIP format for packages (which is the 'new' OGP format if you like).

VIPC is a file that contains packages (VIP or OGP files).

Seeing that the VIP format is an extension of the OGP I would say is better - i.e. in VIPM 2010 supports non-symbolic installation locations using sub-packages.

That now makes sense to me. So forget about the rants...

I just downloaded Beyond Compare as it can be configured to handle diff's of packages, but it seems you have to pay for it

I also look for a non-commercial way, preferable OS, as this allows to better adjust it to the specific needs of LV.

For mercurial there is an option to store zip files uncompressed (should me more efficient):

http://www.selenic.com/mercurial/hgrc.5.html#decode-encode

I'll see if I get that working.

Felix

[jgcode]

That now makes sense to me. So forget about the rants...

Yer, it all makes sense now.

I also look for a non-commercial way, preferable OS, as this allows to better adjust it to the specific needs of LV.

For mercurial there is an option to store zip files uncompressed (should me more efficient):

http://www.selenic.c...l#decode-encode

I'll see if I get that working.

Well, apparently Beyond Compare works well, but my trial is running out and its prompting me to buy a key.

Its 30-50 bucks, so its not a ball-breaker, but I look forward to seeing if you come up with an alternative.

[Black Pearl]

For mercurial there is an option to store zip files uncompressed (should me more efficient):

http://www.selenic.c...l#decode-encode

I'll see if I get that working.

This doesn't work. I contacted the mercurial support and they told me that it's only working if a single file is zipped, not for a complete archive.

Well, apparently Beyond Compare works well, but my trial is running out and its prompting me to buy a key.

Its 30-50 bucks, so its not a ball-breaker, but I look forward to seeing if you come up with an alternative.

Free & OS alternative is WinMerge

Make sure you also download the 7-zip-Plugin.

I'd like too know how it's compared (using user-diff ) to BeyondCompare.

Felix

[Justin Goeres]

Felix,

I have a few thoughts on your new points, just to give you my perspective as someone who uses VIPM all the time.

* Propagating the revision from the repository inside the package. My idea was to add this as extra info (sections) to the spec file or to autogenerate a file that goes into the package and is installed alongside.

Maybe I'm sort of misunderstanding your goals, but in my experience it's not necessary to capture the repository revision inside the package itself. What we typically do at JKI is add a tag, or at least a commit comment, to the source repository (we use SVN) when we build a package. The package itself is versioned, of course (1.0, 1.1, 2.0, etc.) and if we want to know what SVN rev it came from we can look that up in the SVN repository itself. We find that's the easiest way to capture the information, since it's basically one of the core features of any SCC system and doesn't require us to write software. This keeps the package format itself lightweight.

* Make the package format transparent to SCC/diff. I found that beyond compare is able to handle .zip and it might be that others diff tools have hooks to unzip the files for them.

I guess I'm not sure what need is solved by having the package format be transparent to SCC/diff. Again, the change history of the source for the package is stored in SCC (plus the release notes of the package). If you find yourself needing to actually diff revs of your libraries it seems to me you've got a change control problem that needs to be solved upstream, when you're developing the libraries in the first place. Or am I misunderstanding?

[Black Pearl]

Maybe I'm sort of misunderstanding your goals, but in my experience it's not necessary to capture the repository revision inside the package itself. What we typically do at JKI is add a tag, or at least a commit comment, to the source repository (we use SVN) when we build a package. The package itself is versioned, of course (1.0, 1.1, 2.0, etc.) and if we want to know what SVN rev it came from we can look that up in the SVN repository itself. We find that's the easiest way to capture the information, since it's basically one of the core features of any SCC system and doesn't require us to write software. This keeps the package format itself lightweight.

Justin, thank you for the feedback.

My goals aren't really clear, that's why I'm posting here.

(a) I'm still working on the overall architecture. I think I should try to get an electronic drawing of it and post it.

(b) I'm learning/evaluating a lot of new stuff (using mercurial, testing diff-tools, using packages).

But your post has brought something to my attention. If I created a newer version of a package, the file name would change (containing the version).

This is useful when I have a share, so for each package all the versions can reside in a single directory and it's easy for the user to get the correct version.

On the other hand, when I place the packages in the project folder, two issues might arise:

(a) conflicting versions, there are 2 packages of the same library with different versions. This is pretty easy to get by accident, as just deleting the file wouldn't remove it from the repository, so on an update it would be restored.

(b) automatic tracking by the SCC would fail. So I need to use a rename command so it's tracked at what stage the newer package was applied.

Felix

[Justin Goeres]

* Propagating the revision from the repository inside the package. My idea was to add this as extra info (sections) to the spec file or to autogenerate a file that goes into the package and is installed alongside.

Another quick note on this. Depending on what you want to do with the information (see my post above where I don't quite understand what that is ), you might be able to do exactly what you want with VIPM's Custom Actions. You can create special VIs that run at pre-/post-build, pre-/post-install, and pre-/post-uninstall time for each package. This gives you a tremendous amount of flexibility in terms of mangling your VIs or various support files exactly the way you want, in an automated way.

Install Actions and Uninstall Actions are available in VIPM Community.

Build Actions are available in VIPM Professional and Enterprise.

On the other hand, when I place the packages in the project folder, two issues might arise:

(a) conflicting versions, there are 2 packages of the same library with different versions. This is pretty easy to get by accident, as just deleting the file wouldn't remove it from the repository, so on an update it would be restored.

(b) automatic tracking by the SCC would fail. So I need to use a rename command so it's tracked at what stage the newer package was applied.

Correct. I'm not a hardcore mercurial user so I can't comment intelligently on exactly how that would happen, but in SVN, if you had a directory of packages under source code control you'd have to delete the old package and then Commit that change.

However, this is another advantage of VIPC files. The VIPC file contains a list of packages (and their versions!) so you can never have the duplicate package problem you're describing. You'd handle your package upgrades this way:

1. Build the new package.
2. Open your existing VIPC file for a given project and drag the new package into it (requires VIPM Pro).
3. Save the VIPC.
4. Commit the modified VIPC file to SCC. Use a commit comment like "Upgraded blackpearl_lib_kraken_tracker.vip to 2.0.1."

This way you're not "adding" or "removing" files from SCC -- you're just updating the package configuration (VIPC file) that you're already tracking.

[Ton Plomp]

...

Correct. I'm not a hardcore mercurial user so I can't comment intelligently on exactly how that would happen, but in SVN, if you had a directory of packages under source code control you'd have to delete the old package and then Commit that change.

In my book Built files do not belong in SCC, for mercurial I excluded the following items from scc:

• .ogp$
• /built*$
• built
• /build*$
• .lvlps$
• .aliases$
• .zip$

This prevents OGP and zip files as wel as built/d folders to get automatically added to SCC, I can add them manually (hg add ......)

Ton

[Black Pearl]

In the meantime I did bring my scetch to an electronic form: EDIT: See at the end of post.

Another quick note on this. Depending on what you want to do with the information (see my post above where I don't quite understand what that is ), you might be able to do exactly what you want with VIPM's Custom Actions. You can create special VIs that run at pre-/post-build, pre-/post-install, and pre-/post-uninstall time for each package. This gives you a tremendous amount of flexibility in terms of mangling your VIs or various support files exactly the way you want, in an automated way.

Install Actions and Uninstall Actions are available in VIPM Community.

Build Actions are available in VIPM Professional and Enterprise.

Yes, I was thinking about using these hooks to pass the revision number from the SCC to the package.

Correct. I'm not a hardcore mercurial user so I can't comment intelligently on exactly how that would happen, but in SVN, if you had a directory of packages under source code control you'd have to delete the old package and then Commit that change.

SCC is a bit troubling when changing file names or locations. The only problems I had with SVN were related to this (saving a vi in a new location and deleting the old, forced update before commit, merge conflicts ) and I think a fellow developer stopped using SVN for this reason. I found hg to be much more robust in this case (automatically find renames, much nicer to resolve merge conflicts on a file basis).

However, this is another advantage of VIPC files. The VIPC file contains a list of packages (and their versions!) so you can never have the duplicate package problem you're describing. You'd handle your package upgrades this way:

1. Build the new package.
2. Open your existing VIPC file for a given project and drag the new package into it (requires VIPM Pro).
3. Save the VIPC.
4. Commit the modified VIPC file to SCC. Use a commit comment like "Upgraded blackpearl_lib_kraken_tracker.vip to 2.0.1."

This way you're not "adding" or "removing" files from SCC -- you're just updating the package configuration (VIPC file) that you're already tracking.

Ok, now I see another usage of the VIPC format.

Well actually you are just making the replacement of the file intransparent to the SCC.

Back to my older posts. Drawing the distribution scetch again braught back why I was looking at propagating the Changeset ID into the package.

I was mainly thinking about how to know if the project has the same packages that are currently installed in the user.lib. If my analysis is right, this is managed by the VIPM having a config file where it stores the currently installed packages with their version number.

So if I apply the vipc file, it can check this against it's own data base and manage all versioning conflicts.

Felix

In my book Built files do not belong in SCC, for mercurial I excluded the following items from scc:

While above statement is true, the point of view towards the packages changes. For the reuse developer, the packages are 'built files' that should not be placed under SCC.

However the user of the reuse lib (packages) will see them as part of his source, hence place them under SCC.

Felix

Edited December 15, 2010 by Black Pearl

[Justin Goeres]

In the meantime I did bring my scetch to an electronic form:

Thanks for the diagram, that makes some things clearer!

I hacked up a diagram similar to yours, but showing where VIPM might fit into the system. See attached.

Notable differences:

• Package Building is simpler because VIPM takes care of the hard parts of the OGB/OGPB hassle.
• Instead of a Network Share, built packages are stored in a VI Package Repository on the network. Note that this is exactly like NI's LabVIEW Tools Network repository, or the VI Package Network repository used by OpenG, but it contains your packages.
• Instead of manually getting packages from the Network Share, each project developer uses VIPM to download packages from the Package Repository, and creates a VIPC file for each project. This VIPC file specifies exactly which packages are in use on the project. When switching between projects, the developer just "applies" the new project's VIPC file and he/she knows the correct packages will be installed.

EDIT: To clarify my 3rd point above... The project-specific VIPC doesn't just allow a single developer to switch projects easily; it also makes it easy for one developer to hand a project off to another developer because in ensures everyone's environment has the correct packages when they start work.

[Black Pearl]

Justin, the similarity of our diagrams are not coincidence, it's by design:

1. VIPM is designed for that purpose, so it is designed to fit in that picture

2. My own design is planned on minimizing the effort to switch over once I get my boss convinced that I need better software tools (that'll be the hard job).

Point 2 is why I'm really happy you joined this discussion and want to thank you, because this confirms to me that my architecture is pretty flexible and I'm on the right track. Thank you for this!

To get a bit more abstract in the debate, I really think you should check out the powers of DRCS (distributed revision control systems) like mercurial. What did stunn me most while playing around with hg is the ability to adapt to any architecture with little effort.

So at the first step, I can have my repositories local

In the next step I share repositories via USB thumbdrive to pass (directionally) the project from one developer to the next

In a third step, I migrate this architecture to a central repository on a server.

The migration from step 2 to step 3 is important, as this is a change in topology. The 'after' developer is now not getting the data by push from the 'before' developer' but by pull from the server. Having such a flexebility allows to (by proof of induction) realize complex topologies as well such as presented in Joel Spolskys post on hg.

Hope this helps you to develope VIPM to the next step. It's really easy to install TotoiseHg, create some repositories with minimal data and try out how to manage topologies.

Felix

[Justin Goeres]

Justin, the similarity of our diagrams are not coincidence, it's by design:

To get a bit more abstract in the debate, I really think you should check out the powers of DRCS (distributed revision control systems) like mercurial. What did stunn me most while playing around with hg is the ability to adapt to any architecture with little effort.

This is a great point. We've played with Hg a little bit at JKI and I've used it on a couple small projects. We're continuing to evaluate it; even if we never switch over to it ourselves we need to understand it for our customers.

[Black Pearl]

This is a great point. We've played with Hg a little bit at JKI and I've used it on a couple small projects. We're continuing to evaluate it; even if we never switch over to it ourselves we need to understand it for our customers.

Although mercurial is a free software, they offer you very good support. I joined their mailing list for the questions related to managing zip files and got a reply from the support team within 24h.

Felix

[Michael Aivaliotis]

I wanted to clarify some things (in case it wasn't clear).

You can Apply VIPC configuration files using the Free VIPM Community Edition. You need VIPM Professional if you are creating VIPC files. So if developers are configuring their project for development then they can use the free VIPM community edition.

VIPC files are configuration files. You have the option to include the actual package inside the VIPC file (the default mode) if you want but this is not mandatory. We recommend this for users configuring development environments on non-networked machines or using sneaker-net.

If you don't include the packages inside the VIPC file, then only the package name is included. When VIPM tries to Apply the VIPC file (install\upgrade\downgrade all the packages defined in the configuration), it tries to look-up the packages on all known internal (corporate) repositories. Or external NI or JKI repositories. The package may even already be located on the local PC cache as well. Once the packages are located, they are installed as normal.

Of course you can also have a combination of the above. Some packages inside the VIPC and others included only by reference.

[Ton Plomp]

(packages not in SCC)...

While above statement is true, the point of view towards the packages changes. For the reuse developer, the packages are 'built files' that should not be placed under SCC.

However the user of the reuse lib (packages) will see them as part of his source, hence place them under SCC.

Felix

Looking at your process this would mean the reuse user placed vi.lib and user.lib under SCC as well?

Ton

[Black Pearl]

Looking at your process this would mean the reuse user placed vi.lib and user.lib under SCC as well?

See the earlier discussion, the packages are under SCC inside the project and applied (extracted) to the user.lib. The user.lib itself is not managed by SCC.

Felix