Reuse, Packages and SCC

[Black Pearl]

I'm planning to put a process in place for managing reusable code using mercurial for SCC.

My old approach (using SVN) was to have the user.lib under SVN. This was ok as I was the only developer accessing this reusable library. Now I would like to share it with others in my team (using OGPB and VIPM community). Also this became a bit difficult as I started using other locations as well (instr.lib, plugins, tools...).

So I was thinking about having the code-base seperate. Primary code would reside in a dedicated reuse-project folder (managed by Hg). Then I'd create packages using OGPB. These packages can now go onto a server for team-sharing. I as well as others would then install the packages in the user.lib and other places using VIPM community.

So here would be the first question:

? How to track the version on a project basis ?

I've been playing with the subrepositories feature in mercurial (this is suggested as analog to SVN-externals). But that would only allow the files to stay in the project folder, not in user.lib.

The same issue with this use-case:

? How to get reusable checked out on the USB thumbdrive ?

I found that Hg makes life easy there, as I can clone my repository for field-testing on customers side. If I install mercurial on the machine, I even can get the changes committed and back in the home-base, I just push what I've done to my developement machine.

But I not really want to install VIPM (and LV2009RT) on the customer machine and have my reuse code deployed there. It's more effort than I save by having all nice palettes.

This is also a problem with my current SVN approach. But here I did use source distributions with OGB and had a tool to check them back in the project folder.

For both issues, I was thinking of a tool that would just pack all VIs that do not reside in the project folder and maintain a flat copy (e.g. zip archive) in the project folder. But this of course leads to relinking problems (which is generating a lot of changesets in the SCC).

Another idea would be some kind of dynamically linking of the LV-IDE locations (user.lib and palette; instr.lib) to the project dir's, if possible. That would allow for subrepositories (and in this case I'd distribute via Hg instead of VIPM).

Felix

[jgcode]

Hey Felix - I don't know if I have understood all of your post correctly (so please forgive me if I didn't) but I thought I chime in anyway, to see if I can offer anything from what I do.

I recommend handling reusable code using a workflow that includes a dedicated tool for reuse, and IMHO, by far the best one is VIPM 2010. OGPB is great, I use both together, but after sorting out some (of my) internal issues I am trying to move to VIPM 2010 predominately (this is a process I am just starting). Because most of the time VIPM is way better - meaning less work for me (namespacing, palettes etc...). If I was going to start a reuse library right now I would look at VIPM 2010 over OGPB as a standard. In the Community Edition (which is free) you can now build your own packages, so I can't see any reason not too.

In terms of disk hierarchy I just have an internal project folder (like any other company >> project folder under SCC) that contains the source - broken down by packages - pretty simple. Packages get uploaded onto a server (share). And using packages make managing code across multiple versions of LabVIEW really simple. In order to support Client installs of development code, again IHMO, its far easy to install VIPM to do it, and then install the required packages (see below on this).

Taking a step back in this discussion - one of the more traditional approaches is to copy reuse files from project to project. I know all too well the feeling of warm, fuzziness that comes with it as your project now contains all the files it needs to run - it feels safe but is terrible for reuse!

But VIPM has a solution - I recommend checking out .vipc files. This is a file that contains multiple packages. This feature is available in the Professional version (but is well worth it), and what it allows you to do is to take a snapshot of your reuse code for a project. With the .vipc file you can then do several clever things. You can check it into SCC - along with your application source code. Now you have everything required to run your project, all under SCC - meaning you can manage project dependencies and changes that occur in reuse using SCC. You get all the security (that warm, fuzziness), with none of the managing multiple copies issues - because its not really the reuse source, its just a copy. As all the updates/bugfixing etc... is done under you internal project (a single location), then redistributed, so its all good!

A fellow developer can get a working copy of the .vipc file and apply the configuration to their version of LabVIEW, now ensuring all package versions are correct in their LabVIEW palettes - super fast. So its great in a multiple developer environment. This applies in the office or on a client machine.

[PaulL]

OK, I've been wondering about this for a while...

What are the advantages of putting code in the user.lib? Is it to put things on the palettes? (See Organization of LabVIEW.) Does it also somehow help to maintain multiple versions? (I'm not aware of anything in this regard.) Is there some other advantage?

What we tend to do is put our reuse libraries in a Common directory in our general code file. What is essential is that each chunk of reuse code is in a project library (.lvlib), which functions as a "package" in our terminology. Since we are using project libraries, a change to a project library in one project will propagate to all projects, and we just need to check out the latest version from Subversion to use the latest code. I guess the limitation is that we don't set up the user.lib palettes, but for our purposes we find it easier to drag anything we need (which is uncommon anyway) from the library in the project. (There is also strictly speaking nothing limiting who is able to make changes to a library. In our experience we haven't found that to be an issue.)

Note that we don't have to maintain lots of versions (2009 vs. 2010, for example) for our applications. That would certainly make life more difficult, although I'm not sure using the user.lib helps in that respect. Does it? (I imagine VIPM can be very useful for this sort of thing.)

The only things that end up in the user.lib for us are error code files, just because LabVIEW requires them to be there (which I think is an unfortunate departure from the project paradigm, by the way).

The Common folder with project library approach is quite simple, works exceptionally well for our use cases, and doesn't rely on a third party package. I can see that if you want to use the palettes (e.g., for core toolsets) you'd want to use the user.lib. Otherwise, why?

Paul

[Ton Plomp]

The Common folder with project library approach is quite simple, works exceptionally well for our use cases, and doesn't rely on a third party package. I can see that if you want to use the palettes (e.g., for core toolsets) you'd want to use the user.lib. Otherwise, why?

Two things come to mind:

• Paths, things in the user.lib (or vi.lib/instr.lib) have an aboslute path, moving a VI that uses one of those functions isn't going to force a re-linking.
  This means I can have my code under SCC in d:\sourcecode, my colleague can have it on his USB-drive at f:\code.
  Which also means it's easier to setup a coherent IDE, just install the packages in a manner you like (copy, VIPM, install with NI installer) and you have them where you expect them
• QuickDrop
  The items show up in a quick drop search (which can be very usefull)
• The other advantage is the thing you allready mentioned but (IMHO) is very important, you can easily access them. It's possible to browse them through the palettes. In some of these dying seconds of the day (when the coffee isn't ready yet, a download is going on, facebook is offline, whatever) you can go through them and see if any of your colleagues has writing a little gem, he mentioned but you weren't listening.

Ton

[Black Pearl]

jgcode, you nicely explained what I had in mind. The problem I have with VIPM is that packages created with it are only LV 8.2+ and I'm stuck with 7.1.

So I'll see how I can implement the features I miss using the hooks provided by the OGPB and Hg.

Paul, it's definitive that I need the palettes. Part of the reuse process is, to make it easy (for you and others) to use the reuse library -> easy access. And 7.1 has no project explorer.

Felix

[jgcode]

jgcode, you nicely explained what I had in mind.

Cool.

The problem I have with VIPM is that packages created with it are only LV 8.2+ and I'm stuck with 7.1.

So I'll see how I can implement the features I miss using the hooks provided by the OGPB and Hg.

Paul, it's definitive that I need the palettes. Part of the reuse process is, to make it easy (for you and others) to use the reuse library -> easy access. And 7.1 has no project explorer.

Bummer! My library is in 8.2 and I am reviewing pulling it up to 2009 soon. I have programmed in 7.1 (by force) and don't really like it. Of course that doesn't help you - it's just me rambling again.

One of the coolest things about VIPB is obviously the palettes - there just so much more work involved in doing it manually.

But you still with a have the benefits as outlined in my first post wrt installation and management of reuse code using VIPM.

[Michael Aivaliotis]

I didn't know we allowed LabVIEW 7.1 users to hang out on LAVA?

[PaulL]

Two things come to mind:

• Paths, things in the user.lib (or vi.lib/instr.lib) have an aboslute path, moving a VI that uses one of those functions isn't going to force a re-linking.
  This means I can have my code under SCC in d:\sourcecode, my colleague can have it on his USB-drive at f:\code.
  Which also means it's easier to setup a coherent IDE, just install the packages in a manner you like (copy, VIPM, install with NI installer) and you have them where you expect them
• QuickDrop
  The items show up in a quick drop search (which can be very usefull)
• The other advantage is the thing you allready mentioned but (IMHO) is very important, you can easily access them. It's possible to browse them through the palettes. In some of these dying seconds of the day (when the coffee isn't ready yet, a download is going on, facebook is offline, whatever) you can go through them and see if any of your colleagues has writing a little gem, he mentioned but you weren't listening.

Ton

I spent some time thinking about your reply overnight:

Symbolic paths, Quick Drop, Palettes--all these are good points! While none of these is particularly compelling for our particular (internal!) use cases (our relative paths are the same--enforced via version control--and we find it simpler to use the project interface than wait for Quick Drop or set up the Palettes), certainly if one ever intends to hand off a tool, for example, to an external user I see that the code should very definitely be in the user.lib, for all the reasons you stated, and it is simpler from a long-term maintenance standpoint to put it there initially. Point taken.

[Also, if one isn't using projects and project libraries one simply can't do what we do, as noted.]

Our team should take a closer look and see if there are frequently-used tools we should put in the user.lib.

Thanks!

Paul

[Justin Goeres]

jgcode, you nicely explained what I had in mind. The problem I have with VIPM is that packages created with it are only LV 8.2+ and I'm stuck with 7.1.

So I'll see how I can implement the features I miss using the hooks provided by the OGPB and Hg.

Hi Felix,

As JG pointed out, the reuse/sharing problem you're trying to solve is exactly what VIPM is designed to do. I'm going to point out a few things you might find relevant. If you're interested in investigating it more, please write to me directly or contact us through our website. We can definitely set you up with an evaluation license for VIPM Pro or Enterprise, and would be happy to talk about your needs.

First, it's a bummer you're on LV7.1, but you can still download VIPM 3.0 from our website (link is at the bottom of that page). Obviously VIPM 3.0 isn't being maintained anymore, but it might provide a solution for you until the day when you can make the jump to a version of LabVIEW that's still supported by NI .

Second, the need to share reuse code among team members and/or with customers is exactly what VIPM Professional & VIPM Enterprise are good for:

• VIPM Professional lets you create VIPC files (as JG mentioned) that you can use to bundle packages together and manually share them with others.
• VIPM Enterprise lets you create a networked package repository so that you can publish packages easily for any number of "client" computers. This is what NI uses for their LabVIEW Tools Network package repository; they publish packages and those libraries automatically show up in users' package lists around the world. Obviously it works just as well for smaller teams, too.

We have some users who manage their reuse libraries using just VIPM Pro and some VIPC files, while others find that VIPM Enterprise is a lot easier/more flexible/more reliable.

[Black Pearl]

If it were my decision, surely I'd run with LabVIEW PDS 2010 + SSP and VIPM Enterprise.

I did already read some posts that described the use of package repository and this is exactly what I would like to have.

As far as I have tested, the latest VIPM community is still able to connect to LV 7.1 and can read the packages I created with the OGPB. I didn't test if they get installed properly, but I suppose so.

I'll post back when I've done some more testing.

Felix

[Black Pearl]

Ok, as expected, there wasn't any issues using VIPM 2010 to install the *.ogp on 7.1.

So the remaining question is, how to ensure the packages in the project are in sync with the packages installed in the LabVIEW folder. I see two possible sources of corruption:

* I use a different (newer) version of my user.lib/subPackage and forget to update the packages in the project

* I use a completely new package and forget to update the project

both sadly would corrupt the repository.

Any 'best practices' or even preferred automation options to deal with this?

Felix

[jgcode]

Ok, as expected, there wasn't any issues using VIPM 2010 to install the *.ogp on 7.1.

So the remaining question is, how to ensure the packages in the project are in sync with the packages installed in the LabVIEW folder. I see two possible sources of corruption:

* I use a different (newer) version of my user.lib/subPackage and forget to update the packages in the project

* I use a completely new package and forget to update the project

both sadly would corrupt the repository.

Any 'best practices' or even preferred automation options to deal with this?

Felix

Hi Felix

This is the use case for the VIPC file.

Each time you start work on a project you configure the workspace by applying the VIPC file.

This sets the correct version of the packages ensuring that your first point does not happen.

As for point two, VIPM allows you to scan the project for dependencies.

This captures any new additions and updates the VIPC file.

The VIPC files should be under SCC so you always have a snapshot of the current state of the project.

It really is a beautiful thing.

[Black Pearl]

jgcode, is this a manual process relying on the discipline of the developer? So after update, do apply the vipc. Before commit, capture a snapshot?

After all, the vipc format isn't really complicated. So I should be able to write my own snapshot tool in a day...

Felix

[jgcode]

jgcode, is this a manual process relying on the discipline of the developer?

For now its manual, but no more manual or discipline like than getting your developer to follow a workflow that uses SCC or comment code etc...

So after update, do apply the vipc.

Yes - but more correctly at the start of opening the project.

In LabVIEW 8.x this would correspond to just before opening the LabVIEW Project file.

Before commit, capture a snapshot?

No, on change to dependencies (this VIPC file is part of the project) update the VIPC file.

Then Commit (on change) just like any other modified file.

After all, the vipc format isn't really complicated. So I should be able to write my own snapshot tool in a day...

Well thats your call - but with a callout like that you know we are all expecting you to post the tool to LAVA within 24 hrs (I have started the countdown already)

[Black Pearl]

Well thats your call - but with a callout like that you know we are all expecting you to post the tool to LAVA within 24 hrs (I have started the countdown already)

Sorry, but to boring...

Actually, my file manager plainly told me what a vipc file is. I leave this to the readers to 'hack' on their own. In the end you have little more than you already get from OpenG Package Builder.

Well, this 'little' prevents me from creating vipc files (I'm not sure if this was intended by JKI), but since ogp's install the same way I don't care.

So back to the big picture of reuse. It's not done with creating a reuse library, but you need to make it damn easy for others to use the reuse library. That's where the team at JKI did put in all the effort for VIPM (which I think is worth the money). That's why I want my vi's in the palette (see earlier in this discussion).

So I better invest my time on thinking: how can I make it easy for my team to use the reuse lib, what are individual concerns, habits and dreams on code reuse. For the installation of reuse code on the developement machines, VIPM community will be the best, because it's professional looking and easy to use.

Creating snapshots, I think about a simple wrapper around the OpenG package builder's core, that just displays a small dialog: 'All your vi's from the reusable library are sucessfully added to your repository. Do you want to see the details? -> NO'.

On the other hand, I think it's easy to create a small 'Installer' (there used to be a 'OpenG Commander' which was the predecessor of VIPM, so propably most code is already written). Then I can just get the snapshot in the user.lib/.. for field testing without the need to install LV2009-RT and VIPM.

Felix

[asbo]

I leave this to the readers to 'hack' on their own.

Maaaan, I hate it when they leave it as an exercise for the reader.

[Black Pearl]

Maaaan, I hate it when they leave it as an exercise for the reader.

You could post to the dark side: 'Urgent - Plz hlp - I have got 24h for this homework assignment: ...'

Felix

[jgcode]

Then I can just get the snapshot in the user.lib/.. for field testing without the need to install LV2009-RT and VIPM.

Snapshotting your entire user.lib is not as efficient as only including the packages which your code actually calls (VIPM can scan for these).

Especially as your reuse library grows.

Additionally you may have calls to VIs outside user.lib from third party packages (e.g. in vi.lib) or your own in different locations (e.g. drivers in instr.lib) etc....

Anyways, probably also worth mentioning is that VIPC's have a cool feature that allows you to create a VIPC as a list only (remove packages).

This removes the footprint of copying all packages into SCC, whilst still maintaining the benefits of SCC (given you never release the same package number twice - which you can't really do in VIPM 2010).

As long as you have a backed-up repository of all packages (internal and external) that are easy to get to - then this could be desirable.

[Black Pearl]

Ok, back to the seriouse side. I've got a tool that checks for every vi that is called by the main vi and isn't sitting in the subfolders down the main. I'll pass them to ogpb (and the install location). done, or not?

Ahh, I see, the IPM has the package list and can track it down to the individual vi... smart. Then it can 'guess' the package and list the oackage in the dependencies. Smart & greate work at JKI.

But still: vipc format is a joke.

Felix

[jgcode]

Ok, back to the seriouse side...

Starting off the week slow - I didn't understand the above posts

But still: vipc format is a joke.

Can you explain in more detail?

[Black Pearl]

Well, check the enternal 'being' of a *.vipc

Felix

[jgcode]

Well, check the enternal 'being' of a *.vipc

I have - I wrote reuse VIs (for VIPM 3.0) that I use in my build scripts for OGPB packages that parse the VIPC for dependencies (i.e. create list) so I could programmatically update the OGPB file's dependencies etc... So by joke I am guessing you mean simple then?

[Daklu]

vipc format is a joke.

Personally I'm quite glad JKI uses well-establish file formats. It allows us to use other tools to manage these packages instead of being completely dependent on JKI for functionality.

[asbo]

Personally I'm quite glad JKI uses well-establish file formats. It allows us to use other tools to manage these packages instead of being completely dependent on JKI for functionality.

Agreed - every now and then I'll have something I need to swipe something from and I'm on a machine that doesn't have VIPM installed. It's very convenient to just rename the file and extract.

[jgcode]

Agreed - every now and then I'll have something I need to swipe something from and I'm on a machine that doesn't have VIPM installed. It's very convenient to just rename the file and extract.

If you don't have VIPM installed then what are you doing with the extracted packages?