LabVIEW and Source Code Control - Checkout

[Staffan]

When working with LabVIEW and Source Code Control, it quite often happens that vi:s need to be saved due to subvi recompilation.

Even if "Checkout Callers..." is checked, those vi:s are not always checked out by LabVIEW when the vi to be modified was checked out.

This is also happens when working with LabVIEW classes. Making a changes in one class makes may affect other classes.

To solve this when using Perforce as SCC, I checkout all vi:s (in Perforce), save all vi:s (in LabVIEW) and the "Revert Unchanged Files" (in Perforce).

When using Clearcase I havn't found an easy way to do this.

Why doesn't LabVIEW always know of which files to check out?

Any suggestions on how to make it more easy when working with LV/SCC?

/Staffan

[Ton Plomp]

QUOTE(Staffan @ Nov 21 2007, 10:49 AM)

When working with LabVIEW and Source Code Control, it quite often happens that vi:s need to be saved due to subvi recompilation.

Even if "Checkout Callers..." is checked, those vi:s are not always checked out by LabVIEW when the vi to be modified was checked out.

This is also happens when working with LabVIEW classes. Making a changes in one class makes may affect other classes.

To solve this when using Perforce as SCC, I checkout all vi:s (in Perforce), save all vi:s (in LabVIEW) and the "Revert Unchanged Files" (in Perforce).

When using Clearcase I havn't found an easy way to do this.

Why doesn't LabVIEW always know of which files to check out?

Any suggestions on how to make it more easy when working with LV/SCC?

/Staffan

I privately use Subversion which doesn't have such a tight check-in/out regime. At work we use MS VSS which suffers from the same issues as Clearcase (what I can deduct from your post)

Ton

[gmart]

QUOTE(Staffan @ Nov 21 2007, 03:49 AM)

When working with LabVIEW and Source Code Control, it quite often happens that vi:s need to be saved due to subvi recompilation.

Even if "Checkout Callers..." is checked, those vi:s are not always checked out by LabVIEW when the vi to be modified was checked out.

This is also happens when working with LabVIEW classes. Making a changes in one class makes may affect other classes.

To solve this when using Perforce as SCC, I checkout all vi:s (in Perforce), save all vi:s (in LabVIEW) and the "Revert Unchanged Files" (in Perforce).

When using Clearcase I havn't found an easy way to do this.

Why doesn't LabVIEW always know of which files to check out?

Any suggestions on how to make it more easy when working with LV/SCC?

/Staffan

In order for "Check out callers" to work, the callers of a VI have to be in memory. The same goes for classes. What I typically have is a "All VIs" VI and load that while developing. This pulls all my working VIs into memory and when I check out a VI who has callers that are not checked out, those should be included in the list of Calllers. This should apply for all SCC providers.

[dannyt]

QUOTE(gmart @ Nov 21 2007, 01:02 PM)

In order for "Check out callers" to work, the callers of a VI have to be in memory. The same goes for classes. What I typically have is a "All VIs" VI and load that while developing. This pulls all my working VIs into memory and when I check out a VI who has callers that are not checked out, those should be included in the list of Calllers. This should apply for all SCC providers.

I find this a major pain in the butt with LabVIEW, it is like trying to source control object files in C. It could be dealt with so much better by NI. In fact the whole ClearCase integration could be a great deal better.

There are think two levels to this problem, one is the basic inconvenient of checking out all these other files and the second is more major, a basic function of configuration control if tracking what has change and why. If you do code reviews or support parallel release or look back at what happened, you wish to see the actual files edited by developers to fix a problem or add a new piece of functionality and not to have to search for those files on a branch that contains 20 or 30 re-compiled files.

One suggestion on how to cope with this is a follows, this works if most of your work and testing is in the development environment, but would be harder if you are doing builds.

You do not have to save all your changes to disk for your VI / project to run. So when you start only checkout and save files you are actually editing and do not save any recompiled Vi changes, this is fine as in memory they are OK for you to test with.

When you are happy with your changes and things are working well. Load up your top level project and then close it. In LabVIEW 8.21 you get a list of the files it wants to save and why. Note down the files that LabVIEW wants to save due to "recompiles" or "sub-Vi changes". Now checkout in ClearCase these noted down files all with the same comment, something like "VI NOT edited - changed due to recompile or sub-vi change", then save all these VI's.

Now using various scripts or LabVIEW you can easily see which files on a branch have actually been edited and which have just been save for recompile by looking at the ClearCase comments.

The next step in this is automate this, as LabVIEW 8.2.1 can see which VI's in memory have change and why (well I assume this is the case) in order to give you the list in the save box. I should be possible to write a VI that generates that list and automatically checkout those that have recompiled with the correct comment.

I am working on something like this at the moment.

Slightly off topic extra,

I already have a bunch of Vi's to get various ClearCase info, view config spec's, comments, etc and carry out ClearCase actions like, checkout, , do a labview diff on versions correctly (in progress).These do not use the NI supplied VI's but use either ClearCase command lines from DOS prompts or the ClearCase Automation Library (CAL) DLL.

With these VI when I do an OpenG build I add to the build log the ClearCase view config spec, and a full bill off material list all the VI's in the build, list for each VI it's full ClearCase info like branch name and version.

The other tick I do is to automatically pass all the VI's on a specific ClearCase branch into the VI analyser.

I was thinking of these as possible candidates for the LAVA code repository but was not sure anyone would be interested.

[Ton Plomp]

QUOTE(dannyt @ Nov 21 2007, 03:31 PM)

I already have a bunch of Vi's to get various ClearCase info, view config spec's, comments, etc and carry out ClearCase actions like, checkout, , do a labview diff on versions correctly (in progress).These do not use the NI supplied VI's but use either ClearCase command lines from DOS prompts or the ClearCase Automation Library (CAL) DLL.

With these VI when I do an OpenG build I add to the build log the ClearCase view config spec, and a full bill off material list all the VI's in the build, list for each VI it's full ClearCase info like branch name and version.

The other tick I do is to automatically pass all the VI's on a specific ClearCase branch into the VI analyser.

What is the main reason you bypass the LV SCC interface? Are their items missing? I was thinking along the same way to build a Post-Build VI that would generate a build report containing those info.

One downside of the LV SCC it's LV 8+.

Ton

[dannyt]

QUOTE(tcplomp @ Nov 21 2007, 02:53 PM)

What is the main reason you bypass the LV SCC interface? Are their items missing? I was thinking along the same way to build a Post-Build VI that would generate a build report containing those info.

One downside of the LV SCC it's LV 8+.

Ton

This is a quick answers and not complete.

• The low level NI SCC are password protected so that is a pain.
  

• The NI SCC tools for the Clearcase interface work at to high a level and do not give you anything useful to work with for an advanced ClearCase user or admin, for example the SCC File Properties.vi and the SCC File History.vi launch the ClearCase properties window and History browser respectivly, there is not a lot you can do with that. Whereas to generate a bill of meterial I want to a File Properties VI to return a string I can then work on. There is an advance option input to both these VI's but with no documentition on the format or API of these (that takes me back to point one)
  

• There is a large amount of ClearCase functionalilty not available with the exposed interface (I am sure somebody will correct me here ), getting or even changing a ClearCase view configuration specification, getting a list of files on a branch, get back checkout / check in commets applying labels baselines and much more. ClearCase can be a great tool, but only if you spent time getting this level of stuff working to help you.
  

• What settings there are for the configuration of using their VI like "checkin even if identical" are up at the configure source control advanced level. If you are trying to auto builds / labeling / deployment you need to do these things programatically.
  

• Finally, and I am not sure if I can explain this correctly but the built in SCC for ClearCAse seems to be built more around ClearCase UCM and not the far better (IMHO) Base ClearCase. The source control Project concept in LabVIEW for ClearCase at anty rate just does not work for me. I have several views and branchs on the go at any one time (1 view = 1 branch = 1 fix) when I finish with a view to goes, in fact to get LabVIEW Clearcase interface working I have a fake view mounted all the time just to stop LabVIEW complaining when I lauch it with a newly created view.
  

• Plus I only had LabVIEW 7.0 when I started. :ninja:
  

anyway that my thought on the matter.

edit sorry I forgot, attached is the sort of extra info I put in the build log file http://lavag.org/old_files/post-7256-1195661907.txt'>Download File:post-7256-1195661907.txt

Dannyt

[gmart]

There's a lot to digest so I hope I get it all.

QUOTE

• The low level NI SCC are password protected so that is a pain.

The implementation VIs for the providers (Perforce, WinCI) are for the most part not password protected. The VIs used to implement the SCC functionality are password protected but that should be expected.
QUOTE

• The NI SCC tools for the Clearcase interface work at to high a level and do not give you anything useful to work with for an advanced ClearCase user or admin, for example the SCC File Properties.vi and the SCC File History.vi launch the ClearCase properties window and History browser respectivly, there is not a lot you can do with that. Whereas to generate a bill of meterial I want to a File Properties VI to return a string I can then work on. There is an advance option input to both these VI's but with no documentition on the format or API of these (that takes me back to point one)
  
  

• There is a large amount of ClearCase functionalilty not available with the exposed interface (I am sure somebody will correct me here
  
  ), getting or even changing a ClearCase view configuration specification, getting a list of files on a branch, get back checkout / check in commets applying labels baselines and much more. ClearCase can be a great tool, but only if you spent time getting this level of stuff working to help you.
  
  

• What settings there are for the configuration of using their VI like "checkin even if identical" are up at the configure source control advanced level. If you are trying to auto builds / labeling / deployment you need to do these things programatically.

  
  The interface to Clearcase is via the Microsoft SCC Interface which is generic in order to support many SCC providers. So as you have pointed out, you don't get detailed functionality for each provider but from LabVIEW's side, we are able to interface with many providers without having to write custom backend code. The intent of the SCC tools in LabVIEW is to allow you to do the basic, everyday operations from within LabVIEW without having to use the SCC client. More advanced functionality needs to be done from the SCC provider. Regarding the advance option input, support for that input is based on the provider and would be extremely difficult to document the usage to most users. It is mainly used by LabVIEW's SCC tools.
  QUOTE

  
  

• Finally, and I am not sure if I can explain this correctly but the built in SCC for ClearCAse seems to be built more around ClearCase UCM and not the far better (IMHO) Base ClearCase. The source control Project concept in LabVIEW for ClearCase at anty rate just does not work for me. I have several views and branchs on the go at any one time (1 view = 1 branch = 1 fix) when I finish with a view to goes, in fact to get LabVIEW Clearcase interface working I have a fake view mounted all the time just to stop LabVIEW complaining when I lauch it with a newly created view.

  
  LabVIEW calls generic functions via the MS SCC interface and Clearcase determines what support it provides. So if Clearcase decided the UCM interface was what they wanted, that's what you get.
  

I've avoided the discussion about why LabVIEW marks VIs as needing to be saved so I can focus on the SCC related issues. There is an option in Tools>>Options>>Environment - Do not save automatic changes. Here is the help for this option:

• Do not save automatic changes—Does not prompt you to save any changes automatically implemented by LabVIEW. You do not need to save these changes because LabVIEW implements the changes each time the VI is loaded. Changes automatically implemented by LabVIEW include recompiling, converting from an older version, and updating type definitions or fonts. This checkbox is unchecked by default.
  

The option is slaved to the "Treat read-only VIs as locked" option but you may find that you want that behavior anyway.

I'm glad you are able to write VIs to perform the operations that are not intrinsic in LabVIEW. I'm sure others will benefit from the work you do if you post to a code repository.

[lavezza]

My 2 cents.

Our current project has 6554 *.vi or *.ctl files. Having an "All VIs" vi is not an option

We are constantly being bitten by recompiles. On one hand you don't want a lot of checkins for VIs that weren't really changed. (Especially when your customer requires an explanation for changes in source files from one delivery to the next) On the other hand you don't want to leave the system in a state where 1000's of VIs are recompiled every time you run.

Here is what I'd like (and I've passed this on to NI already):

Right now we have source & object code in same file, run time menus in separate file.

[sample.vi & sample.rtm]

I would like the option of saving source and object code separately and the option of saving run time menus in the source.

[sample.lvs & sample.lvo] = LabVIEW source with internal menu and object code files

[sample.lvs & sample.lvo & common.rtm] = LabVIEW source, object code file and external run time menu (because sometimes multiple displays share a menu structure)

[sample.vi] = Single file with source, object code and menu

[sample.vi & sample.rtm] = Just like now

So you would be able to configuration control the *.lvs & *.rtm files. The *.lvo files would be recompiled as needed and not be tracked by version control. Same as traditional languages. Basically, every SCC system assumes SOURCE is being controlled, not OBJECT files. LV needs to support this separation if we really want to take advantage of common SCC systems. Single files were fine when most LV projects were small programs developed by small teams. Not every LV program fits that description anymore.

The other option would be for someone to alter a SCC program to compare *.vi files only by the source part of the file. This would require someone to know that internal structure of the current *.vi filetype. I'm guessing it would also be slower. Not something I want to tackle.

As for the integration into the LV editor, it would be nice if the interface was opened up a little. Obviously there are limits to what can be done with MSSCC. Couldn't NI provide a way to let you develop your own VIs that are called when the SCC menu options are selected in LV. Right now the Perforce VIs call the Perforce command line. Someone could to the same thing for Subversion. If the SCC menus were configurable, someone could develop VIs that call the ClearCase command line (or use ActiveX?) for the advanced options that aren't supported by the MSSCC API. I think that NI could definitely update LV so that the SCC integration is more of an API. They could provide the basic MSSCC and Perforce VIs and let the community develop more advanced options.

[gmart]

QUOTE

Our current project has 6554 *.vi or *.ctl files. Having an "All VIs" vi is not an option

Why is this an issue? Are you concerned with possible slowdows or memory usage?

QUOTE

I would like the option of saving source and object code separately and the option of saving run time menus in the source.

Having separate source and object code is a long standing request. Just because it hasn't happened doesn't mean it hasn't been considered.

QUOTE

As for the integration into the

LV

editor, it would be nice if the interface was opened up a little. Obviously there are limits to what can be done with MSSCC. Couldn't NI provide a way to let you develop your own VIs that are called when the SCC menu options are selected in

LV

. Right now the Perforce VIs call the Perforce command line. Someone could to the same thing for Subversion. If the SCC menus were configurable, someone could develop VIs that call the ClearCase command line (or use ActiveX?) for the advanced options that aren't supported by the MSSCC API. I think that NI could definitely update

LV

so that the SCC integration is more of an API. They could provide the basic MSSCC and Perforce VIs and let the community develop more advanced options.

If you are interested in this, I would contact your sales rep and work through the system to see if something can be arranged. It would be great if the LabVIEW community would develop custom SCC providers. If you wanted to work with the functionality that is currently there, let's just say that it wouldn't be too hard to look at the current provider implementations and deduce your way to a functioning SCC backend :-)

[lavezza]

QUOTE(gmart @ Nov 21 2007, 04:36 PM)

Why is this an issue? Are you concerned with possible slowdows or memory usage?

Both. I don't even think I have enough RAM to open all the VIs, let alone run them. Obviously a lot of our VIs are called dynamically. Besides, having 6000 open windows would be a little hard to handle.

QUOTE(gmart @ Nov 21 2007, 04:36 PM)

Having separate source and object code is a long standing request. Just because it hasn't happened doesn't mean it hasn't been considered.

I can read that two ways: "It's been considered and rejected" OR "It's been considered and I'm not going to tell you more than that."

QUOTE(gmart @ Nov 21 2007, 04:36 PM)

If you are interested in this, I would contact your sales rep and work through the system to see if something can be arranged. It would be great if the LabVIEW community would develop custom SCC providers. If you wanted to work with the functionality that is currently there, let's just say that it wouldn't be too hard to look at the current provider implementations and deduce your way to a functioning SCC backend :-)

I've looked at the Perforce specific VIs and came to the same conclusion. I didn't dig much deeper than that but I did see that there are other things that would need to be done for it to work. For example, the preferences for LabVIEW only display Perforce and providers that have MSSCC interfaces, but only if they are installed. And it looks like the code that determines which VIs to call is locked. There needs to be a way to tell LabVIEW 'Add Subversion to the list of SCC Providers and call the VIs in the \vi.lib\SourceControl\Providers\SVNCommandLine folder'. There may be a way to do this now, but it certainly isn't documented. Adding provider specific menu items to LabVIEW would be another (currently impossible?) task.

[Ton Plomp]

QUOTE(lavezza @ Nov 22 2007, 03:21 AM)

....'Add Subversion to the list of SCC Providers...

There is a plugin to access SVN via the MSSCC interface, it'll cost you €19 at http://www.pushok.com' target="_blank">www.pushok.com.

But when I tried to acces the MSSCC API from LabVIEW, it would give pop-ups with the low-level information (not good for a automation tool te be run in a buildbatch)

Ton

[dannyt]

QUOTE(tcplomp @ Nov 22 2007, 07:28 AM)

There is a plugin to access SVN via the MSSCC interface, it'll cost you €19 at www.pushok.com.

But when I tried to acces the MSSCC API from LabVIEW, it would give pop-ups with the low-level information (not good for a automation tool te be run in a buildbatch)

Ton

Ton,

The answer, with any SCC provider, is the same if you wish to run automated build / deployment sysyems. You need to get down with the low level commands provided by the SCC provider. In this respect LabVIEW is no different for any other language.

ClearCase, CVS and Subversion all provide access to thier full functionallity via the comand line. All three of them can only really be used to thier full potential when used with tools you write yourself to get them to do what you want, it is that ability that makes them so great to use in these situations....... but you do need to really understand the SCC tools you are using.

I have worked on many automated build & deploy systems using C, C++, Java and Databases in both ClearCase and CVS and in all case's, the automation was achieved by writting our own scritps (usually for me in perl or ksh) to direclty access and mulipulate the SCC systems, they are intended to be used in this way.

I have never worked on a project where the tool suite to automate build / release etc, used the MSSCC interface, it provides a very small level of functionallity that only really allow IDE's to be tied into a SCC system and makes checkout checkin easy.

For ClearCase there are three ways to go 1) the ClearCase automation library (CAL) this will not do everything you masy want. 2) ClearPerl a well established Perl module for doing this in ClearCase and 3) directly calling the cleartool commands via either a command promt or better still a unix shell.

I have not done this stuff with subversion but I would start with writting my own LabVIEW tools library for subversion calling the subversion command line client

an added after-thought

Having just written this reply I realise my previous post was being a little unfair to NI, when I replied to Ton's question of why I bypassed the LVSCC interface, though I still stand by everything I said, it is fair to say that these issues are true to all languages I have come across and it is not unreasonable that as automated build / deployments issues vary so much from user to user that one needs to step beyond the normal integrated functions provided.

cheers

Dannyt

[dannyt]

QUOTE(tcplomp @ Nov 22 2007, 07:28 AM)

There is a plugin to access SVN via the MSSCC interface, it'll cost you €19 at www.pushok.com.

But when I tried to acces the MSSCC API from LabVIEW, it would give pop-ups with the low-level information (not good for a automation tool te be run in a buildbatch)

Ton

Ton,

The answer, with any SCC provider, is the same if you wish to run automated build / deployment sysyems. You need to get down with the low level commands provided by the SCC provider. In this respect LabVIEW is no different for any other language.

ClearCase, CVS and Subversion all provide access to thier full functionallity via the comand line. All three of them can only really be used to thier full potential when used with tools you write yourself to get them to do what you want, it is that ability that makes them so great to use in these situations....... but you do need to really understand the SCC tools you are using.

I have worked on many automated build & deploy systems using C, C++, Java and Databases in both ClearCase and CVS and in all case's, the automation was achieved by writting our own scritps (usually for me in perl or ksh) to direclty access and mulipulate the SCC systems, they are intended to be used in this way.

I have never worked on a project where the tool suite to automate build / release etc, used the MSSCC interface, it provides a very small level of functionallity that only really allow IDE's to be tied into a SCC system and makes checkout checkin easy.

For ClearCase there are three ways to go 1) the ClearCase automation library (CAL) this will not do everything you masy want. 2) ClearPerl a well established Perl module for doing this in ClearCase and 3) directly calling the cleartool commands via either a command promt or better still a unix shell.

I have not done this stuff with subversion but I would start with writting my own LabVIEW tools library for subversion calling the subversion command line client

an added after-thought

Having just written this reply I realise my previous post was being a little unfair to NI, when I replied to Ton's question of why I bypassed the LVSCC interface, though I still stand by everything I said, it is fair to say that these issues are true to all languages I have come across and it is not unreasonable that as automated build / deployments issues vary so much from user to user that one needs to step beyond the normal integrated functions provided.

cheers

Dannyt

[silmaril]

QUOTE(lavezza @ Nov 21 2007, 08:12 PM)

Basically, every SCC system assumes SOURCE is being controlled, not OBJECT files. LV needs to support this separation if we really want to take advantage of common SCC systems. Single files were fine when most LV projects were small programs developed by small teams. Not every LV program fits that description anymore.

You are right. Separating source and object code would help "real" LabVIEW programmers a lot.

It seems to me that NI is already on the way to do this.

In fact they are already doing it for the FPGA VIs (ok, it's no object code there, it's a bitstream).

So let's hope this really is a good omen for the future of "standard" LabVIEW...

[gmart]

QUOTE

I've looked at the Perforce specific VIs and came to the same conclusion. I didn't dig much deeper than that but I did see that there are other things that would need to be done for it to work. For example, the preferences for LabVIEW only display Perforce and providers that have MSSCC interfaces, but only if they are installed. And it looks like the code that determines which VIs to call is locked. There needs to be a way to tell LabVIEW 'Add Subversion to the list of SCC Providers and call the VIs in the \vi.lib\SourceControl\Providers\SVNCommandLine folder'. There may be a way to do this now, but it certainly isn't documented. Adding provider specific menu items to LabVIEW would be another (currently impossible?) task.

You are correct that the mechanism is not documented but that doesn't mean that it's not extendable. The list of VIs in the "sccapi" directory should give you an idea as to what VIs need to be written for a new plugin (as a hint, "Find All Providers" is where the list of "installed" plugins is returned to the SCC config page). As far as provider specific menus, the task is not impossible, but if you want to go down that road, I would again suggest you talk to your sales rep and see what arrangements could be made to give you more information.

I think it would be great if the LabVIEW developer with domain expertise with various SCC plugins developed custom SCC providers. Maybe someone could develop an SVN plugin for LabVIEW so I'd quit hearing about how Tortoise SVN is the best thing since sliced bread

[lavezza]

QUOTE(gmart @ Nov 26 2007, 08:36 AM)

You are correct that the mechanism is not documented but that doesn't mean that it's not extendable. The list of VIs in the "sccapi" directory should give you an idea as to what VIs need to be written for a new plugin (as a hint, "Find All Providers" is where the list of "installed" plugins is returned to the SCC config page).

You're right. After my last post I took another look and realized it was easier than I thought. I'm not sure how I missed that the last time. I think it would be fairly easy to take the P4CMD folder and make an equivalent for SVN (or any other SCC that has a command line interface).

Pat

[gmart]

QUOTE(lavezza @ Nov 26 2007, 11:17 AM)

You're right. After my last post I took another look and realized it was easier than I thought. I'm not sure how I missed that the last time. I think it would be fairly easy to take the P4CMD folder and make an equivalent for SVN (or any other SCC that has a command line interface).

Pat

Good luck and I look forward to seeing an SVN command line plugin in the future :thumbup: