Is anyone successfully running a build server that actually builds reliably?

[ThomasGutzler]

Hi,

I've been working on this for years but I still haven't managed to set up a build server that builds reliably.

My inputs are a vipc file and a .lvproj with all relevant files checkout via git. From there it's a two-step process. A powershell script makes 2 separate calls to labview using two small helper vis:

1.) apply vipc file using VIPM API; close labview entirely

2.) build the project given a build spec name

In the end of step 2 it comes down to this

And I have no idea if the last 2 steps are needed but I'm definitely trying to work around problems there, such as:

- some files have changed, because LabVIEW can't just open a project without touching it. So I need to save everything to avoid the "Don't you want to save" dialog blocking everything

- I can't remember what the "Exit" is for

I've played with clearing the build and compiled object cache before every build. I've just turned off automatic recovery saving on my build servers. But I still find myself with builds that just never finish. Often, the last line in my build log is "Attempting to build <whatever>". Of course there is no way to debug a build that's gone wrong because I can't just remote into the build server and look at what's going on because the labview instance that's been started by the build pipeline isn't visible on my desktop even though it's the same user. So all I can do is manually restart, and in most cases that fixes the problem. Of course, the problem is not fixed then ...

Sometimes I remote into the build server and I see a vipm crash handler window. Sometimes I get the old "Sorry for the inconvenience" thing from labview but nothing is ever repeatable

Are there any other recommended settings? Should I mark all files as read-only to stop labview from touching them?

Any tricks, recommendations or other mildly related wisdom?

[ShaunR]

This may help.

 

There is also

but I can't remember which dialogues it suppresses.

Edited April 19, 2021 by ShaunR

[LogMAN]

4 hours ago, ThomasGutzler said:

So I need to save everything to avoid the "Don't you want to save" dialog blocking everything

You could run the build in its own context, which (to my knowledge) does not save any changes unless option 0x2 is active (see Wiki for more details).

Application class/Context.Create Local Host App Instance method - LabVIEW Wiki

[ThomasGutzler]

Thanks! I'm going to try the LocalHostAppInstance.

Looking at the options we have, is it fair to assume that I can ALSO set unattended mode or might that already be included?

Would I be able to set all those things in the .ini file instead of starting an instance inside an instance? I can't find any documentation about it

[MikaelH]

I've not dared to outsource the building to a build server yet, mostly because we get these strange build errors from time to time, like "This VI broke during the build".

Also since my team does quite many builds and we would need multiple build servers since a build could take an hour, and sometime I need to build 6 version right away and that could lock up the build server for 6 hours.
But if you get it working close to 100% let me know, then we might switch over.

[ThomasGutzler]

We've been building executables and vipm packages on Azure for quite a while now and most builds work. Our driver package (6500 files) takes about 45 minutes on a pretty low spec PC. Applications usually build within 10 minutes.

All our builds are LabVIEW 2018 64bit though, no multi-version doom. But there's no reason why you couldn't build multiple versions.

With Azure build pipelines you have the choice of either self-hosting your build server or using an Azure VM which you have to install the complete toolchain on. Azure VMs can spin up/down on demand which is supposed to save you money because you only pay for the time they're up. We're not doing that.

Another cool thing is unit testing. The caraya test framework integrates nicely and posts your test results stright to the pipeline. More here: https://forums.vipm.io/topic/3293-unit-testing-with-azure/

Happy to share more thoughts - probably in a different thread or PM

[LogMAN]

1 hour ago, ThomasGutzler said:

Looking at the options we have, is it fair to assume that I can ALSO set unattended mode or might that already be included?

It creates a headless instance, so no dialogs. I haven't checked the unattended mode flag, but I'd assume it is set by default 🤷‍♂️

[joerghampel]

Answering the question in this post's title: Yes! 

We're very happy with our solution, we can apply .vipc's, validate DQMH modules, run VI Analyzer and Unit Tests, create documentation from source code, execute build specifications, create .vip's, package results into zip archives and deploy (move the result files somewhere). We have also created plugins for our dokuwiki which will automatically compile a release list with links to the artefacts. At the moment, we're working on adding FPGA compilation to the list of features.

It's a commercial product, but you might get some inspiration from the information on the product website at https://rat.hampel-soft.com. 

Edited April 28, 2021 by joerghampel

[ThomasGutzler]

On 4/24/2021 at 2:32 AM, joerghampel said:

Answering the question in this post's title: Yes!

It's a commercial product, but you might get some inspiration from the information on the product website at https://rat.hampel-soft.com. 

Thanks Joerg, I've seen your website before. Probably while I was looking for solutions.

I guess, in theory, my stuff works too. It's just the pain of debugging when things don't go as they should because LabVIEW decided to corrupt its compiled objects cache, or that build error that we get when we have the build folder open while it's building that sometimes happens even that folder isn't open, or solar flares...

The fact that I can't debug a build that froze or do anything useful with an instance that got started programmatically and isn't finishing just opens this immense guessing game. I don't like it.

[joerghampel]

19 hours ago, ThomasGutzler said:

I guess, in theory, my stuff works too. It's just the pain of debugging when things don't go as they should because LabVIEW decided to corrupt its compiled objects cache, or that build error that we get when we have the build folder open while it's building that sometimes happens even that folder isn't open, or solar flares...

I don't disagree. I've been working on our tools for ~ 10 years now, maybe 5 years running them automatically on a server. Some kinks could be ironed out, others worked around. It's definitely still a pain sometimes (add to the LabVIEW woes some other troubles like VMs losing network connectivity etc.).

All in all, the whole experience has taught us so much process-wise. And of course, once you got used something, you don't want to do without it anymore.

I believe that's the actual USP of our tools for our customers: The process we teach while integrating our tools into their environment.

[joerghampel]

...and for what it's worth, we use G-CLI, which sets the "App.UnattendedMode" property. That seems to be all it needs. G-CLI also allows to kill the LabVIEW process after some timeout in case it should hang.

[ThomasGutzler]

5 hours ago, joerghampel said:

...and for what it's worth, we use G-CLI, which sets the "App.UnattendedMode" property. That seems to be all it needs. G-CLI also allows to kill the LabVIEW process after some timeout in case it should hang.

Thanks for the insight! G-CLI has been on my to-look-into list for a while. Haven't had the time yet.
At the moment our only timeouts come from the build pipelines, which means if something goes wrong the pipeline fails and the developer has the press the "try again" button (After inspecting the build log to check if it's a true fail or not). Not the end of the world if the second build works.

[Rolf Kalbermatter]

On 4/23/2021 at 6:32 PM, joerghampel said:

It's a commercial product, but you might get some inspiration from the information on the product website at https://rat.hampel-soft.com. 

This link doesn't work. Maybe you need to setup an alias on your website. The correct location seems to be https://www.hampel-soft.com/release-automation-tools-for-labview/

Edited April 27, 2021 by Rolf Kalbermatter

[Antoine Chalons]

The link will work if you remove the extra period at the end

https://rat.hampel-soft.com

[Rolf Kalbermatter]

1 hour ago, Antoine Chalons said:

The link will work if you remove the extra period at the end

https://rat.hampel-soft.com

😃 lol, missed that one.

[joerghampel]

Yes, sorry, and thanks for pointing that out. It seems the editor thought the dot would make a nice addition to the URL, and I didn't catch it. I just fixed.

[Michael Aivaliotis]

Why are you saving the all VIs after the build?

[ThomasGutzler]

8 hours ago, Michael Aivaliotis said:

Why are you saving the all VIs after the build?

That was there to avoid the "Save modified files" dialog blocking everything.
That was before I knew about the headless instance.