How do you debug your RT code running on Linux targets ?

[Zyl]

Hi everybody,

Currently working on VeriStand custom devices, I'm facing a 'huge' problem when debugging the code I make for the Linux RT Target. The console is not availbale on such targets, and I do not want to fall back to the serial port and Hyperterminal-like programs (damn we are in the 21st century !! )...

Several years ago (2014 if I remember well) I posted an request on the Idea Exchange forum on NI's website to get back the console on Linux targets. NI agreed with the idea and it is 'in development' since then. Seems to be so hard to do that it takes years to have this simple feature back to life.

On my side I developped a web-based console : HTML page displaying strings received through a WebSocket link. Pretty easy and fast, but the integration effort (start server, close server, handle push requests, ...)must be done for each single code I create for such targets.

Do you have any good tricks to debug your code running on a Linux target ?

[hooovahh]

I think the more difficult part isn't debugging a Linux RT system, but instead debugging a Linux RT system and Veristand.  For me debugging a Linux RT system is pretty easy, just plugin a monitor and look at the front panel of a VI that I push debug data to through a functional global or other methods.  But with Veristand taking over a bit I imagine you don't have that level of control.  I'd almost just suggest you log to a file the status and debug information, and then view that file via FTP other means on a remote computer.  It does suck that the console isn't available, I didn't realize it wasn't an option since in MAX it still looks like it can be enabled there.

[Zyl]

Thank you for your answer. Yes indeed, the debug is difficult in the case you're using VeriStand : you have no way to get the RT VI FP when it runs on the target. File is an idea but you can only read it when th execution is done, not 'live'.

In MAX you have the console option, yes. But it only enables sending strings on the serial port.

Edited May 10, 2017 by Zyl

[smithd]

https://forums.ni.com/t5/NI-Linux-Real-Time-Documents/Getting-the-Most-Out-of-your-NI-Linux-Real-Time-Target/ta-p/3523211

syslog. I believe you can automatically view it from the web interface if you walk through their steps. You can also open up a console and just type "cat syslog" every few seconds to view it semi-real-time.

also: I don't know how but I'm willing to bet you can make some sort of pipe redirect on linux where everything written to syslog also gets displayed on the console.

also: I don't remember how but I could swear there is a file in the /dev folder which will output to a ssh session via putty, and so you could have labview open a write session to one of those files and get it to display on your shell.

probably simpler is to try to encapsulate the core parts of the code and debug outside of veristand before putting it into the engine.

Edited May 10, 2017 by smithd

[Zyl]

Syslog is good point! I'll try that in future devs.

But I guess that it will again a 'pain' to have this working with VeriStand...

[Rolf Kalbermatter]

We don't use Veristand, but we definitely use syslog in our RT applications quite extensively. In fact we use a small Logger class library that implements either file or syslog logging. I'm not sure what you would consider a pain to have such a solution working in VeriStand though.

Somewhere during your initialization you configure and enable the syslog (or filelog) and then you simply have a Logger VI that you can drop in anywhere you want. Ours is a polymorphic VI with one version acting as a replacement for the General Error Handler.vi  and the other being for simply reporting random messages to the logging engine.

After that you can use any of the various syslog viewer applications to have a life update of the messages on your development computer or anywhere else on the local network.

Edited May 12, 2017 by rolfk

[Zyl]

Hi Rolf, 

I definitely should look deeper at Syslog. I don't like the 'file' option because it doesn't tell me 'immediately' what is going on in the target. And most of the targets I have are equipped with a tiny HDD (mostly 9063) so the file size monitoring would be 'critical'.

I'm pretty sure it can work with VeriStand, but I must ensure that this solution can work on other supported targets (I've got PharLap and Windows targets as well). And again, I've got to design a specific (but reusable) monitoring thread in each custom device that may be played on the targets. Compared to the console (very simple and enough for fast debug) that was native on the other targets, IMO the lack of this tool on Linux RT targets complicates things.

Do you have a syslog viewer app to recommend ? Also a specific syslog LV lib that can be used ?

Thanks !

[Rolf Kalbermatter]

6 hours ago, Zyl said:

Hi Rolf, 

I definitely should look deeper at Syslog. I don't like the 'file' option because it doesn't tell me 'immediately' what is going on in the target. And most of the targets I have are equipped with a tiny HDD (mostly 9063) so the file size monitoring would be 'critical'.

I'm pretty sure it can work with VeriStand, but I must ensure that this solution can work on other supported targets (I've got PharLap and Windows targets as well). And again, I've got to design a specific (but reusable) monitoring thread in each custom device that may be played on the targets. Compared to the console (very simple and enough for fast debug) that was native on the other targets, IMO the lack of this tool on Linux RT targets complicates things.

Do you have a syslog viewer app to recommend ? Also a specific syslog LV lib that can be used ?

Thanks !

Well as far as the Syslog functionality itself is concerned, we simply make use of the NI System Engineering provided library that you can download through VIPM. It is a pure LabVIEW VI library using the UDP functions and that should work on all systems.

As to having a system console on Linux there are many ways for that which Linux comes actually with, so I'm not sure why it couldn't be done. The problem under Linux is not that there are none, but rather that there are so many different solutions that NI maybe decided to not use any specific one, as Unix users can be pretty particular what they want to use and easily find everything else simply useless. 

[smithd]

7 hours ago, rolfk said:

Well as far as the Syslog functionality itself is concerned, we simply make use of the NI System Engineering provided library that you can download through VIPM. It is a pure LabVIEW VI library using the UDP functions and that should work on all systems.

As to having a system console on Linux there are many ways for that which Linux comes actually with, so I'm not sure why it couldn't be done. The problem under Linux is not that there are none, but rather that there are so many different solutions that NI maybe decided to not use any specific one, as Unix users can be pretty particular what they want to use and easily find everything else simply useless. 

Thats the one I had used forever, but the post I put above is a specific binary+configuration for wrapping the built-in code on linux rt. If you click on August 2016 pdf 1559 kb and go to pg 4 you'll see what I'm talking about where the log gets routed automatically to the event viewer in system web server.

 

As for viewers with different features, I've used and like Kiwi: http://www.kiwisyslog.com/kiwi-syslog-server
Its got a ton of features for filtering and routing and such, and it has a nice web interface. 

If you just want to monitor 1 node then maybe you don't need something like that, I'm sure you can find a free option out there like https://github.com/tmanternach/WebSysLog or https://github.com/rla/mysql-syslog-viewer in conjunction with http://www.rsyslog.com/

Personally though I think the viewers end up getting in the way, so I normally just open the log in notepad++ and ctrl+f for stuff.

Edited May 16, 2017 by smithd

[Rolf Kalbermatter]

5 hours ago, smithd said:

Thats the one I had used forever, but the post I put above is a specific binary+configuration for wrapping the built-in code on linux rt. If you click on August 2016 pdf 1559 kb and go to pg 4 you'll see what I'm talking about where the log gets routed automatically to the event viewer in system web server.

That's of course another possibility but the NI Syslog Library works well enough for us. It doesn't plug directly into the Linux syslog but that is not a big problem in our case.

Quote

Personally though I think the viewers end up getting in the way, so I normally just open the log in notepad++ and ctrl+f for stuff.

It depends. In a production environment it can be pretty handy to have a life view of all the log messages, especially if you end up having multiple cRIOs all over the place which interact with each other. But it is always a tricky decision between logging as much as possible and then not seeing the needle in the haystack or to limit logging and possibly miss the most important event that shows where things go wrong. With a life viewer you get a quick overview but if you log a lot it will be usually not very useful and you need to look at the saved log file anyhow afterwards to analyse the whole operation.

Generally, once debugging is done and the debug message generation has been disabled, a life viewer is very handy to get an overall overview of the system, where only very important system messages and errors get logged anymore.

[Zyl]

17 hours ago, rolfk said:

Well as far as the Syslog functionality itself is concerned, we simply make use of the NI System Engineering provided library that you can download through VIPM. It is a pure LabVIEW VI library using the UDP functions and that should work on all systems.

As to having a system console on Linux there are many ways for that which Linux comes actually with, so I'm not sure why it couldn't be done. The problem under Linux is not that there are none, but rather that there are so many different solutions that NI maybe decided to not use any specific one, as Unix users can be pretty particular what they want to use and easily find everything else simply useless. 

I perfectly understand that there are many options to trace that comes with Linux. But I don't think that NI not picking one solution is a good idea. When you use for years the RT targets and get use to use the console to debug, when you switch to Linux RT you've got nothing anymore... or at least nothing immediately operational... All previous code routing messages to the console work without error, but actually do nothing, and you keep searching for the 'console'. When you do that in conjonction with 'linux' keyword, you just fall onto posts saying that the console is now on ... the serial port !

So I agree with you that Linux RT provides a lot of possibilities but most of the people using RT targets are not Unix-fan-professionnals... A clear position from NI saying that 'RT console' VIs should be changed to 'Syslog' VIs or even better, directly provide the syslog functionnality onto Linux RT targets when you use this VI would have been clever... (my 2 cents...)

[smithd]

2 hours ago, Zyl said:

I perfectly understand that there are many options to trace that comes with Linux. But I don't think that NI not picking one solution is a good idea. When you use for years the RT targets and get use to use the console to debug, when you switch to Linux RT you've got nothing anymore... or at least nothing immediately operational...

If I'm understanding you, the console you describe is only available on pharlap systems with video hardware. cRIOs have always had to debug through something like syslog or through the serial port, and in my experience I've only ever used the serial port to monitor NI's failures, not my own. 

I do agree that NI should build the syslog stuff into the product and it would be nice if there was a clear and obvious way to say 'write this output string to any connected ssh session'...but its not like the current level of functionality is a change for most cRIO users so I can't imagine this is a priority.

[Zyl]

4 minutes ago, smithd said:

If I'm understanding you, the console you describe is only available on pharlap systems with video hardware. cRIOs have always had to debug through something like syslog or through the serial port, and in my experience I've only ever used the serial port to monitor NI's failures, not my own. 

I do agree that NI should build the syslog stuff into the product and it would be nice if there was a clear and obvious way to say 'write this output string to any connected ssh session'...but its not like the current level of functionality is a change for most cRIO users so I can't imagine this is a priority.

The 'console' is there on VxWorks and PharLaps system since ... a long time (I would say since the first release of LV RT, but I'm not 100% sure). You can see it indeed with video-hardware equipped targets but also on the configuration web page. So the video hardware is not necessary. Publishing string is done calling 'RT Debug String' VI.

Works fine on VxWorks and PharLap cRIOs, and has absolutely no effect on Linux RT cRIOs (no error, no effect, seems to be no op when 'Write to monitor' is selected). When you are migrating code from 1 platform to another, you expect keeping the same behaviour (tracing access in this case).

[hooovahh]

I didn't see it linked here but I found your Idea Exchange, and luckily I already kudo'ed it.  Too bad the RT Idea Exchange, along with several others, get very little attention from NI.  A little while ago I was looking for a feature in RT and noticed an idea was posted for it so I kudo'ed it and talked to NI.  Turns out NI already had implemented it internally and just had never released it.  The feature could be documented and released 8 years ago but NI didn't know there was that much of a need for it.  I pointed them to the Idea Exchange where at least one employee (close to RT support) seemed unaware that the idea was even posted, let alone that it was gaining traction.

[Zyl]

12 minutes ago, hooovahh said:

I didn't see it linked here but I found your Idea Exchange, and luckily I already kudo'ed it.  Too bad the RT Idea Exchange, along with several others, get very little attention from NI.  A little while ago I was looking for a feature in RT and noticed an idea was posted for it so I kudo'ed it and talked to NI.  Turns out NI already had implemented it internally and just had never released it.  The feature could be documented and released 8 years ago but NI didn't know there was that much of a need for it.  I pointed them to the Idea Exchange where at least one employee (close to RT support) seemed unaware that the idea was even posted, let alone that it was gaining traction.

This idea is mine and I posted back in 2014... Since then it is maked as 'In dev'.... In 2014 I was mainly developping my own RT apps, with my own framework (that is why I built a web console based on WebSocket). But now I mainly work with VeriStand. Even if my console is still usefull, it is even less easy to use it with VS. Looking at the number of Kudos on thsi idea I tought that NI would have put more eforts into it, it such a basic option...

[Rolf Kalbermatter]

1 hour ago, Zyl said:

This idea is mine and I posted back in 2014... Since then it is maked as 'In dev'.... In 2014 I was mainly developping my own RT apps, with my own framework (that is why I built a web console based on WebSocket). But now I mainly work with VeriStand. Even if my console is still usefull, it is even less easy to use it with VS. Looking at the number of Kudos on thsi idea I tought that NI would have put more eforts into it, it such a basic option...

Unfortunately, 27 kudos is very little! Many of the ideas that got implemented had at least 400 and even that doesn't guarantee at all that something gets implemented.

[hooovahh]

Well it's the 16th most kudo'ed idea on the RT section.  (again highlighting the small amount of attention it gets)

[JKSH]

5 hours ago, hooovahh said:

Well it's the 16th most kudo'ed idea on the RT section.  (again highlighting the small amount of attention it gets)

Kudo added, FWIW