Jump to content

How do you debug your RT code running on Linux targets ?


Recommended Posts

Hi everybody,

Currently working on VeriStand custom devices, I'm facing a 'huge' problem when debugging the code I make for the Linux RT Target. The console is not availbale on such targets, and I do not want to fall back to the serial port and Hyperterminal-like programs (damn we are in the 21st century !! :P)...

Several years ago (2014 if I remember well) I posted an request on the Idea Exchange forum on NI's website to get back the console on Linux targets. NI agreed with the idea and it is 'in development' since then. Seems to be so hard to do that it takes years to have this simple feature back to life.

On my side I developped a web-based console : HTML page displaying strings received through a WebSocket link. Pretty easy and fast, but the integration effort (start server, close server, handle push requests, ...)must be done for each single code I create for such targets.

Do you have any good tricks to debug your code running on a Linux target ?

Link to post
Share on other sites

I think the more difficult part isn't debugging a Linux RT system, but instead debugging a Linux RT system and Veristand.  For me debugging a Linux RT system is pretty easy, just plugin a monitor and look at the front panel of a VI that I push debug data to through a functional global or other methods.  But with Veristand taking over a bit I imagine you don't have that level of control.  I'd almost just suggest you log to a file the status and debug information, and then view that file via FTP other means on a remote computer.  It does suck that the console isn't available, I didn't realize it wasn't an option since in MAX it still looks like it can be enabled there.

Link to post
Share on other sites

Thank you for your answer. Yes indeed, the debug is difficult in the case you're using VeriStand : you have no way to get the RT VI FP when it runs on the target. File is an idea but you can only read it when th execution is done, not 'live'.

In MAX you have the console option, yes. But it only enables sending strings on the serial port.

Edited by Zyl
  • Like 1
Link to post
Share on other sites

https://forums.ni.com/t5/NI-Linux-Real-Time-Documents/Getting-the-Most-Out-of-your-NI-Linux-Real-Time-Target/ta-p/3523211

syslog. I believe you can automatically view it from the web interface if you walk through their steps. You can also open up a console and just type "cat syslog" every few seconds to view it semi-real-time.

also: I don't know how but I'm willing to bet you can make some sort of pipe redirect on linux where everything written to syslog also gets displayed on the console.

also: I don't remember how but I could swear there is a file in the /dev folder which will output to a ssh session via putty, and so you could have labview open a write session to one of those files and get it to display on your shell.

probably simpler is to try to encapsulate the core parts of the code and debug outside of veristand before putting it into the engine.

Edited by smithd
  • Like 1
Link to post
Share on other sites

Syslog is good point! I'll try that in future devs.

But I guess that it will again a 'pain' to have this working with VeriStand...

Link to post
Share on other sites

We don't use Veristand, but we definitely use syslog in our RT applications quite extensively. In fact we use a small Logger class library that implements either file or syslog logging. I'm not sure what you would consider a pain to have such a solution working in VeriStand though.

Somewhere during your initialization you configure and enable the syslog (or filelog) and then you simply have a Logger VI that you can drop in anywhere you want. Ours is a polymorphic VI with one version acting as a replacement for the General Error Handler.vi  and the other being for simply reporting random messages to the logging engine.

After that you can use any of the various syslog viewer applications to have a life update of the messages on your development computer or anywhere else on the local network.

Edited by rolfk
  • Like 1
Link to post
Share on other sites

Hi Rolf, 

I definitely should look deeper at Syslog. I don't like the 'file' option because it doesn't tell me 'immediately' what is going on in the target. And most of the targets I have are equipped with a tiny HDD (mostly 9063) so the file size monitoring would be 'critical'.

I'm pretty sure it can work with VeriStand, but I must ensure that this solution can work on other supported targets (I've got PharLap and Windows targets as well). And again, I've got to design a specific (but reusable) monitoring thread in each custom device that may be played on the targets. Compared to the console (very simple and enough for fast debug) that was native on the other targets, IMO the lack of this tool on Linux RT targets complicates things.

Do you have a syslog viewer app to recommend ? Also a specific syslog LV lib that can be used ?

Thanks !

Link to post
Share on other sites
6 hours ago, Zyl said:

Hi Rolf, 

I definitely should look deeper at Syslog. I don't like the 'file' option because it doesn't tell me 'immediately' what is going on in the target. And most of the targets I have are equipped with a tiny HDD (mostly 9063) so the file size monitoring would be 'critical'.

I'm pretty sure it can work with VeriStand, but I must ensure that this solution can work on other supported targets (I've got PharLap and Windows targets as well). And again, I've got to design a specific (but reusable) monitoring thread in each custom device that may be played on the targets. Compared to the console (very simple and enough for fast debug) that was native on the other targets, IMO the lack of this tool on Linux RT targets complicates things.

Do you have a syslog viewer app to recommend ? Also a specific syslog LV lib that can be used ?

Thanks !

Well as far as the Syslog functionality itself is concerned, we simply make use of the NI System Engineering provided library that you can download through VIPM. It is a pure LabVIEW VI library using the UDP functions and that should work on all systems.

As to having a system console on Linux there are many ways for that which Linux comes actually with, so I'm not sure why it couldn't be done. The problem under Linux is not that there are none, but rather that there are so many different solutions that NI maybe decided to not use any specific one, as Unix users can be pretty particular what they want to use and easily find everything else simply useless. 

  • Like 2
Link to post
Share on other sites
7 hours ago, rolfk said:

Well as far as the Syslog functionality itself is concerned, we simply make use of the NI System Engineering provided library that you can download through VIPM. It is a pure LabVIEW VI library using the UDP functions and that should work on all systems.

As to having a system console on Linux there are many ways for that which Linux comes actually with, so I'm not sure why it couldn't be done. The problem under Linux is not that there are none, but rather that there are so many different solutions that NI maybe decided to not use any specific one, as Unix users can be pretty particular what they want to use and easily find everything else simply useless. 

Thats the one I had used forever, but the post I put above is a specific binary+configuration for wrapping the built-in code on linux rt. If you click on August 2016 pdf 1559 kb and go to pg 4 you'll see what I'm talking about where the log gets routed automatically to the event viewer in system web server.

 

As for viewers with different features, I've used and like Kiwi: http://www.kiwisyslog.com/kiwi-syslog-server
Its got a ton of features for filtering and routing and such, and it has a nice web interface. 

If you just want to monitor 1 node then maybe you don't need something like that, I'm sure you can find a free option out there like https://github.com/tmanternach/WebSysLog or https://github.com/rla/mysql-syslog-viewer in conjunction with http://www.rsyslog.com/

Personally though I think the viewers end up getting in the way, so I normally just open the log in notepad++ and ctrl+f for stuff.

Edited by smithd
  • Like 1
Link to post
Share on other sites
5 hours ago, smithd said:

Thats the one I had used forever, but the post I put above is a specific binary+configuration for wrapping the built-in code on linux rt. If you click on August 2016 pdf 1559 kb and go to pg 4 you'll see what I'm talking about where the log gets routed automatically to the event viewer in system web server.

That's of course another possibility but the NI Syslog Library works well enough for us. It doesn't plug directly into the Linux syslog but that is not a big problem in our case.

Quote

Personally though I think the viewers end up getting in the way, so I normally just open the log in notepad++ and ctrl+f for stuff.

It depends. In a production environment it can be pretty handy to have a life view of all the log messages, especially if you end up having multiple cRIOs all over the place which interact with each other. But it is always a tricky decision between logging as much as possible and then not seeing the needle in the haystack or to limit logging and possibly miss the most important event that shows where things go wrong. With a life viewer you get a quick overview but if you log a lot it will be usually not very useful and you need to look at the saved log file anyhow afterwards to analyse the whole operation.

Generally, once debugging is done and the debug message generation has been disabled, a life viewer is very handy to get an overall overview of the system, where only very important system messages and errors get logged anymore.

Link to post
Share on other sites
17 hours ago, rolfk said:

Well as far as the Syslog functionality itself is concerned, we simply make use of the NI System Engineering provided library that you can download through VIPM. It is a pure LabVIEW VI library using the UDP functions and that should work on all systems.

As to having a system console on Linux there are many ways for that which Linux comes actually with, so I'm not sure why it couldn't be done. The problem under Linux is not that there are none, but rather that there are so many different solutions that NI maybe decided to not use any specific one, as Unix users can be pretty particular what they want to use and easily find everything else simply useless. 

I perfectly understand that there are many options to trace that comes with Linux. But I don't think that NI not picking one solution is a good idea. When you use for years the RT targets and get use to use the console to debug, when you switch to Linux RT you've got nothing anymore... or at least nothing immediately operational... All previous code routing messages to the console work without error, but actually do nothing, and you keep searching for the 'console'. When you do that in conjonction with 'linux' keyword, you just fall onto posts saying that the console is now on ... the serial port !

So I agree with you that Linux RT provides a lot of possibilities but most of the people using RT targets are not Unix-fan-professionnals... A clear position from NI saying that 'RT console' VIs should be changed to 'Syslog' VIs or even better, directly provide the syslog functionnality onto Linux RT targets when you use this VI would have been clever... (my 2 cents...)

Link to post
Share on other sites
2 hours ago, Zyl said:

I perfectly understand that there are many options to trace that comes with Linux. But I don't think that NI not picking one solution is a good idea. When you use for years the RT targets and get use to use the console to debug, when you switch to Linux RT you've got nothing anymore... or at least nothing immediately operational...

If I'm understanding you, the console you describe is only available on pharlap systems with video hardware. cRIOs have always had to debug through something like syslog or through the serial port, and in my experience I've only ever used the serial port to monitor NI's failures, not my own. 

I do agree that NI should build the syslog stuff into the product and it would be nice if there was a clear and obvious way to say 'write this output string to any connected ssh session'...but its not like the current level of functionality is a change for most cRIO users so I can't imagine this is a priority.

Link to post
Share on other sites
4 minutes ago, smithd said:

If I'm understanding you, the console you describe is only available on pharlap systems with video hardware. cRIOs have always had to debug through something like syslog or through the serial port, and in my experience I've only ever used the serial port to monitor NI's failures, not my own. 

I do agree that NI should build the syslog stuff into the product and it would be nice if there was a clear and obvious way to say 'write this output string to any connected ssh session'...but its not like the current level of functionality is a change for most cRIO users so I can't imagine this is a priority.

The 'console' is there on VxWorks and PharLaps system since ... a long time (I would say since the first release of LV RT, but I'm not 100% sure). You can see it indeed with video-hardware equipped targets but also on the configuration web page. So the video hardware is not necessary. Publishing string is done calling 'RT Debug String' VI.

Works fine on VxWorks and PharLap cRIOs, and has absolutely no effect on Linux RT cRIOs (no error, no effect, seems to be no op when 'Write to monitor' is selected). When you are migrating code from 1 platform to another, you expect keeping the same behaviour (tracing access in this case).

cRIO Console.PNG

Link to post
Share on other sites

I didn't see it linked here but I found your Idea Exchange, and luckily I already kudo'ed it.  Too bad the RT Idea Exchange, along with several others, get very little attention from NI.  A little while ago I was looking for a feature in RT and noticed an idea was posted for it so I kudo'ed it and talked to NI.  Turns out NI already had implemented it internally and just had never released it.  The feature could be documented and released 8 years ago but NI didn't know there was that much of a need for it.  I pointed them to the Idea Exchange where at least one employee (close to RT support) seemed unaware that the idea was even posted, let alone that it was gaining traction.

  • Like 1
Link to post
Share on other sites
12 minutes ago, hooovahh said:

I didn't see it linked here but I found your Idea Exchange, and luckily I already kudo'ed it.  Too bad the RT Idea Exchange, along with several others, get very little attention from NI.  A little while ago I was looking for a feature in RT and noticed an idea was posted for it so I kudo'ed it and talked to NI.  Turns out NI already had implemented it internally and just had never released it.  The feature could be documented and released 8 years ago but NI didn't know there was that much of a need for it.  I pointed them to the Idea Exchange where at least one employee (close to RT support) seemed unaware that the idea was even posted, let alone that it was gaining traction.

This idea is mine and I posted back in 2014... Since then it is maked as 'In dev'.... In 2014 I was mainly developping my own RT apps, with my own framework (that is why I built a web console based on WebSocket). But now I mainly work with VeriStand. Even if my console is still usefull, it is even less easy to use it with VS. Looking at the number of Kudos on thsi idea I tought that NI would have put more eforts into it, it such a basic option...

Link to post
Share on other sites
1 hour ago, Zyl said:

This idea is mine and I posted back in 2014... Since then it is maked as 'In dev'.... In 2014 I was mainly developping my own RT apps, with my own framework (that is why I built a web console based on WebSocket). But now I mainly work with VeriStand. Even if my console is still usefull, it is even less easy to use it with VS. Looking at the number of Kudos on thsi idea I tought that NI would have put more eforts into it, it such a basic option...

Unfortunately, 27 kudos is very little! Many of the ideas that got implemented had at least 400 and even that doesn't guarantee at all that something gets implemented.

Link to post
Share on other sites
5 hours ago, hooovahh said:

Well it's the 16th most kudo'ed idea on the RT section.  (again highlighting the small amount of attention it gets)

Kudo added, FWIW

  • Like 1
Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Similar Content

    • By Yaw Mensah
      I have installed Labview 2020 on Dedian Buster using the rpm to deb conversion method via alien. Due to Architecture mismatch i deleted the *i386.rpm files before conversion.
      My Problem is that after creating a project at "Build Specification"-> "rigth click" i am only able to select "Source Distribution". Application does not show up as an option. 
      I will be grateful for any suggestions.
      Thank you in advance.
    • By McQuillan
      Hi Everyone,
      I (re)watched James Powell's talk at GDevCon#2 about Application Design Around SQLite. I really like this idea as I have an application with lots of data (from serial devices and software configuration) that's all needed in several areas of the application (and external applications) and his talk was a 'light-bulb' moment where I thought I could have a centralized SQLite database that all the modules could access to select / update data.
      He said the database could be the 'model' in the model-view-controller design pattern because the database is very fast. So you can collect data in one actor and publish it directly to the DB, and have another actor read the data directly from the DB, with a benefit of having another application being able to view the data.
      Link to James' talk: https://www.youtube.com/watch?v=i4_l-UuWtPY&t=1241s)
       
      I created a basic proof of concept which launches N-processes to generate-data (publish to database) and others to act as a UI (read data from database and update configuration settings in the DB (like set-point)). However after launching a couple of processes I ran into  'Database is locked (error 5) ', and I realized 2 things, SQLite databases aren't magically able to have n-concurrent readers/writers , and I'm not using them right...(I hope).
      I've created a schematic (attached) to show what I did in the PoC (that was getting 'Database is locked (error 5)' errors).
      I'm a solo-developer (and SQLite first-timer*) and would really appreciate it if someone could look over the schematic and give me guidance on how it should be done. There's a lot more to the actual application, but I think once I understand the limitations of the DB I'll be able to work with it.
      *I've done SQL training courses.
      In the actual application, the UI and business logic are on two completely separate branches (I only connected them to a single actor for the PoC) 
      Some general questions / thoughts I had:
      Is the SQLite based application design something worth perusing / is it a sensible design choice? Instead of creating lots of tables (when I launch the actors) should I instead make separate databases? - to reduce the number of requests per DB? (I shouldn't think so... but worth asking) When generating data, I'm using UPDATE to change a single row in a table (current value), I'm then reading that single row in other areas of code. (Then if logging is needed, I create a trigger to copy the data to a separate table) Would it be better if I INSERT data and have the other modules read the max RowId for the current value and periodically delete rows? The more clones I had, the slower the UI seemed to update (should have been 10 times/second, but reduced to updating every 3 seconds). I was under the impression that you can do thousands of transactions per second, so I think I'm querying the DB inefficiently. The two main reasons why I like the database approach are:
      External applications will need to 'tap-into' the data, if they could get to it via an SQL query - that would be ideal. Data-logging is a big part of the application. Any advice you can give would be much appreciated.
      Cheers,
      Tom
      (I'm using quite a few reuse libraries so I can't easily share the code, however, if it would be beneficial, I could re-work the PoC to just use 'Core-LabVIEW' and James Powell's SQLite API)

    • By PaulL
      Out of the box text in the icon editor looks awful. (See attached image, which is better looking than most.)
      (Yes, even with small fonts: https://forums.ni.com/t5/Linux-Users/Labview-Icons-under-GNOME/gpm-p/3379530.)
      Details:
      LabVIEW 2016 64-bit, CentOS 7 Linux OS
       
      We have tried many things to get this to work, to no avail. 
      Does anyone have a solution?

    • By szewczak
      I wanted to cross post metux's discovery here asap, and have a separate discussion.
      Metux's original post:
      The recent Linux driver package introduces a CRITICAL security vulnerability:
       
      http://www.ni.com/download/ni-linux-device-drivers-2018/7664/en/
       
      It adds additional yum/zypper repos, but explicitly disabling package signing and using unencrypted HTTP transport. That way, it's pretty trivial to completely takeover the affected systems, by injecting malicious packages.
       
       
      DO NOT INSTALL THIS BROKEN SOFTWARE - IT IS DANGEROUS !
       
      CERT and BSI are already notified.
       
       
       
       
       
    • By ensegre
      More out of curiosity than of hope: has anybody any idea why SVs are almost unsupported on linux? By almost I mean that controls and indicators cannot be bound to shared variables, and that shared variables cannot be programmatically created and looked up. I know that SVs hosted on windows can be accessed in linux LV using datasocket nodes, but that is all it gets. And it has been said that datasocket is despicable. What are the missing pieces that make SV windows-only?
      I didn't find much in the canonical places, so I posted a dumb zero-kudos attracting idea.
×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.