High Frequency, Browser-Based Monitoring for cRIO-9068

[Mechatroner]

Hello LAVA.

 

Here's an idea for those interested in browser-based monitoring of cRIO-9068 performance.

 

A few years ago I developed the collectdViewer system as a demonstration to the "DevOps" community of the benefits of using WebSockets for server monitoring.  Operation of the system is quite simple conceptually: The system uses a collectd daemon (http://collectd.org) to monitor the performance parameters (eg CPU, memory, network activity) of a Linux platform.  This performance data is then sent via a message broker and a WebSocket connection to a browser for display.

 

The WebSocket connection to the browser enables a display update rate of 2x per second and as a result, it's possible to observe short transient phenomenon in the behavior of a server.  For example, attached is a screenshot of collectdViewer monitoring the performance of an EC2 instance during various system load tests in a 60 second period.  The high update rate also make it possible to observe phenomenon such as the network activity associated with serving a single web page or the CPU usage of the login process. 

 

Since the cRIO-9068 uses NI's Real-Time Linux operating system, it seems that it should be possible to use collectdViewer for browser-based monitoring of this platform as well.  If you'd like to try the system yourself for this application, more information and a downloadable demo are available at: http://collectdviewer.com. Let me know if you need help with setting up the demo system to monitor the performance of your cRIO-9068.

 

A few of other notes:

1) The downloadable demo of the system at the above link uses a message broker hosted on my cloud server.  If there's interest in using this system for autonomously monitoring your cRIO-9068, I can create completely standalone systems for a modest fee.  These systems would  include a Virtual Machine that hosts an HTTP server and message broker and can run on a PC, Mac or Linux platform within your LAN.

 

2) The video at the following link shows the use of collectdviewer for monitoring and studying the transfer of data from the (recently defunct) Nirvanix Internet File System to an EC2 Server:  http://collectdviewer.com/applications.html#monitoring_cloud_storage_data_transfer.

 

3) It would be feasible to incorporate the display of values of LabVIEW variables in the collectdViewer plots.  One way to do this would be to use the STOMP library that I developed for the LabSocket system to send LabVIEW data to the message broker of the collectdViewer system.

 

This idea is somewhat half-baked at this stage, but I thought I'd put it out there in case anyone had an interest in it.  Any thoughts or comments are welcome.

 

PS - Apologies for not including live hyperlinks in this post.  It appears that the forum editor software is blocking these links for some reason.

 

-John

jbergmans // at bergmans // dot com

[ShaunR]

The WebSocket connection to the browser enables a display update rate of 2x per second 

Is that a limitation of the STOMP server? I would expect far better than that with websockets.

[hooovahh]

PS - Apologies for not including live hyperlinks in this post.  It appears that the forum editor software is blocking these links for some reason.

This is a recent change to the forums to combat the recent influx of spam.  Sorry for the inconvenience.  Admins at LAVA did not make the decision lightly and are trying their best to keep a community as free from spam as possible.

[Mechatroner]

Is that a limitation of the STOMP server? I would expect far better than that with websockets.

@ShauwnR

 

I haven't investigated the maximum update rate for collectdViewer, but 2 Hz is actually slightly beyond the limit at which the mobile devices that I've tested can reliably display the collectd data on a continuous basis.

 

This limit isn't due to WebSockets or the RabbitMQ message broker of the system.  Instead, I believe that the reason for this limit is due to the way the collectd data is processed in the browser:  Every 500 ms, the collectd daemon sends out a stream of updates for each measured parameter of the host platform (eg CPU User, CPU Nice, Memory Used, etc), in random order.  On the browser side, the JavaScript code allocates the data in this stream to the correct location in a multi-dimensional array containing the time-series data for each parameter.  Once all the parameters for the time step are received, all the plots are redrawn.  All of this activity must occur before the next stream of updates is received (ie within 500 ms).  For mobile devices, i think that what's happening is that any glitch in the transmission of data over the air throws off the processing of the data stream by the JS code.

 

To increase the reliability of the system when used with mobile devices, additional preprocessor logic could be added on the collectdViewer server which would organize and buffer the data before sending it on to the browser.

 

The polling interval of the collectd daemon is set in a configuration file, so it would be very easy to try decreasing this interval to see if a particular browser and platform combination can process the incoming data stream.

 

I should mention that the system does work very reliably with a Chrome browser on my Mac.

 

Finally, as an aside, have you seen the collectdViewer "Hall of Fame"?  http://collectdviewer.com/hall_of_fame.html.  These screenshots suggest to me that some interesting insights could be gained by using collectdViewer to monitor the performance of a LabVIEW Real-Time application on a cRIO.

 

@hooovahh

Thanks for that info.

 

-John

Edited November 5, 2013 by Mechatroner

[ShaunR]

@ShauwnR

 

I haven't investigated the maximum update rate for collectdViewer, but 2 Hz is actually slightly beyond the limit at which the mobile devices that I've tested can reliably display the collectd data on a continuous basis.

 

This limit isn't due to WebSockets or the RabbitMQ message broker of the system. Instead, I believe that the reason for this limit is due to the way the collectd data is processed in the browser: Every 500 ms, the collectd daemon sends out a stream of updates for each measured parameter of the host platform (eg CPU User, CPU Nice, Memory Used, etc), in random order. On the browser side, the JavaScript code allocates the data in this stream to the correct location in a multi-dimensional array containing the time-series data for each parameter. Once all the parameters for the time step are received, all the plots are redrawn. All of this activity must occur before the next stream of updates is received (ie within 500 ms). For mobile devices, i think that what's happening is that any glitch in the transmission of data over the air throws off the processing of the data stream by the JS code.

 

Indeed. Mobile dynamic display of graphical data is constrained and very browser dependent. Firefox especially seems to be very resource intensive since for something like the following demo which is using 2 channels @ 100 ms each; most mobile browsers can cope, but Firefox glitches often

 

http://37.235.49.79/example_gauges.html

 

However, the following demo chokes all mobile browsers as there are a lot of graphics to update and there are 4 channels @ 100ms each and one channel @ 1sec.. Additionally, the datapoint stack for the graphs is in the browser (the server is sending single datapoints) with each line having a history length of 100 datapoints once the graphs start scrolling.

 

http://37.235.49.79/example_dash.html

 

If you turn off the graphs then even Firefox can keep up, so the data rate and number of channels isn't the restriction. It's the graphical rendering that brings mobile devices to their knees. The demos aren't doing anything special though (like your matrix of data). The server is just spewing data out and the javascript redraws happen when each piece of data arrives. Desktops can cope with the redraw rate, but mobile devices can't.

Edited November 16, 2013 by ShaunR