dynamic content in other web servers

[smithd]

I've occasionally seen or heard from people wondering about hosting labview web services in other servers, like apache, microsoft IIS, or nginx.  They may already have a server they just want to plug into, or maybe they want to use some of the more advanced features available in other servers -- for example LDAP/activedirectory authentication (http://httpd.apache.org/docs/2.2/mod/mod_authnz_ldap.html). However dynamic content still needs to be provided somehow, from LabVIEW. I've never really done too much with web services and I was curious how this stuff works in other languages. I'm fairly certain I'm 'rediscovering' some of this, but I couldn't find what I was looking for anywhere in labview-land....so I thought I'd share what I found out and what I did about it in case anyone else was curious about the same. 

 

What seems to be the case is that when people using other languages want to add code to the back end of a web server, they have a few basic options:

• Compile code into the web server. This option is more difficult to reuse, as you're making a plugin for a particular server, but some servers can use plugins for other servers.
  
  • Apache has modules, for example mod_perl which is basically a dll which runs perl.
    
  • IIS can run all sorts of .net code and scripts
    
  • I *think* this is what LabVIEW has done for its web service, but can't confirm.
    
  
  
• Use a protocol between the front-end web server and the back-end application.
  
  • The simplest version is CGI -- the web server runs an exe, passes any post data as standard in, and waits for data on standard out or standard error. This is slooooow as it runs an exe every time.
    
  • This was improved with FastCGI where the web server launches N copies of the exe on boot and then leaves them running. These EXEs are sent packets which correspond to the CGI standard in data and respond with standard out and/or standard error packets. These packets can be sent through standard I/O (which is tough with lv) or through TCP (which is easy).
    
  • Some people use a simpler HTTP server running in their program as the back-end. It can be simpler and less robust than, say, apache because in theory you're only getting well-formed packets from the front-end http server.
    
    • Compared to FastCGI this can be even faster because the HTTP request doesn't have to be re-packaged into an FCGI request, but HTTP is single-threaded (for lack of a better term -- requests must be responded to in the order they were received). The other advantage is you can use a web browser or other http client to talk directly to the simple http server, which you can't do with FCGI (for debugging purposes). This is essentially a https://en.wikipedia.org/wiki/Reverse_proxy
      
    
    
  • There are some custom protocols used like WSGI which seems to be python-only, or things like java applets.
    
  
  

 

The point is, its pretty common to let something big and complicated like apache take care of the incoming requests on port 80 while letting one or more back-end servers handle specific requests as needed.

 

This could handled by the standard labview web services (ie Apache reverse-proxies to the standard labview web service) but I was still interested in how this stuff works so I made some (rough) code, which I've posted here, a few weeks back:

https://github.com/smithed/LVWebTools

 

tl;dr: Built, monolithic web-service-code like what labview does seems to be pretty uncommon. Most (citation needed) languages seem to just use a protocol between the main, very separate web server responsible for authentication, security, and static file serving and the tiny backend exe responsible for everything else. I made some neat tools to do this, in LabVIEW. Also, the internet really is just a series of tubes.

 

 

-----------------------------------------------------------------------------------------------------------

 

Note: the following is really into some of the weeds, if you don't want to try the code stop here.

 

I started with FastCGI because it seemed simpler...but it turned out not to be. You're still responsible for adding most of the headers and response codes and such into your response, so it seemed like it had less structure but that structure was actually necessary anyway most of the time.

 

If you want to take a look at it, FCGI/FCGI server.vi is the main VI and FCGI/default responder action.vi is the main VI I was using to test out different responses. With apache, you'd set it up using mod_proxy and mod_proxy_fcgi, but I'd actually recommend a new web server called Caddy (http://caddyserver.com/) because it is more developer friendly. The "caddyfile" you'd need would be as follows:

"0.0.0.0:80

fastcgi /lvroot 127.0.0.1:9000"

(adjusting the ports and such as needed of course). If you run the caddy server with that file, and then navigate to http://localhost/lvroot/anything it will invoke the labview code. With fastcgi, you manually add any headers you need to the top of the response, so as an example you'd have to write:

"Content-Type: text/html

 

<http><body>blah</body></http>"

 

After that I decided to try out http, which I quickly learned was a really gross protocol. The core of it is nice and simple, but the issue to me is that the transport-related headers are mixed in with content-related headers, and oh by the way you can shove extra headers in the content section if you want to, oh and hey if you want to change protocols in the middle of operation thats supported too. Its weird. But I got a verrrrry basic server loop up and running and its located in http/http server.vi. This one I actually used for something (the Freeboard thing I mentioned in this thread: https://lavag.org/topic/19254-interactive-webpage-vi-options/) so I made a basic class to inject some behavior. That class is located in http/http responder and provides the "get.vi", "post.vi", "put.vi", and "delete.vi" you'd expect. Since I was 90% of the way there anyway, I added a protocol upgrade function in order to pass functionality off to (for example) a websocket handler. This was totally not worth it (the code got more complex), but its cool that it works.

 

As above, I'd recommend caddy server and the appropriate line you'd want to add to your file is 

"proxy /lvhttp localhost:9001"

Because HTTP allows sending partial results, my implementation uses a queue to send data back...I think most of the fields are pretty obvious, except to note that the headers field is a variant attribute look up table with a string/string implementation (header/value). If a response code isn't specified, 200 is assumed.

 

Side note: Because I have a giant NI next to my name I feel the need to note that this is just a fun project, not intended in any way to replace the fully-supported in-product server which includes all the fancy things like authentication, URL routing, etc. My thought was that this could be handy for making really stupidly simple services that plug into big ones *already running* in apache, nginx, or IIS.

[ShaunR]

Nothing really to say except nice work.

I have come to the conclusion that too little rediscovery is sought and many would learn a great deal by doing exercises from first principles like this of technology we all take for granted.

Edited October 14, 2015 by ShaunR

[smithd]

Thanks

 

In the other thread I mentioned I used this to make a quick demo for freeboard using the CVT. I've attached the demo here.

To use it:

1. Unzip the attached freeboard demo.zip

2. Install the CVT (current value table) from VI Package manager

3. Download https://caddyserver.com/download and extract to the same folder as the zip. Caddy.exe and "caddyfile" should be in the same directory.

4. Optional: switch up the ports in "caddyfile".

5. Open zip/lv/sampleweb.lvproj, then open server.vi

6. Optional: switch up the ports

7. Press run

8. Run caddy.exe

9. Go to localhost in your browser.

10. At the top, press "Load Freeboard", then select zip/lv/http and ws.json

11. Change values on the front panel of server.vi.

 

There are 2 sets of data sources, one set uses websockets and the other uses http requests. Both provide the same data which looks like this:

{

"humidity": 2.377959,

"temp": 0.000000,

"label": "jkjk",

"set": [1],

"onoff": true

}

 

Note: the code is not great, just something I threw together really fast with pieces of the CVT web library on ni.com/referencedesigns. However one nice thing is you can see that creating a web service like this requires 2 functions, "get" and an initialization helper. The other nice thing is that the web service didn't start until after my application did, so I know exactly what the state of my system is when the ws starts (vs deploying from the project and then initializing later which is what CVT web has to do).

freeboard demo.zip