13 replies to this topic

[ShaunR]

Name: Dispatcher
Submitter: ShaunR
Submitted: 07 Sep 2010
File Updated: 03 Jan 2011
Category: Remote Control, Monitoring and the Internet
LabVIEW Version: 2009
License Type: Other (included with download)

This is a Publisher/Subscriber implementation of network communications.
The package is comprised of two main parts.

1. A Dispatcher that handles connection requests.
2. An API for interacting with the Dispatcher.

Overview.
Most people are probably familiar with RSS feeds. An RSS feed is an example of a publisher/subscriber implementation,
where information is "published" onto the internet and people may "subscribe" to updates using their browser or
some other client. A single published service may have millions on subscribers and publish its data every few hours as
data changes or periodically, maybe once or twice a day.

This package contains a similar implementation where a client "subscribes" to a "publisher", however, it is geared towards
high bandwidth data streaming across a local network. So, unlike its internet counterpart, it is not attempting to
service millions of clients who require an update once or twice a day. But service a few (maybe 1-10) clients every few seconds
or even milliseconds.

Detail:
The main focus of this package is the Dispatcher which facilitates the publisher/subscriber environment.
The implementation can support many network topologies that suit the developers requirements.
For example:
A single dispatcher may be located on a remote machine and all publishers and subscribers are remote and communicate
through it (centralised). Alternatively, there may be many dispatchers on different machines, each with their own
cluster of local services and the subscriber connects to the machine that contains the publisher of interest (decentralised).
Or even a mixture of both. The implementation is not rigid and allows for the developer to choose
the topology simply by installing once or more dispatchers and initiating a connection.

An API is provided to enable interaction with the Dispatcher to query its services, request a connection
and send/receive data.

Additional Features:
Supports data-stream encryption (blowfish).

Installation:
Unzip to a directory of your choice.

Required Packages:
Labview 9.0 or greater.
Transport.lvlib (Included)
Queue.vi (Included)
Position Form.vi (Included)
Stop.vi (Included)
Elapsed Time.vi (Included)

Known Issues.
None.

Versioning:
Current version 1.0.
See changelog.txt.

Contact:
PM ShaunR on lavag.org (http://www.lavag.org)

Click here to download this file

[Eugen Graf]

Hi!

Nice made, did you just seen this one http://lavag.org/top...ish-subscriber/ ?
Seems to be almost the same thing

Regards, Eugen

[ShaunR]

Hi!

Nice made, did you just seen this one http://lavag.org/top...ish-subscriber/ ?
Seems to be almost the same thing

Regards, Eugen

No I didn't Great minds think alike eh?

You talk about it being in the repository, but I'm unable to find it. Maybe it was lost along with the "The Great Crash" since the posts seem from quite a while ago. I wouldn't mind taking a look. Do you have a link?

The initial idea was born out of being fed up with writing similar things every time I worked with a new company. Although with this incarnation I wanted a more generic nature by removing a lot of the infrastructure dependency (which probably isn't that visible.....yet;)) and have the ability to cluster rather than rely on a central server (i.e. filtering). That way I can transparently use a distributed architecture. It is really just a multi client/server tracker. ( I also wanted a viable light-weight alternative to DCS. )

I see you also talk about "Topics" which suggests (to me) a single server with segmented data filtered for the particular subscribers (classic pub/sub). I think mine is more content-based where each "Topic" is in fact a dedicated server and after registration or subscription, the Dispatcher has little to do with data.

[Yair]

Is there any particular reason you didn't use LVOOP for the transport code? This is a natural candidate for a hierarchy of classes.

[ShaunR]

Is there any particular reason you didn't use LVOOP for the transport code? This is a natural candidate for a hierarchy of classes.

You make it sound as though thats a good thing

[Eugen Graf]

No I didn't Great minds think alike eh?

You talk about it being in the repository, but I'm unable to find it. Maybe it was lost along with the "The Great Crash" since the posts seem from quite a while ago. I wouldn't mind taking a look. Do you have a link?

Yes, it was lost after updating LAVAs software (I think). Here you can download my version:
http://labviewportal...hp?f=19&t=9#p19

I see you also talk about "Topics" which suggests (to me) a single server with segmented data filtered for the particular subscribers (classic pub/sub). I think mine is more content-based where each "Topic" is in fact a dedicated server and after registration or subscription, the Dispatcher has little to do with data.

I think it's the difference to your dispatcher. I send all the data to the server(dispatcher) and the dispatcher sends them to subscribers. It makes some advantages like debugging on one place, but disadvantages like more traffic on the server side.

But, sorry, I don't see where you send your data from one publisher to many subscribers. TCP/IP is one to one connection, so you have to send your data twice if you have two subscribers. I can't see any for-loop in TCIP Write.vi.


EDIT: my the web chat project (based on Pub/Sub Pattern) may be interesting to you http://lavag.org/top...uss-eugen-graf/

Edited by Eugen Graf, 14 September 2010 - 10:40 AM.

[ShaunR]

Yes, it was lost after updating LAVAs software (I think). Here you can download my version:
http://labviewportal...hp?f=19&t=9#p19

Sweet. I'll take a look!

But, sorry, I don't see where you send your data from one publisher to many subscribers. TCP/IP is one to one connection, so you have to send your data twice if you have two subscribers. I can't see any for-loop in TCIP Write.vi.

Indeed. That's where the "Disp Service Handler.vi" comes in.
This dispatcher uses a similar method to a web server where each incoming connection is negotiated to a different port and a process is spawned to handle that client. However. This spawning only happens for the Publishers. If you look in the Registration part of the Dispatcher you will see I dynamically launch the service handler (whos front panel is hidden). The publisher writes to the service handler, not the clients and the handler deals out the data to the clients. There is a service handler for each publisher and when a subscriber wishes to connect, the Dispatcher sends a new subscriber list to the handler for that service.

Edited by ShaunR, 14 September 2010 - 03:10 PM.

[Eugen Graf]

Ok, now I understand it. So you have a sender task for each publisher (service) on the dispatcher side. Why not one for all? My dispatcher does all, registration, unregistration an sending in one loop, but each service has own receiver loop. I think this is the biggest difference between our realisations of this pattern. The second less difference is you use functional global variable for the client list, but I don't.
For all, I think it's very interesting pattern, which can be used for dynamic applications, where many publishers and subscribers can communicate to each other.

Thank you

[ShaunR]

Ok, now I understand it. So you have a sender task for each publisher (service) on the dispatcher side. Why not one for all? My dispatcher does all, registration, unregistration an sending in one loop, but each service has own receiver loop. I think this is the biggest difference between our realisations of this pattern. The second less difference is you use functional global variable for the client list, but I don't.
For all, I think it's very interesting pattern, which can be used for dynamic applications, where many publishers and subscribers can communicate to each other.

Thank you

Mainly load balancing and performance. The performance is dictated by the computers resources rather than the dispatcher. With the filtered method you have a congestion point that affects ALL "Topics" and gets worse by a factor of subscribers x topics as you add more topics.

Lets say (for example) you are sending 100MB of data on 1 topic which takes (say 10 ms) to transmit to 1 subscriber (we'll ignore filtering overhead). The Dispatcher is able to service all subscribers to that topic every 10ms. If we now add a subscriber to that topic. The time to service all subscribers is now 20 ms. 3 subscribers...30 and so-on. Now we add another topic that also takes 10ms to tx 100MB of data to 3 more subscribers (keep the maths easy). The time now required to service all 6 subscribers on both topics is now 60 ms. 3 Topics, 90ms.......

However. with the service handler, each service handler is only handling 3 subscribers so in our theoretical example, each handler is servicing all 3 of the subscribers every 30 ms still. Increasing the number of topics doesn't change the time to service all clients. So whereas before the time to service all subscribers was a compound of the number of subscribers AND topics. Now it is only the number of subscribers.

Additionally, whilst the dispatcher is servicing the clients, it is effectively "deaf" to incoming connections which also gets worse as you increase topics and subscribers. Having the dispatcher only handle connections means that this "deaf" time is not dependent on the number of subs/pubs. You can use a queue as, I have, and build a list of connection request but you then have to consider the trade off between handling data or handling a connection request. With significant network lag you can get halts in the data whilst the dispatcher is creating a connection. In my example, the Dispatcher only handles connections (its very responsive to connection requests). Therefore the dispatcher can take as long as is necessary without interrupting the data streams.

[Eugen Graf]

I agree to you.

A little issue I found. If I let the IP-Adress blank, I get errors on openning connections. So I have to write "localhost" to be connected.
May be it would be better for the example application to set "localhost" as default value?

[ShaunR]

I agree to you.

A little issue I found. If I let the IP-Adress blank, I get errors on openning connections. So I have to write "localhost" to be connected.
May be it would be better for the example application to set "localhost" as default value?

Interesting.

What version of LV and OS are you using?

.It was tested on 3 machines using LV 32 & 64 on win7 and vista (I don't have XP or a linux licence). And everything seemed fine.

But as it HAS happened, I'll include your suggestion with the next release. Thanks for persevering

[Eugen Graf]

I use LV 2009 under Win7.

[ShaunR]

I use LV 2009 under Win7.

Thanks I'll look into it.

I noticed I didn't address your other observation. I expect you read the note in the diagram of the FG. I too am not happy with it (apart from just being a mess ).
Mainly because It currently is not only used for storage, but also to relay disconnects and close connections. This needs to change so as to separate the roles (its a legacy from a previous incarnation).

I have to come back to it for the addition of other features and I'm still deciding on the best approach at the moment.

[Mark Balla]

Certified 9-14-2010 \
Placed in Remote Control, Monitoring and the Internet