Jump to content
ShaunR

[CR] Dispatcher

Recommended Posts

Hi!

Nice made, did you just seen this one http://lavag.org/top...ish-subscriber/ ?

Seems to be almost the same thing :D

Regards, Eugen

No I didn't :thumbup1: Great minds think alike eh? :yes:

You talk about it being in the repository, but I'm unable to find it. Maybe it was lost along with the "The Great Crash" since the posts seem from quite a while ago. I wouldn't mind taking a look. Do you have a link?

The initial idea was born out of being fed up with writing similar things every time I worked with a new company. Although with this incarnation I wanted a more generic nature by removing a lot of the infrastructure dependency (which probably isn't that visible.....yet;)) and have the ability to cluster rather than rely on a central server (i.e. filtering). That way I can transparently use a distributed architecture. It is really just a multi client/server tracker. ( I also wanted a viable light-weight alternative to DCS. :lol: )

I see you also talk about "Topics" which suggests (to me) a single server with segmented data filtered for the particular subscribers (classic pub/sub). I think mine is more content-based where each "Topic" is in fact a dedicated server and after registration or subscription, the Dispatcher has little to do with data.

Share this post


Link to post
Share on other sites

Is there any particular reason you didn't use LVOOP for the transport code? This is a natural candidate for a hierarchy of classes.

Share this post


Link to post
Share on other sites

Is there any particular reason you didn't use LVOOP for the transport code? This is a natural candidate for a hierarchy of classes.

You make it sound as though thats a good thing :D

Share this post


Link to post
Share on other sites

No I didn't :thumbup1: Great minds think alike eh? :yes:

You talk about it being in the repository, but I'm unable to find it. Maybe it was lost along with the "The Great Crash" since the posts seem from quite a while ago. I wouldn't mind taking a look. Do you have a link?

Yes, it was lost after updating LAVAs software (I think). Here you can download my version:

http://labviewportal...hp?f=19&t=9#p19

I see you also talk about "Topics" which suggests (to me) a single server with segmented data filtered for the particular subscribers (classic pub/sub). I think mine is more content-based where each "Topic" is in fact a dedicated server and after registration or subscription, the Dispatcher has little to do with data.

I think it's the difference to your dispatcher. I send all the data to the server(dispatcher) and the dispatcher sends them to subscribers. It makes some advantages like debugging on one place, but disadvantages like more traffic on the server side.

But, sorry, I don't see where you send your data from one publisher to many subscribers. TCP/IP is one to one connection, so you have to send your data twice if you have two subscribers. I can't see any for-loop in TCIP Write.vi.

EDIT: my the web chat project (based on Pub/Sub Pattern) may be interesting to you http://lavag.org/top...uss-eugen-graf/

Edited by Eugen Graf

Share this post


Link to post
Share on other sites

Yes, it was lost after updating LAVAs software (I think). Here you can download my version:

http://labviewportal...hp?f=19&t=9#p19

Sweet. I'll take a look!

But, sorry, I don't see where you send your data from one publisher to many subscribers. TCP/IP is one to one connection, so you have to send your data twice if you have two subscribers. I can't see any for-loop in TCIP Write.vi.

Indeed. That's where the "Disp Service Handler.vi" comes in.

This dispatcher uses a similar method to a web server where each incoming connection is negotiated to a different port and a process is spawned to handle that client. However. This spawning only happens for the Publishers. If you look in the Registration part of the Dispatcher you will see I dynamically launch the service handler (whos front panel is hidden). The publisher writes to the service handler, not the clients and the handler deals out the data to the clients. There is a service handler for each publisher and when a subscriber wishes to connect, the Dispatcher sends a new subscriber list to the handler for that service.

Edited by ShaunR

Share this post


Link to post
Share on other sites

Ok, now I understand it. So you have a sender task for each publisher (service) on the dispatcher side. Why not one for all? My dispatcher does all, registration, unregistration an sending in one loop, but each service has own receiver loop. I think this is the biggest difference between our realisations of this pattern. The second less difference is you use functional global variable for the client list, but I don't.

For all, I think it's very interesting pattern, which can be used for dynamic applications, where many publishers and subscribers can communicate to each other.

Thank you :worshippy:

Share this post


Link to post
Share on other sites

Ok, now I understand it. So you have a sender task for each publisher (service) on the dispatcher side. Why not one for all? My dispatcher does all, registration, unregistration an sending in one loop, but each service has own receiver loop. I think this is the biggest difference between our realisations of this pattern. The second less difference is you use functional global variable for the client list, but I don't.

For all, I think it's very interesting pattern, which can be used for dynamic applications, where many publishers and subscribers can communicate to each other.

Thank you :worshippy:

Mainly load balancing and performance. The performance is dictated by the computers resources rather than the dispatcher. With the filtered method you have a congestion point that affects ALL "Topics" and gets worse by a factor of subscribers x topics as you add more topics.

Lets say (for example) you are sending 100MB of data on 1 topic which takes (say 10 ms) to transmit to 1 subscriber (we'll ignore filtering overhead). The Dispatcher is able to service all subscribers to that topic every 10ms. If we now add a subscriber to that topic. The time to service all subscribers is now 20 ms. 3 subscribers...30 and so-on. Now we add another topic that also takes 10ms to tx 100MB of data to 3 more subscribers (keep the maths easy). The time now required to service all 6 subscribers on both topics is now 60 ms. 3 Topics, 90ms.......

However. with the service handler, each service handler is only handling 3 subscribers so in our theoretical example, each handler is servicing all 3 of the subscribers every 30 ms still. Increasing the number of topics doesn't change the time to service all clients. So whereas before the time to service all subscribers was a compound of the number of subscribers AND topics. Now it is only the number of subscribers.

Additionally, whilst the dispatcher is servicing the clients, it is effectively "deaf" to incoming connections which also gets worse as you increase topics and subscribers. Having the dispatcher only handle connections means that this "deaf" time is not dependent on the number of subs/pubs. You can use a queue as, I have, and build a list of connection request but you then have to consider the trade off between handling data or handling a connection request. With significant network lag you can get halts in the data whilst the dispatcher is creating a connection. In my example, the Dispatcher only handles connections (its very responsive to connection requests). Therefore the dispatcher can take as long as is necessary without interrupting the data streams.

  • Like 2

Share this post


Link to post
Share on other sites

I agree to you.

A little issue I found. If I let the IP-Adress blank, I get errors on openning connections. So I have to write "localhost" to be connected.

May be it would be better for the example application to set "localhost" as default value?

Share this post


Link to post
Share on other sites

I agree to you.

A little issue I found. If I let the IP-Adress blank, I get errors on openning connections. So I have to write "localhost" to be connected.

May be it would be better for the example application to set "localhost" as default value?

Interesting.

What version of LV and OS are you using?

.It was tested on 3 machines using LV 32 & 64 on win7 and vista (I don't have XP or a linux licence). And everything seemed fine.

But as it HAS happened, I'll include your suggestion with the next release. Thanks for persevering :yes:

Share this post


Link to post
Share on other sites

I use LV 2009 under Win7.

Thanks I'll look into it.

I noticed I didn't address your other observation. I expect you read the note in the diagram of the FG. I too am not happy with it (apart from just being a mess :lol: ).

Mainly because It currently is not only used for storage, but also to relay disconnects and close connections. This needs to change so as to separate the roles (its a legacy from a previous incarnation).

I have to come back to it for the addition of other features and I'm still deciding on the best approach at the moment.

Share this post


Link to post
Share on other sites

ShaunR, is this thing still alive? I'm curious about using it on a VxWorks target, since common (non-NI) libraries like 0MQ and nanomsg aren't available for that OS.

Share this post


Link to post
Share on other sites

ShaunR, is this thing still alive? I'm curious about using it on a VxWorks target, since common (non-NI) libraries like 0MQ and nanomsg aren't available for that OS.

 

If it's here (or on my site), it's alive. It just does everything I need (so no new versions) and has had no bug reports.

You won't be able to use compression (windows only) and it is untested on VxWorks because I didn't have that hardware platform back then but everything should be fine.

 

If you run into problems just shout.

Share this post


Link to post
Share on other sites

Hi, ShaunR, I would like to suggest you to check the Disp Query Get Service.vi. There is a wire mistake when TCPIP open.

And I think that you should add a case "GETSUBSCRIBE" in Dispatcher.vi. Because there is no handler for Dispatcher Query.vi(Get Subscribers). Also should add a vi to deal with it. Thanks.

Share this post


Link to post
Share on other sites

Hi, ShaunR, I would like to suggest you to check the Disp Query Get Service.vi. There is a wire mistake when TCPIP open.

And I think that you should add a case "GETSUBSCRIBE" in Dispatcher.vi. Because there is no handler for Dispatcher Query.vi(Get Subscribers). Also should add a vi to deal with it. Thanks.

OK. I'll take a look at the weekend.

Share this post


Link to post
Share on other sites

Any updates?

 

Ah Yes. Sorry. Kind of slipped off the radar what with Xmas and everything.

 

Yes, yes, yes.

 

I actually have a later version which updates Transport.lvlib shipped with the latest version (for Version 1.1). That gives dispatcher the compression (zlib) and compression/timestamp information like so.

 

post-15232-0-15391500-1419446564.png

 

The Get Subscribers was removed as it was not intended for Users (more for internal use - which it never was) and just leaks information that could be exploited.

 

There were a few more tweaks and bug fixes (including the one reported) so there will be an update. I'm just umming and arring about an installer (e.g. putting into a VIPM package) ATM. I've been burned badly by VIPM so am tempted to use my wizard instead. So once the grog has worn of and the hangovers have subsided; I'll upload (just after Xmas but before the new year).

Share this post


Link to post
Share on other sites

Short progress update......

 

Having a bit of a problem with installers but hopefully will be uploaded next week.

Share this post


Link to post
Share on other sites

Transport.lvlib, which this package depends on, has been updated. It uses a new installer which this one will also use so I will leave it a couple of days for people to check out and break it - then update Dispatcher.

Edited by ShaunR
  • Like 1

Share this post


Link to post
Share on other sites

Transport.lvlib, which this package depends on, has been updated. It uses a new installer which this one will also use so I will leave it a couple of days for people to check out and break it - then update Dispatcher.

So glad to hear so many good news. And thanks. :thumbup1:

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.