Application Design & Framework

[Kas]

Hello

 

I'm currently writing a personal application that deals with NNTP server protocol (i.e. a command based TCP communication) related to binary Usenet newsgroups. This application will be used for general data backup on Usenet.

 

While I have nearly finished the application itself (and it works), I am drastically falling into various problems, mainly parallel execution of various SubVI's and handling various errors within them. 

 

Nearly all of these problems seem to come from a lack of a solid framework for the whole application itself. 

The amount of commands that I'm dealing with are very few, (around 5) so building a basic version of this application is straight forward. However, my application follows the functionality similar to "Internet Download Manager" or various download accelerators that are found in internet, and this is where majority of the complexity is situated. 

And so, I'm hoping if I can get some help/pointers in 2 categories.

 

1. I keep having a TCP Error 1 problem. So, a user may select to have up to 30 TCP connections simultaneously to the same server and port (I have tested this and it is supported). However, every once in a while, a TCP connection is timed out (this is normal). This connection is than closed (By sending a "QUIT" command) and opening a new one in its stead. So far all this works. The problem comes when trying to use this same connection for operations again. Every time I use it, it throws a error 1 out. and I cant seem to get rid of it. 

 

2. This may be a more difficult problem to deal with since its more of a framework problem. I really want to use a proper programming structure for this application, but I've been using state machines, QMH based structures for a long time now, and I can't get out of this habit. The way this app is designed is full of holes, and I would like to make a proper structure for it. Unfortunately, I'm not into OOP programming yet so I went with normal Labview programming structure. This may also solve the problem 1 above.  

 

Attached is a ZIP file that I've included everything so far. The program itself is not yet finished, but it works with basic uploading operations. 

Everything is down-converted to LabView version 2009, but  if there are any VI's missing then please let me know and I'll provide them. 

 

Again, any solutions, thoughts are most appreciated.

[ShaunR]

I would suggest modularisation.

 

Create a VI that connects and does everything you want it to do for one connection, then just launch 30 clones (or however many) of them on demand asynchronously. You only need to send status data and the final file data to the main UI.

 

(Some VIs are missing btw Pars Que States, Add State To Que etc)

 

Take a look at the quick and dirty example in this post. Download.zip

[Daklu]

I agree with Shaun.  I'll also suggest putting your code in a project.  It helps keep your code organized, and you have to do it if you ever want to build an executable and/or installer.

[Kas]

Hi guys. 

This actually follows very closely to what Shaun suggested. The only difference is that it uses an intermediary VI i.e. in this case its the Upload Main VI. This upload main VI throws asynchronously called VI's for each connection. In this example maybe 30 asynchronously called VI's where they individually Upload a piece/chunk of data to the server and when finished they are closed and the TCP connection is passed back to the Upload Main VI for reuse again. 

This is shown in the "TCP: Upload" state on the "Upload Main - GUI". 

On that state I call "Upload Article" asynchronously where 1 call is 1 TCP connection. 

 

Also, Attached I cleaned up the files a bit and placed them into a project as Daklu suggested. All the relevant VI's should now be there. 

 

Thanks

 

Kas

 

P.S. I'm using some of Shaun's VI's for calculating the percentage and Transmission speed  , I found them from one of the examples in the forum (I think it was something to do with Bluetooth connections)

 

 

Oups, for some reason this thread is not allowing me to Upload a file that bigger than 733.62K. My ZIP file is 1,184 KB.

 

 

So please use the link below instead. 

 

https://mega.co.nz/#!NMdUXbLI!IxWKyhj8zrq60S2xOL4Mc3VTPC12qCsL3F00nxbrN4Y

Edited August 17, 2013 by Kas

[ShaunR]

There are still a load of VI's missing (like the ones that actually upload and download. I would suggest you put all the files in the project (not just the top level one) then make a build specification for "Source Distribution" (don't forget to manually add the dynamically loaded VIs).

 

However.

You are opening all the TCP connections (the TCP connection pool) in the top level GUI and then passing the refs to both upload and download things that are missing (so I don't know whats in them). Unless the upload and download managers (the things) negotiate between each other, I can guess,that when you set the No to 30 they both think they have those 30 connections

 

I would let each manager open it's own pool of connections or use a LV2 (not re-entrant) self initialising global that creates them on the first call only and place that in both (i.e not in the top level and not passing them).

 

Not knowing what is down further, so you may already be doing it or it may not be pragmatic............

Generally with this sort of thing, I would let each download/upload subVI connect, login and download/upload and not open all connections at once, rather, limit how many spawned processes there are with the manager which would ensure that only X are running and queue additional requests until one or more has finished. This suits a messaging architecture, however, since the manager is always running and you just send it the data (as a message or series of messages) that you wish to action (upload or download). This approach completely modularises and separates the UI from the manager and the manager from the downoader/uploader code so they all can be run in isolation as stand-alone modules whilst developing and debugging

[Kas]

Ayayay, when things happen they all seem to happen at the same time. I get the attached error when trying the built specification. I think this may have to do with clashes between various VIPM packages that I've installed. I have to go though them and sort this out later. 

 

As for the files itself, I'm now providing the full folder in my current LabView version i.e. 2012. 

 

As usual the link is below since the attachment is bigger than the forum allows. 

 

https://mega.co.nz/#!oA8SnRrJ!Uop2vwECVGQk_pCsovLhFnb9kCwtEwc0VMvD6ZVVm8E

 

 

As for what you've just mentioned Shaun, you are correct. The main VI first initializes the whole set of connections and then provides the exact same ones to all the jobs waiting. However, the main VI organizes and makes sure that only one of the jobs (i.e. either downloading or Uploading) can use the connection at a time. The reason I provide them all with the same connection is that when a job or process is paused, the next job in the main queue waiting starts to work straight away, since the previous job is now paused and no longer using these connections, the next job can now start using them until finished or the user pauses that job as well, and so on. 

 

 

 

Kas

[ShaunR]

Ayayay, when things happen they all seem to happen at the same time. I get the attached error when trying the built specification. I think this may have to do with clashes between various VIPM packages that I've installed. I have to go though them and sort this out later. 

 

As for the files itself, I'm now providing the full folder in my current LabView version i.e. 2012. 

 

As usual the link is below since the attachment is bigger than the forum allows. 

 

https://mega.co.nz/#!oA8SnRrJ!Uop2vwECVGQk_pCsovLhFnb9kCwtEwc0VMvD6ZVVm8E

 

 

As for what you've just mentioned Shaun, you are correct. The main VI first initializes the whole set of connections and then provides the exact same ones to all the jobs waiting. However, the main VI organizes and makes sure that only one of the jobs (i.e. either downloading or Uploading) can use the connection at a time. The reason I provide them all with the same connection is that when a job or process is paused, the next job in the main queue waiting starts to work straight away, since the previous job is now paused and no longer using these connections, the next job can now start using them until finished or the user pauses that job as well, and so on. 

 

 

 

Kas

 

Everything seems to be there. Sweet. I'll take a more detailed look later. QSMs are notoriously difficult to follow without state diagrams and there are a few in there (Did I mention how much I hate the JKI statemachine )

 

So. If the main VI organises and "makes sure that only one of the jobs (i.e. either downloading or Uploading) can use the connection at a time". This is what I mean about modularisation. The UI just handles the UI stuff only and a separate module has responsibility for managing the connections/jobs.

 

As a first pass (very quick) glance, you seem to be sending a lot of messages and data to different loops which don't really do much apart from pass them to other loops or VIs (please correct me if I'm wrong). This is a code smell to me that you may be better off with events rather than queues as different parts of the program can register for the same messages and have their own message filters if necessary. This approach also allows better modularity as you can create self contained message handlers that deal with very specific usage aspects of the message data. A good example of this would be the status where you want different parts of the program to obtain info about the individual downloads from the connection manager and update log files, and the UI . So the sender broadcasts it's data for anyone interested to hear, rather than pushing the data to a number of specific queues.

Edited August 18, 2013 by ShaunR

[Kas]

(Did I mention how much I hate the JKI statemachine )

Haha. I do, I remember from this thread, for both you and Daklu.

http://lavag.org/topic/16154-qsm-producerconsumer-template/#entry98307

 

So. If the main VI organises and "makes sure that only one of the jobs (i.e. either downloading or Uploading) can use the connection at a time". This is what I mean about modularisation. 

Correct, currently the Main VI does this through the "Check Availabiltiy" state. However, I'm worried the way that I am doing this may be abit of an overkill. i.e. I'm thinking that there should be a much easier way of doing the same operation that would result is less confusion. Usually, the simplest method is the best method, and I'm worried I haven't approached it that way. 0

So far, the main VI keeps track of all the jobs (i.e. Uploads and Downloads) through their properties of queues where they were initially generated from. Their state such as Pause, Cancel, Exit, Ready, etc. is kept track by the main VI and when the right conditions are met, the token is then given by the main VI to the right job. When that Job is paused, the Main VI then goes through the list again and checks the next Job that meets the right criteria, and so on. 

 

As a first pass (very quick) glance, you seem to be sending a lot of messages and data to different loops which don't really do much apart from pass them to other loops or VIs (please correct me if I'm wrong).

Well, the Upload VI, has 3 such loops. The Progress loop, the Log loop and Pre-processing loop. The progress loop updates the progress of the Upload VI (i.e. the progress bar, speed etc.) where the same values are then also sent back to the main Update loop of the Main VI. The same happens with the Log loop, where the log indicator of the Upload VI is Updated and the same log is also sent back to the Main Log on the Main VI. I kept this loops seperate because depending on the amount of Logs generated or the Updates, I didn't wand to occupy the Main loop that deals with Uploading articles itself. 

 

I will also try and buy a block account from newsreaders so you can also run the program and see how it works. I will post the information as soon as I'm finished. 

 

In the mean time, tweaknews https://www.tweaknews.eu/?page=order&product=1   seem to be providing free accounts for 10 days, but since I've used it in the past I can't create another account with them. 

 

Thanks 

 

Kas

[ShaunR]

Correct, currently the Main VI does this through the "Check Availabiltiy" state. However, I'm worried the way that I am doing this may be abit of an overkill. i.e. I'm thinking that there should be a much easier way of doing the same operation that would result is less confusion. Usually, the simplest method is the best method, and I'm worried I haven't approached it that way. 0

So far, the main VI keeps track of all the jobs (i.e. Uploads and Downloads) through their properties of queues where they were initially generated from. Their state such as Pause, Cancel, Exit, Ready, etc. is kept track by the main VI and when the right conditions are met, the token is then given by the main VI to the right job. When that Job is paused, the Main VI then goes through the list again and checks the next Job that meets the right criteria, and so on. 

 

Indeed. I suspect that a separate "Job Manager" would simplify things immensely which is why we are talking about modularisation.

 

 

Well, the Upload VI, has 3 such loops. The Progress loop, the Log loop and Pre-processing loop. The progress loop updates the progress of the Upload VI (i.e. the progress bar, speed etc.) where the same values are then also sent back to the main Update loop of the Main VI. The same happens with the Log loop, where the log indicator of the Upload VI is Updated and the same log is also sent back to the Main Log on the Main VI. I kept this loops seperate because depending on the amount of Logs generated or the Updates, I didn't wand to occupy the Main loop that deals with Uploading articles itself. 

Again. If you have a "Log" module, then it can do its thing whilst the UI gets on with just updating the UI. On the surface, the UI seems to be trying to do too much (it's an aesthetically pleasing one however - much better than I could mange) I am still maintaining that delegating much of the current UI functionality into separate modules will be beneficial in terms of maintenance, readability and robustness. I'll see what I can come up with.

 

 

I will also try and buy a block account from newsreaders so you can also run the program and see how it works. I will post the information as soon as I'm finished. 

 

In the mean time, tweaknews https://www.tweaknews.eu/?page=order&product=1   seem to be providing free accounts for 10 days, but since I've used it in the past I can't create another account with them. 

That'd be great. It's an interesting project and I'm keen to play so if you can get that done, so much the better (PM me log-in details if necessary). I'll take a detailed look at the code when you've set that up.

What will be the end product of this software? Commercial? Open Source? Lavag? Do you have a deadline?

[Kas]

Hi Shaun. I just sent you the account details.

 

And no, there is no deadline for this project. After using "Usenet" for sometime now, I saw something that I needed and I started this on my own.

Edited August 19, 2013 by Kas

[Kas]

I am still maintaining that delegating much of the current UI functionality into separate modules will be beneficial in terms of maintenance, readability and robustness.

I definetly agree with you in this. Initially, I was planning on having the "Upload VI" as nothing more than a very simple VI that does nothing more than reading the updates, logs etc. But as the project went on and new functionality that I wanted to add to the main VI, then the Main VI became too congested and I started making the Upload VI more complicated until it became what you see now.

 

Having separate modules for this, and having a communication link for data and messages for these modules would certainly be the way forward, but unfortunately, I'm used to statemachine type of structures and not much in module based work.

I started thinking on how to handle different errors, i.e. have each module sort its own errors out or delegate them to a central error handling module, where error logging then comes into play.  While all that is going on than I remembered various race issues that can creep up (specially if I continue using my current framework), and somewhere around here my head started hurting   . 

 

So I thought coming here and get some help along the way  . 

[ShaunR]

I definetly agree with you in this. Initially, I was planning on having the "Upload VI" as nothing more than a very simple VI that does nothing more than reading the updates, logs etc. But as the project went on and new functionality that I wanted to add to the main VI, then the Main VI became too congested and I started making the Upload VI more complicated until it became what you see now.

 

Having separate modules for this, and having a communication link for data and messages for these modules would certainly be the way forward, but unfortunately, I'm used to statemachine type of structures and not much in module based work.

I started thinking on how to handle different errors, i.e. have each module sort its own errors out or delegate them to a central error handling module, where error logging then comes into play.  While all that is going on than I remembered various race issues that can creep up (specially if I continue using my current framework), and somewhere around here my head started hurting   . 

 

So I thought coming here and get some help along the way  . 

You can still keep your state machines. Just contain them in modules with defined responsibilities and interfaces so that they are encapsulated..

[Kas]

Hi Shaun.

 

Would it be of any help if  start doing a flow chart of the execution functions of the program? I haven't done it yet, but I just wanted to check if it would be of any help. 

 

Kas

[Jordan Kuehn]

Hi Shaun.

 

Would it be of any help if  start doing a flow chart of the execution functions of the program? I haven't done it yet, but I just wanted to check if it would be of any help. 

 

Kas

 

In general it is almost always beneficial to spend some time in advance modeling what you want it to do before actually touching any code.  The cases where it isn't are typically trivial.

[Kas]

In general it is almost always beneficial to spend some time in advance modeling what you want it to do before actually touching any code.  The cases where it isn't are typically trivial.

You are right, but since my University work (Electronic and Physics), any programming work I've needed was done by trial and error. I concentrated fixing the problem without thinking of how. So now my brain doesn't work until I start coding and fix whatever needs fixing along the way. In the end, Panic time arrives and I'm off to LAVA   .

I've been using LabView for a long time, but its always been on-and-off. Its only recently that I'm more invested on it and I'm finding it rather difficult shaving of the bad habits of programming I've accumulated over the years. It was very easy to learn LabView the wrong way   .

Edited August 20, 2013 by Kas