Multiple queues and Reentrant VIs.

[Kas]

Hi

I'm currently doing something that involves multiple TCP/IP connections and multiple downloads at the same time. I achieve this through reentrant VI's and queues. Effectively, everytime a reentrant VI is opened, I also create a queue with the clone name of that VI. This way, I can than enqueue all the segments and parts that need to be downloaded equally among the available reentrant VI's.

My problem is this. I am downloading multiple files at a time, where each file is made of multiple segments or parts to be downloaded. Some reentrant VIs are downloading parts from file (lets say) "B" while other reentrat VIs are still downloading segments or parts from file "A". How do I know when File "A" is finished so then I can put it through post processing mechanism.

This will probably come down to proper design architecture for Reentrant VI and queues, but so far I can't see a proper way of implementing this.

Attached is the code I've done so far.

Thanks

Kas

[ShaunR]

I am downloading multiple files at a time, where each file is made of multiple segments or parts to be downloaded.

That's the key, right there. You're only lacking encapsulation depth at the moment. The thing to remember about queues is that they are a "Many-To-One" paradigm. So you have Many segments to One file and Many Files to One Batch.

I would tackle it this way.........

Create a "Download File" VI (launched re-entrantly) which you pass all the segments (a list?) that are required for that file to. It, in turn, then launches the individual segment downloads (again re-entrantly) by iterating through the list. The last thing a segment download VI does just before it exits is to post the segment to the Download File vis queue (Which will be named something like Download File.vi:3 i.e the clone name). The "download File" vi receives all the segments (for that one file) and, when it has all of them, reports back to your main app. The main app just sits and waits for completed files to come in.

In terms of queues, you only really need the return queues. So the Download File vi creates a queue to receive the segments (using it's clone name) and also has a queue named, say, "~download_files~" that is used to send back to the main app. All the download VI clones share the latter queue. The Download Files VI passes it's clone named queue to each of the segment queues (or you could get fancy and calculate it from the caller) so that they can use it to report back the individual segments to it's parent (the Download VI clone that launched them).

[Kas]

Currently I store each segment to the HDD as it comes. This way I avoid memory issues when getting a large file. Each segment consists of few hundred KB but each file can be few hundred MB large, where a batch of files can be even bigger. So, everytime I download a segment I store it in HDD (where sometimes 1000s segments are needed for a complete file), which is why I don't wait for the file to be completed before storing it. After a file is completed, I then can parse the content through post processing.

If I understand your idea correctly, I'm not sure if this would solve the problem of knowing when a particular file is finished without holding all the segments in memory. Currently, each segment is sent back to the main VI with the file path that the segment should be stored to. This way, if a segment belonging to a different file comes in, then a new file is created based on the path name provided.

Sorry if I misunderstood your explanation, but if this is the case, would you be able to provide these two VIs? of course no coding required, I just want to see how you would re-arrange the queueing mechanism that you mentioned between these two VI's.

Thanks

Kas

[asbo]

If I understand your idea correctly, I'm not sure if this would solve the problem of knowing when a particular file is finished without holding all the segments in memory. Currently, each segment is sent back to the main VI with the file path that the segment should be stored to. This way, if a segment belonging to a different file comes in, then a new file is created based on the path name provided.

Whether or not you store to disk is irrelevant as to whether you can track if a file is done. The Download File VI can still maintain a list of all the segments it expects and all the segments which have reported done. The Download File VI doesn't need to hang on to any of the actual data (let the segment downloaders write the file themselves, if you like). When the segment reports done, remove that segment from the working list of remaining segments.

[ShaunR]

Whether or not you store to disk is irrelevant as to whether you can track if a file is done. The Download File VI can still maintain a list of all the segments it expects and all the segments which have reported done. The Download File VI doesn't need to hang on to any of the actual data (let the segment downloaders write the file themselves, if you like). When the segment reports done, remove that segment from the working list of remaining segments.

Indeed. In fact. I would let the individual segment downloaders save the pieces, then probably get the download VI to sew them all together in the right order before sending the file name back to the main app.

Currently I store each segment to the HDD as it comes. This way I avoid memory issues when getting a large file. Each segment consists of few hundred KB but each file can be few hundred MB large, where a batch of files can be even bigger. So, everytime I <snip>

Sorry if I misunderstood your explanation, but if this is the case, would you be able to provide these two VIs? of course no coding required, I just want to see how you would re-arrange the queueing mechanism that you mentioned between these two VI's.

Thanks

Kas

As Asbo says. You don't need to send the actual data back. The segment downloaders only need to send the filename of the segment they downloaded and saved to the Download VI.

I think this demonstrates it for you (quick and dirty though it is). Of course with these things there is little error checking and no limitation to the number of spawned sub-processes. But I think you'll get the idea.Some of your VIs were missing so I couldn't use those. But I made it download some URLs using the HTTP VIs instead. Exactly the same idea except instead of passing the HTML back, you will pass a filename.

Edited November 6, 2012 by ShaunR

[Kas]

Aha. Thanks Shaun and asbo, now I get it. Initially, I thought of creating one reentrant VI for 1 segment. But since there can be 1000s of segments, I thought it might be best to have a set amount of reentrant VI's (depending on how many TCP conections I have, and then distribute the segments to be downloaded equally amongst the created reentrant VIs through queues. Since I'm allready sending each segment to the main vi for storing, then I guess I can use the same queue for sending a pointer "i.e. segment name" back to the main VI for monitoring purposes, instead of creating a new Download VI just for this. But now I understand the point that you guys were making.

Thanks again.

kas

A question that might have allready been answered. Do I have to make all the SubVI's within a reentrant VI's as "reentrant" or not. I tried it both ways and I didn't really see a performance difference.

Edited November 6, 2012 by Kas

[asbo]

A question that might have allready been answered. Do I have to make all the SubVI's within a reentrant VI's as "reentrant" or not. I tried it both ways and I didn't really see a performance difference.

If you don't, the reentrant VIs will block on whatever VIs are not reentrant. Instinct says this will only become a problem when you forget that you did it that way.

[hooovahh]

If you don't, the reentrant VIs will block on whatever VIs are not reentrant. Instinct says this will only become a problem when you forget that you did it that way.

Yes, yes, and more yes. Some VIs shouldn't be reentrant (things like functional globals and uninitialized shift registers), but any other subVI in a reentrant VI should be made reentrant. I found an NI article that talks about it more a while ago but can't find it right now.

I've been burned more than once, where I had a reentrant VI spin off, but one of the subVIs were not reentrant. Under almost all circumstances the program would run like normal, but once in a while there would be a dead lock, where one VI was running this subVI that was waiting on another VI, which blocked all the other reentrant VIs from running.

Attached is a VI if you're interested, which can take a folder of VIs and turn them all into reentrant.

Save VIs as Reentrant.vi

[Kas]

Attached is a VI if you're interested, which can take a folder of VIs and turn them all into reentrant.

Now why did I not think of that. So simple, and yet so useful. Cheers

kas