Accessing Queues

[AlexA]

What happens if I have 3 while loops trying to enqueue elements to the same queue? Is the queue "locked" while in the process of being enqueued? If this is the case, do the while loops wait until the queue has unlocked and then grab it for their own enqueue operation?

Regards

Alex

Whoops, forgot to include picture of code in first post, here it is.

[ned]

What happens if I have 3 while loops trying to enqueue elements to the same queue? Is the queue "locked" while in the process of being enqueued? If this is the case, do the while loops wait until the queue has unlocked and then grab it for their own enqueue operation?

Your image shows three loops trying to dequeue, not enqueue. This will not work, since each element in the queue can only be dequeued once, after which it's no longer in the queue. Which loop will dequeue a particular element is random. You should consider using a notifier, if you don't need to capture every element, or use three separate queues.

[Ton Plomp]

Your code shows one 'enqueue' and 3 'dequeue'.

When an image gets enqueued one of the three loops will get the image and starts processing, if a second image gets enqueued during process one of the other loops will get it.

Then only problem is (I think) that an imaq image is basically a reference and so there might be actually just one image. So perhaps it's best to convert the image into the 2D array before enqueuing it.

Ton

[AlexA]

I know, I haven't actually written the enqueue part in, but for the sake of argument, assume that the results of the calculation in each loop (the average frequency values) are enqueued into a single queue, what will happen?

[Mark Smith]

I know, I haven't actually written the enqueue part in, but for the sake of argument, assume that the results of the calculation in each loop (the average frequency values) are enqueued into a single queue, what will happen?

I don't claim to have any knowledge of how the LabVIEW queue is coded under the hood, but my experiance tells me the enqueue operation is atomic. That is, if you have two operations writing to the same queue, one write will complete before the other begins. For example, if you're writing arrays to the queue, you don't have to worry that some queue entry will contain a mixture of array data from two attempted writes. You will not, unless you take extra steps (like including a source tag/time stamp on the data), have any idea in which order the elements were enqueued.

Mark

[Ton Plomp]

An enqueue node has a timeout input, and if somehow you manage to run out of this timeout while enqueuing, the enqueueing will not take place (this normally happens with length limited queues).

However I would check your code to work since I don't think it will work with IMAQ references.

Ton

[shoneill]

What about this type of approach assuming you either have all your data ready at once OR know how many data packets you will have.

Parallel for-loops are new in LV 2009.

Shane.

[AlexA]

Hi Shane

Sorry but I don't quite understand your architecture here? I'm having a bit of a dull moment at work so I'm not thinking at my best. Is what you propose supposed to be for the intermediary step of processing the data and enqueing it to an output?

Regards

Alex

[shoneill]

Hi Shane

Sorry but I don't quite understand your architecture here? I'm having a bit of a dull moment at work so I'm not thinking at my best. Is what you propose supposed to be for the intermediary step of processing the data and enqueing it to an output?

Regards

Alex

Your picture included a single Queue which was being filled and emptied by three parallel loops. As said before, each entry int he queue will go to ONE of the THREE loops, but you don't know which one. I'm assuming that you will re-gather the results of these loops somewhere later in the program.

My code basically does the same thing, but uses a parallel for-loop to do the same thing. It only deals with the processing part of your problem, not the filling of the first queue.

The For-Loop can be set (programatically) to a certain number of parallel tasks, each of which waits for an entry int he Queue. Each entry will be read by one and only one process, this is due to the atomic nature of the reads and writes to a queue.

The clue is that the parallel for-loop actually spawns "P" processes in the background which all perform the same task. Each of the "P" processes can dequeue an image and work on it independent of the others. Due to the atomic nature of Queue reads and writes, you already know that any entry to the first queue will go to one and only one recipient. This way you can have "P" process working concurrently ont he problem, processing the images as fast as your multi-code PC will allow. Tis continues until all of the expected images have been rpocessed "N".

In order to be able to collate the results after processing, you need some kind of output scheme. I have used a queue. You could theoretically place the "Enqueue" INSIDE the for-loop but this requires the user to re-order manually. By placing it outside the For-Loop, the compiler ensures the ordering for you and you just need to enqueue them. Since we're dealing with IMAQ references (4 Bytes only)e memory overhead is small.

Shane.

[shoneill]

Your picture included a single Queue which was being filled and emptied by three parallel loops. As said before, each entry int he queue will go to ONE of the THREE loops, but you don't know which one. I'm assuming that you will re-gather the results of these loops somewhere later in the program.

My code basically does the same thing, but uses a parallel for-loop to do the same thing. It only deals with the processing part of your problem, not the filling of the first queue.

The For-Loop can be set (programatically) to a certain number of parallel tasks, each of which waits for an entry int he Queue. Each entry will be read by one and only one process, this is due to the atomic nature of the reads and writes to a queue.

The clue is that the parallel for-loop actually spawns "P" processes in the background which all perform the same task. Each of the "P" processes can dequeue an image and work on it independent of the others. Due to the atomic nature of Queue reads and writes, you already know that any entry to the first queue will go to one and only one recipient. This way you can have "P" process working concurrently ont he problem, processing the images as fast as your multi-code PC will allow. Tis continues until all of the expected images have been rpocessed "N".

In order to be able to collate the results after processing, you need some kind of output scheme. I have used a queue. You could theoretically place the "Enqueue" INSIDE the for-loop but this requires the user to re-order manually. By placing it outside the For-Loop, the compiler ensures the ordering for you and you just need to enqueue them. Since we're dealing with IMAQ references (4 Bytes only)e memory overhead is small.

Shane.

Here's an example to play with. Try changing the number of cores. Ideally the graph ont he right should show two identical lines shifted by 1 (and the array should be all 1.000) but contrary to what I expected, the array order is NOT being retained.

For some reason, the order seems to get mixed up exiting the parallel for loop.... I thought this was supposed to retain the array order (Not the processing order) of the data.

I think the problem is the asynchronous access via the Queue. Seems like you need to definitely incorporate some ordering information into your data so that you can reconstruct the array order afterwards.

Shane.

Parallel for loop.vi

[shoneill]

Here's an example to play with. Try changing the number of cores. Ideally the graph ont he right should show two identical lines shifted by 1 (and the array should be all 1.000) but contrary to what I expected, the array order is NOT being retained.

For some reason, the order seems to get mixed up exiting the parallel for loop.... I thought this was supposed to retain the array order (Not the processing order) of the data.

I think the problem is the asynchronous access via the Queue. Seems like you need to definitely incorporate some ordering information into your data so that you can reconstruct the array order afterwards.

Shane.

I think I understand why my example doesn't retain the array order. Seems logical once you've tortured your brain with it.

Here's an example where it might work better.

I'm creating an array of single-element queues instead of one large queue to pass the data to the parallel for loop. That way the array order is maintained. The OTHER queue (single large queue) is there only to be able to monitor the number of elements yet to be processed. It has no influence on the functionality of the VI, simply for visualisation of the enqueuing / dequeuing process (as is the bottom While-loop in its entirety).

Shane.

Parallel for loop.vi

[AlexA]

Hey Shane

Thanks heaps for your help, that's definitely an interesting architecture. I've split things up a little differently than in my original code but I'm still using while loops. Your examples were much appreciated though.

Regards

Alex