Sorting a table according to one element

[postformac]

I am having difficulty figuring out how to sort an array according to one of the columns, I'm sure there's a really easy way to do it but I can't think of a nice solution that doesn't use loads of memory!

I have a large array that contains data from a CAN bus. Each line has the message identifier and 8 bytes of data. All I want to do is to split the array up so I end up with separate arrays each with all of the data for one individual message identifier.

I have attached a quick VI that demonstrates the principle. This VI works fine and does what I am intending, but the actual files of CAN data are thousands of lines long and could contain upto around 30 different identifiers depending on how many nodes are active on the bus and what messages are present. That would mean that I would need to pass 30 arrays back around every iteration of my loop, some of which may contain thousands of lines of data depending on how active the ID is. It ends up being quite slow, I assume because I am sending so much data round and round my loop on each iteration.

I would prefer to just terminate all of the possible array lines at auto-indexed tunnels at the edge of my loop so that I'm not passing that data back and forth. I can't work out how to make it operate like this though as I still need some output to send to every tunnel on each iteration, not just the tunnel of the identifier that the current line being examined refers to. I could output some kind of constant to all the other tunnels but then I have to go back through and remove all of the unwanted constants from each array afterwards.

I'm sure there must be a better way to do this, could anyone suggest anything?

Thanks

Sorting array.vi

[Grampa_of_Oliva_n_Eden]

I am having difficulty figuring out how to sort an array according to one of the columns, I'm sure there's a really easy way to do it but I can't think of a nice solution that doesn't use loads of memory!

I have a large array that contains data from a CAN bus. Each line has the message identifier and 8 bytes of data. All I want to do is to split the array up so I end up with separate arrays each with all of the data for one individual message identifier.

I have attached a quick VI that demonstrates the principle. This VI works fine and does what I am intending, but the actual files of CAN data are thousands of lines long and could contain upto around 30 different identifiers depending on how many nodes are active on the bus and what messages are present. That would mean that I would need to pass 30 arrays back around every iteration of my loop, some of which may contain thousands of lines of data depending on how active the ID is. It ends up being quite slow, I assume because I am sending so much data round and round my loop on each iteration.

I would prefer to just terminate all of the possible array lines at auto-indexed tunnels at the edge of my loop so that I'm not passing that data back and forth. I can't work out how to make it operate like this though as I still need some output to send to every tunnel on each iteration, not just the tunnel of the identifier that the current line being examined refers to. I could output some kind of constant to all the other tunnels but then I have to go back through and remove all of the unwanted constants from each array afterwards.

I'm sure there must be a better way to do this, could anyone suggest anything?

Thanks

I believe OpneG has a ready made solution but if you want to "roll-your-own" you can take advatage of the order of a cluster to accomplish this task.

Create an array of clusters with each cluster representing one line. Set the cluster order such that the primary serach field is first then secondary seach field.

Pass that througha "sort 1-d array" , unpack the cluster, done.

Ben

[ned]

Can you change the format of your data? Strings aren't that efficient, and CAN packets are fixed-size. You could convert to a cluster of numeric values, where the first element is the ID. Then, one route would be to use that ID as an index into an array, where each element of the array contains a cluster that contains an array of all data found for that ID. Ideally you would pre-allocate each array, since one reason your example is slow is the use of build array in your loop.

What output do you actually need? Instead of moving your CAN data around, you could build arrays of indices into the original data, one per ID. This would give you a quick way to retrieve all the data for one ID without making a large number of copies. I've attached a quickly-written sample that demonstrates this, then uses the generated array of indices to extract the data into separate arrays for each node.

Oops, forgot the attachment - and the "Use Full Editor" button doesn't seem to be working, so I couldn't attach it to the previous post.

Sorting array-1.vi

[Grampa_of_Oliva_n_Eden]

Can you change the format of your data? Strings aren't that efficient, and CAN packets are fixed-size. You could convert to a cluster of numeric values, where the first element is the ID. Then, one route would be to use that ID as an index into an array, where each element of the array contains a cluster that contains an array of all data found for that ID. Ideally you would pre-allocate each array, since one reason your example is slow is the use of build array in your loop.

What output do you actually need? Instead of moving your CAN data around, you could build arrays of indices into the original data, one per ID. This would give you a quick way to retrieve all the data for one ID without making a large number of copies. I've attached a quickly-written sample that demonstrates this, then uses the generated array of indices to extract the data into separate arrays for each node.

Oops, forgot the attachment - and the "Use Full Editor" button doesn't seem to be working, so I couldn't attach it to the previous post.

The indexed sort is superior to my simplistic method described previously. If you do your own try what ned posted.

Ben

[postformac]

Thanks very much for the replies guys. I have had a look at the example you posted Ned and its got lots of stuff in it I have never seen before so it might take me a while to digest! It does sound like a good way to do it though so I will have a go at repeating what you have done.

Thanks

[postformac]

Ok, I think I understand what you have done. Basically you have created an array of clusters, with one cluster for each ID where each cluster contains a 1D array that specifies the row number(s) that the ID occurs on and a count of how many times that ID occurs in total. Then you use that to extract the information again at the end from the original array to create another array containing clusters of 2D arrays....

An array of clusters of arrays, why didn't I think of that ha ha.

Seriously though that works really well, I would never have thought of it myself so thanks for your help.

When you say in your note to create a lookup table if the IDs are not concurrent (which in reality they aren't) is there actually such a thing as a lookup table in labview or do you just mean to go through the array one row at a time and replace the IDs with pre-determined values manually (such as via a case statement)?

Thanks,

Mark

[postformac]

Oh no, I see what you mean. You mean to create my own lookup table just using an array or whatever, like this right?

Thanks

Sorting array-2.vi

[ned]

Oh no, I see what you mean. You mean to create my own lookup table just using an array or whatever, like this right?

That's one way to do it; another is simply to build an array of all your IDs and use Search 1D Array. Slightly slower but uses slightly less memory; if you're only handling about 30 IDs, I doubt you'll see any difference in speed. Here's a modified version that generates the lookup table on the fly, using Search 1D Array. I don't think the use a build array in a loop in this version will cause noticeable speed issues.

The array of clusters of arrays looks odd, but in LabVIEW you can't have an array of arrays - it would just be an array with an additional dimension, which forces all the sub-arrays to have matching dimensions. The way around it is to wrap each array in a cluster. If you're expecting roughly the same number of messages from each ID and you know in advance how many messages that is, you could efficiently use a 2D array instead.

Sorting array-3.vi

[postformac]

Ok I get it, I think your way is better as it means I don't have to hard code all of the IDs in advance, it will just populate as it goes and generate a new cluster for whatever messages are on the bus.

I have never used clusters before but they are looking pretty useful right about now, I will have to play with them some more...

Thanks very much for your help!