Displaying Large Data

[ensegre]

If it helps

persistence5.vi

[ShaunR]

Isn't the issue that the OP is referring to, that when large data amounts are sent to the graph (regardless of how they were acquired or organised) that the indicator is slow at rendering them? Yes. Ring buffers are great for recording data at high rates but you are still left with the rendering problem, are you not?

I think the OPs issue becomes a little clearer if you make your 500 plots 8192 data points in length instead of 128

[bbean]

1 hour ago, Norm Kirchner said:

Is that snippit working for anyone else? I'm not having any luck w/ it

the last snippet from ensegre is not working for me either.

33 minutes ago, ShaunR said:

Isn't the issue that the OP is referring to, that when large data amounts are sent to the graph (regardless of how they were acquired or organised) that the indicator is slow at rendering them? Yes. Ring buffers are great for recording data at high rates but you are still left with the rendering problem, are you not?

I think the OPs issue becomes a little clearer if you make your 500 plots 8192 data points in length instead of 128

Point taken re 8192 points.  On my laptop it takes about 20ms per plot update.  How long does the daq take?  Maybe pipe-lining would help here.

Option 2 - decimate the data for display

8192 points or more exceeds the pixel width of the graph indicator area (unless he needs a super big xy graph and has a super monitor).  For display purposes he could decimate the data with min/max and only display the #of points = pixels width of graph indicator.  Then store the actual data in a separate buffer.   

[M.H.K]

Hello

23 hours ago, bbean said:

I think you will have to create two ring buffers and two graphs, one for displaying realtime data, and another for displaying your "selected plots", then make one graph transparent and overlay it over the other

Thank you for your reply.

For this,Why should not I change their colors?

And,how should I implement this "transparency"?

21 hours ago, ensegre said:

"mine is faster..."

 

Thank you, Is so good.

But,why do you use of visible property and bbean use of defer panel updates?Which one is faster?

20 hours ago, ensegre said:

If it helps

persistence5.vi

Thank you so much

[bbean]

11 minutes ago, mohamad said:

For this,Why should not I change their colors?

And,how should I implement this "transparency"?

Changing the colors of individual plots occurs in the UI thread and is "expensive".  The secondary XY graph avoids this by making all plots the same color...in your case yellow?

Some other things to think about:

1.   if you change your threshold value, do you want to remove the old yellow traces from the display if they no longer meet the threshold criteria?  It gets more complicated that way.
2.   how long does the daq read take, you should explore pipelining to help speed up things as well.  In the link A would be your DAQ and B would be your update of the buffer and display

12 minutes ago, mohamad said:

And,how should I implement this "transparency"?

You make an exact copy of the graph and then use the tools pallette paintbrush, switch the color to transparent T, start erasing things on the graph that aren't needed (everything except the signal plots for the most part), and then move that graph over top of the other graph.   Some ppl will call this a "cluge" and you have to make sure that if the x or y scale change on the visible graph, it also changes on the transparent graph.  So programming challenges arise for instance if you allow the user to autoscale.

 

[M.H.K]

Hello

20 hours ago, ShaunR said:

Isn't the issue that the OP is referring to, that when large data amounts are sent to the graph (regardless of how they were acquired or organised) that the indicator is slow at rendering them? Yes. Ring buffers are great for recording data at high rates but you are still left with the rendering problem, are you not?

I think the OPs issue becomes a little clearer if you make your 500 plots 8192 data points in length instead of 128

It is good point.

First,I wanted to overlay all of them,but ensegre convinced me that I limited their numbers.

anyway,if you have a good idea,i will be happy to hear it.

19 hours ago, bbean said:

Maybe pipe-lining would help here.

Can you explain more?

19 hours ago, bbean said:

Option 2 - decimate the data for display

I cannot use of way of "decimate the data for display".because I only have 30-50 data points for  each of overlaid curves .you can see it and its source code in first post.

[bbean]

28 minutes ago, mohamad said:

I cannot use of way of "decimate the data for display".because I only have 30-50 data points for  each of overlaid curves .you can see it and its source code in first post.

I'm a little confused.  On your first post you show code and an example where you do a peak detection and than pop off traces from the data around those points (samples before and after).  Is that all you need to do?

 

Or do you need to do additional post processing on the samples before and after the peaks (analyze each set of data around the peaks).  For example in the post with these screenshots:

you show another threshold line.  Is there a second post processing step (on the traces that were pop'd off of the original signal) that you want to do "online" with another threshold?

Also do you want to be able to change the samples before and after in real-time as you are collecting data or can it be fixed for each session?

[M.H.K]

47 minutes ago, bbean said:

Changing the colors of individual plots occurs in the UI thread and is "expensive"

 

53 minutes ago, bbean said:

if you change your threshold value, do you want to remove the old yellow traces from the display if they no longer meet the threshold criteria?

Yes,they will display until meet the threshold criteria,as template curve has such condition.

59 minutes ago, bbean said:

  how long does the daq read take, you should explore pipelining to help speed up things as well.  In the link A would be your DAQ and B would be your update of the buffer and display

in my daq,FIFO size is 1024,and I read 8192 samples each time.

 

1 hour ago, bbean said:

You make an exact copy of the graph and then use the tools pallette paintbrush, switch the color to transparent T, start erasing things on the graph that aren't needed (everything except the signal plots for the most part), and then move that graph over top of the other graph.   Some ppl will call this a "cluge" and you have to make sure that if the x or y scale change on the visible graph, it also changes on the transparent graph.  So programming challenges arise for instance if you allow the user to autoscale.

I have to test it and I will say you its result.
 

[M.H.K]

40 minutes ago, bbean said:

I'm a little confused.  On your first post you show code and an example where you do a peak detection and than pop off traces from the data around those points (samples before and after).  Is that all you need to do?

It was my first problem, and it is may goal :

40 minutes ago, bbean said:

Or do you need to do additional post processing on the samples before and after the peaks (analyze each set of data around the peaks).  For example in the post with these screenshots:

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------

40 minutes ago, bbean said:

you show another threshold line.  Is there a second post processing step (on the traces that were pop'd off of the original signal) that you want to do "online" with another threshold?

what do you mean ? I have a threshold line that show on plot.

In this picture,we have all of curves ( white ),a group of curves ( yellow ),a template curve ( blue that it is template of yellow curves ), and a threshold line ( blue ).

And in this picture,all of curves have sorted in 5 templates that they show in different plots that the main graph contains all of them is:

It maybe contains with templates curves or not that in this templates curves are hide.

Edited June 23, 2016 by mohamad

[bbean]

how many points are in each plot of graphs you shown in your last post?  8192 or 30-50

Is the template trace pre-calculated before starting or is it an average of some set of the white traces (exceeding some threshold)?  I guess I'm asking if you have multiple different thresholds for each color shown in the graphs?

 

[ensegre]

What I'm led to understand is that the OP wants to emulate this: http://www.plexon.com/products/omniplex-software. I presume that various spike templates are drawn at hand with a node tool (more than just one threshold), and that the computational engine does statistical classification of all what is identified as a spike event in the vast stream of incoming electrophysiological data. Classification could label, say, each spike according to the one among the templates which comes closest in rms sense, but PCA may be more sound.

Now we are at the level of "how to plot many colored curves". At some point we'll be at "how to draw a polyline with draggable nodes", then "how to compute PCA of a 2D array"... Or?

Don't ask, I have my adventures with neourobiologists too...

[M.H.K]

1 hour ago, bbean said:

how many points are in each plot of graphs you shown in your last post?  8192 or 30-50

First,I receive my datas from DAQ with 8192 points, then I detect their picks and show 8 samples of before peak and 24 samples of after peak.

So, I want to show at least 32 sample,not 8192

2 hours ago, bbean said:

Is the template trace pre-calculated before starting or is it an average of some set of the white traces (exceeding some threshold)?  I guess I'm asking if you have multiple different thresholds for each color shown in the graphs?

It is RMS of some set of the white traces.But we have a threshold with several templates that each of them shows firing of a neuron.

[M.H.K]

1 hour ago, ensegre said:

What I'm led to understand is that the OP wants to emulate this: http://www.plexon.com/products/omniplex-software. I presume that various spike templates are drawn at hand with a node tool (more than just one threshold), and that the computational engine does statistical classification of all what is identified as a spike event in the vast stream of incoming electrophysiological data. Classification could label, say, each spike according to the one among the templates which comes closest in rms sense, but PCA may be more sound.

Now we are at the level of "how to plot many colored curves". At some point we'll be at "how to draw a polyline with draggable nodes", then "how to compute PCA of a 2D array"... Or?

,I have made this hardware for my laboratory and now I am trying to write its software with your help.

Edited June 23, 2016 by mohamad

[M.H.K]

Hi

I want to have a graph with a number of plot ( such as 500 ) without changing color ( with a fix color ).

In Labview,when you set a graph with 500 plot in a color and after that you reduce the number of plot and again to 500 plot,color change.

How can I have a graph with a color?
Thank you

[ensegre]

Enforcing the plot color as was already shown above, and commented as being somewhat slow. To the point that a better strategy could be not to change the colors but to cycle the data.

[M.H.K]

I dont want to change the color.I want to have a graph with a color without changing it.I prepare the graph for a color but after reducing the number of plot and then increasing them,color change.

[ensegre]

Again, refer to the examples above, just give the same color to each ActivePlot. A strategy could be not to delete plots, just to replace data with NaN to keep the number of plots constant. Another may be to set the Plot.Visible? property to false individually for the plots to be hidden, but having to do that in a loop may be slow too. Try.