Jump to content
M.H.K

Displaying Large Data

Recommended Posts

Isn't the issue that the OP is referring to, that when large data amounts are sent to the graph (regardless of how they were acquired or organised) that the indicator is slow at rendering them? Yes. Ring buffers are great for recording data at high rates but you are still left with the rendering problem, are you not?

I think the OPs issue becomes a little clearer if you make your 500 plots 8192 data points in length instead of 128 ;)

Share this post


Link to post
Share on other sites
1 hour ago, Norm Kirchner said:

Is that snippit working for anyone else? I'm not having any luck w/ it

the last snippet from ensegre is not working for me either.

33 minutes ago, ShaunR said:

Isn't the issue that the OP is referring to, that when large data amounts are sent to the graph (regardless of how they were acquired or organised) that the indicator is slow at rendering them? Yes. Ring buffers are great for recording data at high rates but you are still left with the rendering problem, are you not?

I think the OPs issue becomes a little clearer if you make your 500 plots 8192 data points in length instead of 128 ;)

Point taken re 8192 points.  On my laptop it takes about 20ms per plot update.  How long does the daq take?  Maybe pipe-lining would help here.

Option 2 - decimate the data for display

8192 points or more exceeds the pixel width of the graph indicator area (unless he needs a super big xy graph and has a super monitor).  For display purposes he could decimate the data with min/max and only display the #of points = pixels width of graph indicator.  Then store the actual data in a separate buffer.   

Share this post


Link to post
Share on other sites

Hello

23 hours ago, bbean said:

I think you will have to create two ring buffers and two graphs, one for displaying realtime data, and another for displaying your "selected plots", then make one graph transparent and overlay it over the other

Thank you for your reply.

For this,Why should not I change their colors?

And,how should I implement this "transparency"?

21 hours ago, ensegre said:

"mine is faster..." :cool:

 

p5.png

Thank you, Is so good.

But,why do you use of visible property and bbean use of defer panel updates?Which one is faster?

20 hours ago, ensegre said:

If it helps

persistence5.vi

Thank you so much

Share this post


Link to post
Share on other sites
11 minutes ago, mohamad said:

For this,Why should not I change their colors?

And,how should I implement this "transparency"?

Changing the colors of individual plots occurs in the UI thread and is "expensive".  The secondary XY graph avoids this by making all plots the same color...in your case yellow?

Some other things to think about:

  1.   if you change your threshold value, do you want to remove the old yellow traces from the display if they no longer meet the threshold criteria?  It gets more complicated that way.
  2.   how long does the daq read take, you should explore pipelining to help speed up things as well.  In the link A would be your DAQ and B would be your update of the buffer and display
12 minutes ago, mohamad said:

And,how should I implement this "transparency"?

You make an exact copy of the graph and then use the tools pallette paintbrush, switch the color to transparent T, start erasing things on the graph that aren't needed (everything except the signal plots for the most part), and then move that graph over top of the other graph.   Some ppl will call this a "cluge" and you have to make sure that if the x or y scale change on the visible graph, it also changes on the transparent graph.  So programming challenges arise for instance if you allow the user to autoscale.

 

Share this post


Link to post
Share on other sites

Hello

20 hours ago, ShaunR said:

Isn't the issue that the OP is referring to, that when large data amounts are sent to the graph (regardless of how they were acquired or organised) that the indicator is slow at rendering them? Yes. Ring buffers are great for recording data at high rates but you are still left with the rendering problem, are you not?

I think the OPs issue becomes a little clearer if you make your 500 plots 8192 data points in length instead of 128 ;)

It is good point.

First,I wanted to overlay all of them,but ensegre convinced me that I limited their numbers.

anyway,if you have a good idea,i will be happy to hear it.

19 hours ago, bbean said:

Maybe pipe-lining would help here.

Can you explain more?

19 hours ago, bbean said:

Option 2 - decimate the data for display

I cannot use of way of "decimate the data for display".because I only have 30-50 data points for  each of overlaid curves .you can see it and its source code in first post.

Share this post


Link to post
Share on other sites
28 minutes ago, mohamad said:

I cannot use of way of "decimate the data for display".because I only have 30-50 data points for  each of overlaid curves .you can see it and its source code in first post.

I'm a little confused.  On your first post you show code and an example where you do a peak detection and than pop off traces from the data around those points (samples before and after).  Is that all you need to do?

 

Or do you need to do additional post processing on the samples before and after the peaks (analyze each set of data around the peaks).  For example in the post with these screenshots:

4.jpg.93c0f20c7e62d0cd67452fd730c36c12.jpg5.jpg.873e7366337ac7bd725949b02576b584.jpg

you show another threshold line.  Is there a second post processing step (on the traces that were pop'd off of the original signal) that you want to do "online" with another threshold?

Also do you want to be able to change the samples before and after in real-time as you are collecting data or can it be fixed for each session?

Share this post


Link to post
Share on other sites
47 minutes ago, bbean said:

Changing the colors of individual plots occurs in the UI thread and is "expensive"

:thumbup1: 

53 minutes ago, bbean said:

if you change your threshold value, do you want to remove the old yellow traces from the display if they no longer meet the threshold criteria?

Yes,they will display until meet the threshold criteria,as template curve has such condition.

59 minutes ago, bbean said:

  how long does the daq read take, you should explore pipelining to help speed up things as well.  In the link A would be your DAQ and B would be your update of the buffer and display

in my daq,FIFO size is 1024,and I read 8192 samples each time.

 

1 hour ago, bbean said:

You make an exact copy of the graph and then use the tools pallette paintbrush, switch the color to transparent T, start erasing things on the graph that aren't needed (everything except the signal plots for the most part), and then move that graph over top of the other graph.   Some ppl will call this a "cluge" and you have to make sure that if the x or y scale change on the visible graph, it also changes on the transparent graph.  So programming challenges arise for instance if you allow the user to autoscale.

I have to test it and I will say you its result.
 

Share this post


Link to post
Share on other sites
40 minutes ago, bbean said:

I'm a little confused.  On your first post you show code and an example where you do a peak detection and than pop off traces from the data around those points (samples before and after).  Is that all you need to do?

It was my first problem, and it is may goal :

40 minutes ago, bbean said:

Or do you need to do additional post processing on the samples before and after the peaks (analyze each set of data around the peaks).  For example in the post with these screenshots:

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------

40 minutes ago, bbean said:

you show another threshold line.  Is there a second post processing step (on the traces that were pop'd off of the original signal) that you want to do "online" with another threshold?

what do you mean ? I have a threshold line that show on plot.

4.jpg

In this picture,we have all of curves ( white ),a group of curves ( yellow ),a template curve ( blue that it is template of yellow curves ), and a threshold line ( blue ).

5.jpg

And in this picture,all of curves have sorted in 5 templates that they show in different plots that the main graph contains all of them is:

7.jpg

It maybe contains with templates curves or not that in this templates curves are hide.

Edited by mohamad

Share this post


Link to post
Share on other sites

how many points are in each plot of graphs you shown in your last post?  8192 or 30-50

Is the template trace pre-calculated before starting or is it an average of some set of the white traces (exceeding some threshold)?  I guess I'm asking if you have multiple different thresholds for each color shown in the graphs?

 

Share this post


Link to post
Share on other sites

What I'm led to understand is that the OP wants to emulate this: http://www.plexon.com/products/omniplex-software. I presume that various spike templates are drawn at hand with a node tool (more than just one threshold), and that the computational engine does statistical classification of all what is identified as a spike event in the vast stream of incoming electrophysiological data. Classification could label, say, each spike according to the one among the templates which comes closest in rms sense, but PCA may be more sound.

Now we are at the level of "how to plot many colored curves". At some point we'll be at "how to draw a polyline with draggable nodes", then "how to compute PCA of a 2D array"... Or?

Don't ask, I have my adventures with neourobiologists too...

Share this post


Link to post
Share on other sites
1 hour ago, bbean said:

how many points are in each plot of graphs you shown in your last post?  8192 or 30-50

First,I receive my datas from DAQ with 8192 points, then I detect their picks and show 8 samples of before peak and 24 samples of after peak.

So, I want to show at least 32 sample,not 8192

2 hours ago, bbean said:

Is the template trace pre-calculated before starting or is it an average of some set of the white traces (exceeding some threshold)?  I guess I'm asking if you have multiple different thresholds for each color shown in the graphs?

It is RMS of some set of the white traces.But we have a threshold with several templates that each of them shows firing of a neuron.

Share this post


Link to post
Share on other sites
1 hour ago, ensegre said:

What I'm led to understand is that the OP wants to emulate this: http://www.plexon.com/products/omniplex-software. I presume that various spike templates are drawn at hand with a node tool (more than just one threshold), and that the computational engine does statistical classification of all what is identified as a spike event in the vast stream of incoming electrophysiological data. Classification could label, say, each spike according to the one among the templates which comes closest in rms sense, but PCA may be more sound.

Now we are at the level of "how to plot many colored curves". At some point we'll be at "how to draw a polyline with draggable nodes", then "how to compute PCA of a 2D array"... Or?

:thumbup1:,I have made this hardware for my laboratory and now I am trying to write its software with your help.:wub:

Edited by mohamad

Share this post


Link to post
Share on other sites

Hi

I want to have a graph with a number of plot ( such as 500 ) without changing color ( with a fix color ).

In Labview,when you set a graph with 500 plot in a color and after that you reduce the number of plot and again to 500 plot,color change.

How can I have a graph with a color?
Thank you

Share this post


Link to post
Share on other sites

Enforcing the plot color as was already shown above, and commented as being somewhat slow. To the point that a better strategy could be not to change the colors but to cycle the data.

Share this post


Link to post
Share on other sites

I dont want to change the color.I want to have a graph with a color without changing it.I prepare the graph for a color but after reducing the number of plot and then increasing them,color change.

Share this post


Link to post
Share on other sites

Again, refer to the examples above, just give the same color to each ActivePlot. A strategy could be not to delete plots, just to replace data with NaN to keep the number of plots constant. Another may be to set the Plot.Visible? property to false individually for the plots to be hidden, but having to do that in a loop may be slow too. Try.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Similar Content

    • By hhtnwpu
      hi, I want to read and process large data(nearly 100 million rows,the file size over 500MB), the original format of the file is .txt, through rename as .dat we get the binary file. The attachment is my vi. When the file is larger than 1 million lines(the data is singal Column), there is a wrong with "the memory is full".I want to read the data and plot a graph in the time domain, on the graph I can see the detail by zoom tools; and then do some anlysis about FFT and Statistics .I don't know how to do decimation in chunks .another thing ,maybe the memory release is also important. can you help me? thanks!
      the data from a dynamic strain test,the sampling rate is 10K/s。if we can get the whole result use a little data decimation from a chunk ,when we zoom some detail, such as in one chunk, we can get the whole data on the graph without decimation?thanks!
      read data.vi
×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.