Displaying Large Data

[M.H.K]

Hello,

I am writing a program for overlapping.

But after about 5000 of overlay,it begin to slow.

Could you help me?

I attached vi,front panel picture and block diagram picture. 

(for simulation, I used Biomedical Toolkit)

Thank you

sample.vi

[JKSH]

14 hours ago, mohamad said:

But after about 5000 of overlay,it begin to slow.

That's normal, because plotting 5000 graphs requires lots of computing resources.

May I ask why you want to plot so many overlays at the same time? After all, it's very difficult to see them.

[Nagaraj Hc]

Turn off synchronous display if it is already enabled, this will save some time but graph updates at lower rate compared based on UI thread availability. I don't why you want to overlay above 5000 plots, instead we should try different representations

[Wouter]

If you want to know if your data is within population I think it would be best to simply calculate the mean, mean + 3*std and mean - 3std of all datasets and plot those along with your new dataset.

[M.H.K]

On 6/17/2016 at 4:37 AM, JKSH said:

That's normal, because plotting 5000 graphs requires lots of computing resources.

May I ask why you want to plot so many overlays at the same time? After all, it's very difficult to see them.

hello JKSH

thanks for your reply.

It is true,but I need to overlay many plots,it is a technique  for neuroscience research ( Spike Detecting ),I saw this feature in other application without slowing:

 

It is 100000 overlay!

I think,if we do not copy all of data,it is possible.

It is mean that if we hold all of plots and only overlay recent plot,it is possible.

Have you an idea?

Thank you

19 hours ago, Nagaraj Hc said:

Turn off synchronous display if it is already enabled, this will save some time but graph updates at lower rate compared based on UI thread availability. I don't why you want to overlay above 5000 plots, instead we should try different representations

Hello Nagaraj Hc

Thank you for your reply.

synchronous display has been off. I explain about your comment in above ( reply of JKSH ).

Have you an idea?

Thank you

15 hours ago, Wouter said:

If you want to know if your data is within population I think it would be best to simply calculate the mean, mean + 3*std and mean - 3std of all datasets and plot those along with your new dataset.

Hello Wouter

Thank you for your reply.

It is not true,because I need to all of data and after that I have to do many calculations over these plots.such as: peak & trough figure,energy figure,PCA figure,spike sorting and etc.

I explain about my goal in above ( reply of JKSH ).

Have you an idea?

Thank you

 

Edited June 18, 2016 by mohamad

[ShaunR]

The speed maybe more to do with the for loop to create the array than the plot itself. You could use the picture plot functions to draw each trace without erasing the previous ones. Then you wouldn't have to manipulate the 2D array. That would probably get you the plot image you want but you will need to think hard about measurements since you won't be able to just read the plot data back later.

Edited June 18, 2016 by ShaunR

[M.H.K]

hello ShaunR

Thank you for your reply.

1 hour ago, ShaunR said:

The speed maybe more to do with the for loop to create the array than the plot itself

If I disable plot it can overlay without finitude. But I need to displaying them.

1 hour ago, ShaunR said:

Then you wouldn't have to manipulate the 2D array.

I need to manipulate data,and I must select each of overlaid plots later.

So,my suggestion is not good.

Have you an idea for overlaying too many plot without slowing?

Thank you

[ensegre]

I think you should part with the idea of regenerating a plot at every new event; storing vast data for later analysis is one thing, display for GUI feedback another.

Instead of trying to plot 5000 curves at every iteration of the loop, you could try maintaining a different loop plotting, at lower rate, the most recent N, with N suitably large.

Another approach I thought of, is using an intensity plot to display a 2D histogram. Grid the time-amplitude space in bins, and increment each bin every time a curve passes in it (you have to write the code for it, I don't know of a ready source); thus you get a plot similar to that of a scope with infinite persistence, brighter where traces overlap. Different loop and lower rate, ditto.

[ensegre]

If this helps...

 

[M.H.K]

Hello ensegre

Thank you for your reply and I am sorry for the late reply.

On 6/19/2016 at 0:37 AM, ensegre said:

Instead of trying to plot 5000 curves at every iteration of the loop, you could try maintaining a different loop plotting, at lower rate, the most recent N, with N suitably large

I cannot bring rate down and cannot separate these two cases from each other.of course, it should not be impossible,because I have already saw this feature in another app :

In this,both threshold line changes curves situation online, and you can select a group of curves online for subsequent analysis.

On 6/19/2016 at 0:37 AM, ensegre said:

is using an intensity plot to display a 2D histogram

It is a good suggestion, but I have to access to each of curves for selecting them and sorting them to different categories based on template, similar this:

If I use of intensity plot, I cannot analyze them later.

Do you think, it is possible to overlay curves without harming any of them?

Have you an idea?

Thank you

[JKSH]

1 hour ago, mohamad said:

I cannot bring rate down

Could you collect all your data first, and then plot them afterwards?

 

1 hour ago, mohamad said:

it should not be impossible,because I have already saw this feature in another app

Out of curiosity, what is the name of this other app?

It's possible that your other app was especially designed and optimized for one specific use-case (visualizing a huge number of ECG graphs). LabVIEW graphs are general-purpose and very flexible, so it can be out-performed by a single-purpose, less flexible graphing tool.

 

[ensegre]

I have somehow the impression that you're assuming that there should be some magic option of the LV plot widget as is, which saves from application design. What you show about hand drawing a template curve, and classifying traces, hints that there is more computation going on behind the scenes. The display in the plot window must be a somewhat sophisticated GUI, only giving the user a feeling of what data is been accumulated and how to interact with it, quite likely not doing the mistake of replotting the whole dataset for every new curve acquired (that would be expensive). The right question might be whether you can use the LV graph widget for what it gives, or rather the picture widget as Shaun suggested above (that would overplot directly to the pixmap, I guess, and be significantly faster), as a starting point for building such a GUI. The LV graph widget, optimized as it may be, is slow because it is complex: it handles autoscaling, abitrary number of curves, arbitrary plotting stiles, filling and whatnot, and this comes with a price. Someone may correct me, but I don't even think it can handle adding incrementally new curves, without replotting the whole bunch.

If I would have to build something the like, I would store the data in memory appropriately and without connection to the graph; I would consider to update that display only periodically, fast enough to give the user a feeling of interactivity, and track user clicks on the graph area in order to replot the data with the right highlights, to give the user the impression s/he is playing with it.

[M.H.K]

Hello JKSH

Thank you for your reply and I am sorry for the late reply.

On 6/20/2016 at 9:46 AM, JKSH said:

Could you collect all your data first, and then plot them afterwards?

 

Actually,no.

Because, Those curves have to see for deciding about sorting them.

On 6/20/2016 at 9:46 AM, JKSH said:

Out of curiosity, what is the name of this other app?

 

Its name is OmniPlex and you can download its demo from this link:

http://www.plexon.com/sites/default/files/OmniPlexDemoV1Setup.zip

It is a software about online spike sorting for neuroscience research.

On 6/20/2016 at 9:46 AM, JKSH said:

It's possible that your other app was especially designed and optimized for one specific use-case (visualizing a huge number of ECG graphs). LabVIEW graphs are general-purpose and very flexible, so it can be out-performed by a single-purpose, less flexible graphing tool.

In my opinion,Labview is able to do it, but it need to more study.

Thank you

Edited June 22, 2016 by mohamad

[M.H.K]

Hello ensegre

Thank you for your reply and I am sorry for the late reply.

On 6/20/2016 at 10:55 AM, ensegre said:

I have somehow the impression that you're assuming that there should be some magic option of the LV plot widget as is, which saves from application design. What you show about hand drawing a template curve, and classifying traces, hints that there is more computation going on behind the scenes. The display in the plot window must be a somewhat sophisticated GUI, only giving the user a feeling of what data is been accumulated and how to interact with it, quite likely not doing the mistake of replotting the whole dataset for every new curve acquired (that would be expensive). The right question might be whether you can use the LV graph widget for what it gives, or rather the picture widget as Shaun suggested above (that would overplot directly to the pixmap, I guess, and be significantly faster), as a starting point for building such a GUI. The LV graph widget, optimized as it may be, is slow because it is complex: it handles autoscaling, abitrary number of curves, arbitrary plotting stiles, filling and whatnot, and this comes with a price. Someone may correct me, but I don't even think it can handle adding incrementally new curves, without replotting the whole bunch.

You have convinced me.

I have 16 channel of analog input and I have to plot them,so I use to picture widget ( Shaun's suggestion ) for all of them and only, I plot one of them with LV graph widget.

But I have a question : How many curves LV graph widget is able to plot without slowing?

On 6/20/2016 at 10:55 AM, ensegre said:

If I would have to build something the like, I would store the data in memory appropriately and without connection to the graph; I would consider to update that display only periodically, fast enough to give the user a feeling of interactivity, and track user clicks on the graph area in order to replot the data with the right highlights, to give the user the impression s/he is playing with it.

It is a good idea and I think about it.

But I have a question: If I want to show 1000 recent curves,which way I have to use?

By that I mean,I decided that 1000 Curves is always displayed and after plotting 1001st, the first curve is removed and it is always repeated.

Have you an idea for it?

Thank you

[ensegre]

5 hours ago, mohamad said:

But I have a question : How many curves LV graph widget is able to plot without slowing?

Plotting over a flattened pixmap (rather than incrementally adding to the picture new plot data, that's the trick) can be fast, and has the same cost no matter the previous image content. Like this:

Quote

But I have a question: If I want to show 1000 recent curves,which way I have to use?

By that I mean,I decided that 1000 Curves is always displayed and after plotting 1001st, the first curve is removed and it is always repeated.

I guess that periodically replotting the whole lot of curves would become too expensive even this way. In a case like this perhaps one should manipulate directly the pixmap, like erasing the pixels of the oldest curve first with an xor operation, and then overplotting the newest. Unfortunately the internals of the picture functions, for attempting something like that, are not well documented, IIRC. I think that, much time ago, I ran into some contribution which cleverly worked them out. Maybe some other member is able to fill in?

Edited June 22, 2016 by ensegre

[M.H.K]

Hello ensegre

Those questions that I asked you, were about waveform graph.

5 hours ago, mohamad said:

How many curves LV graph widget is able to plot without slowing

 

5 hours ago, mohamad said:

But I have a question: If I want to show 1000 recent curves,which way I have to use?

By that I mean,I decided that 1000 Curves is always displayed and after plotting 1001st, the first curve is removed and it is always repeated.

Have you an idea about it?

Thank you

[ensegre]

16 minutes ago, mohamad said:

Those questions that I asked you, were about waveform graph.

This may depend on many factors, such as what is the acceptable cpu load on your system for that part of the application, the pixel size of your plot, your graphic card, and just about everything else. You have to find it out in your working conditions.

Quote

Have you an idea about it?

From the ringbuffer, you have the data of the oldest curve which you're going to discard, and of the newest which is going to replace it. If it is a waveform graph, you can access its data, replace the relevant wave, and replot everything (that is going to be somewhat slow). You could begin with a waveform graph initialized with N NaN filled waves, and replace them with data as it comes in, to avoid growing, but I doubt that it will make a difference a regime.

Maybe the histogram way is still the easiest. Because in its way the histogram flattens the data, but makes still easy to add +1 to the new points and -1 to remove points. For a change you could just binarize the histogram for plotting, and plot in the same color all bins with a count>0. The limitation of the histogram in the form I sketched is that bins are filled according to the datapoint falling in them and not according to the segment traversing them, but perhaps that can be worked out too. I mean, complex requirement, complex solution to be sought.

Edited June 22, 2016 by ensegre

[M.H.K]

I think,I set it wrong.

5 hours ago, mohamad said:

But I have a question: If I want to show 1000 recent curves,which way I have to use?

By that I mean,I decided that 1000 Curves is always displayed and after plotting 1001st, the first curve is removed and it is always repeated.

It is also about waveform graph.

I am sorry.

[ensegre]

Just tried out this out of curiosity, and no, obviously replotting the whole ringbuffer periodically is not a good idea; at least not like this. On my system, it takes about  5 secs each time.

But OTOH, to my great surprise, a plain waveform graph with 500 plots is faster:

YMMV...

[bbean]

2 hours ago, ensegre said:

But OTOH, to my great surprise, a plain waveform graph with 500 plots is faster:

YMMV...

Nice.

Building off of ensegre's solution, here is another option for ring buffer (below).  I think its an order of magnitude slower, but it allows older plots to fade away.  

I think you will have to create two ring buffers and two graphs, one for displaying realtime data, and another for displaying your "selected plots", then make one graph transparent and overlay it over the other

[ensegre]

FWIW, I realized that in my previous snippet the waveform graph needed several seconds of initialization time just because of the loop setting the plot colors, which vanish if I initialize the ringbuffer with NaNs instead of zeros. (Maybe hiding the graph while setting could do, too, I haven't tried).

Also, my snippet with independent loops is clearly just an example and not the real thing, one should plot only if there is really new data and at a lower rate, some mechanism of notification must be in place. As said, application design, not magic bullet.

8 minutes ago, bbean said:

I think you will have to create two ring buffers and two graphs, one for displaying realtime data, and another for displaying your "selected plots", then make one graph transparent and overlay it over the other

Agree. If changing colors of a group of curves is expensive (and moreover these have to be brought to foreground), something must happen behind the scenes.

[ensegre]

"mine is faster..."

 

Edited June 22, 2016 by ensegre

[bbean]

55 minutes ago, ensegre said:

"mine is faster..."

Ha! I knew someone would be able to make it more efficient.  Is the visible property faster than defer panel updates?  

Is there anyway to do the array operation inplace? or is there a "rotate 2d array" primitive hidden somewhere? Also I'm guessing a good portion of the execution time is redrawing all the plots when they shift 1 plot location on each iteration...but maybe I'm wrong.

[ensegre]

Hm. Still we started telling the OP - don't look for tricks, don't try to plot all data blindly. I fear that with this or that optimization the idea can hold with acceptable performance up to some ringbuffer limit, taxing cpu and buffer allocations, but at some point, design should kick in or else.

20 minutes ago, bbean said:

Is the visible property faster than defer panel updates?  

I didn't really time it...

[Norm Kirchner]

Is that snippit working for anyone else? I'm not having any luck w/ it