working with large datasets

MegaSoftware · Post by **MegaSoftware** » Tue Dec 09, 2003 7:26 pm

We're working with very large data sets (10^5..10^8) and we need the ability to display data in scrollable chunks. For example, we need to render 10^6 data points into, say, 10^5 pixels. However, we want the user to be able to scroll across the display window width, eg., 500 or 1000 pixels, without having to recompute and redisplay the graph window. We're mainly interested in line charts for this.

What approach is recommended to accomplish this?

One idea we had is to have a horizontal spinner that shows target data values (along the X-axis) and allows increments in 1/2 the visible scrolling window. Clicking the left and right scroll buttons would scroll the data in the window as expected to the left or right.

Thanks
-David Schwartz

Marjan · Post by **Marjan** » Tue Dec 09, 2003 8:58 pm

Hi, David.

Interesting problem. I'd personally use two charts: first one displays full downsampled data and second shows only section of data (not more than one point per pixel). I wouldn't store full data in series, but rather store it outside TeeChart (array, stream, ...) and then pass downsampled values (for first) or only subrange (for second) to specific chart series.

For example, if first chart shows 1.E+10 points and chart width is 1.e+4 pixels, then each screen pixel coordinate on first chart will represent 1.0e+6 points. So, first downsample data to 1e+4 points and add them to first chart. Then based on first chart cursor (scroll bar) coordinate, populate second chart with (zoomed) data range.

Your idea about horizontal spinner is good, but I'd rather use TCursorTool for this. I think using this tool you should be able to implement your idea.

MegaSoftware · Post by **MegaSoftware** » Wed Dec 10, 2003 1:53 am

Thanks, Marjan, this helps a little. I guess what I'm looking for is something more along the lines of: how do I implement the scrolling within TeeChart? The data sampling isn't a problem -- we're used to dealing with that issue.

Can you point me to an example of how to deal with situations where the "paper" that the chart is drawn on is, say, 10x as wide (but same height) as the display window? How do we manage scrolling and all that stuff?

What is TCursorTool? Is it part of the TeeChart package?

Also, unrelated question: is it possible to disable the various mouse-based functions on a chart, like zoom, reposition, etc?

Marjan · Post by **Marjan** » Wed Dec 10, 2003 8:59 am

4205487 wrote:how do I implement the scrolling within TeeChart? The data sampling isn't a problem -- we're used to dealing with that issue.

Ok, this is relatively simple. Scrolling is merely changing axis range (minimum and maximum value). I've posted an article about this some time ago. You can check it at:
http://www.teechart.net/support/modules ... =0&thold=0

4205487 wrote: What is TCursorTool? Is it part of the TeeChart package?

Yes. It more less does exactly what you need

4205487 wrote: Also, unrelated question: is it possible to disable the various mouse-based functions on a chart, like zoom, reposition, etc?

Yes, sure. You can enable/disable scrolling or zooming. This can be done at design time or at runtime via code. There are couple of examples of this available in TeeChart v6 demo.

MegaSoftware · Post by **MegaSoftware** » Thu Dec 11, 2003 6:16 pm

Ok, through some trial-and-error, I've come up with the following chunk of code. It's located in a Redraw function that's called each time a spinner value changes.

Code: Select all

if (ScrollFactor_spin.Value = 1) then begin
     Chart.BottomAxis.Automatic := true;
     Chart.AllowPanning := pmNone;
     ChartScrollBar1.Enabled := false;
     ChartScrollBar1.Visible := false;
     Chart.BottomAxis.SetMinMax(0,full_len);
 end else begin
     Chart.BottomAxis.Automatic := False;
     Chart.AllowPanning := pmHorizontal;
     ChartScrollBar1.Enabled := true;
     ChartScrollBar1.Visible := true;
     ChartScrollBar1.Position := 0;
     n := full_len div ScrollFactor_spin.Value;
     ChartScrollBar1.SmallChange := n div 5;
     ChartScrollBar1.LargeChange := n div 2;
     Chart.BottomAxis.SetMinMax(0,n);
     ChartScrollBar1.Min := 0;
     ChartScrollBar1.Max := full_len;
 end;

When there are 10,000 data points (ie., full_len = 10000), it starts off showing everything in the single window as expected, with 0 at the left and 10,000 at the right. (We are actually using a sliding-window algorithm to smooth the data, so there are in fact only 1/10th that many data points in the default data set, but the axis goes from 0 to 10,000.)

The problem is that when the spinner's value > 1, this always repositions the graph so rather than going from 0..10,000 it goes from some '-x' to 10,000-x, where 'x' appears to be something like (full_len / ScrollFactor_spin.Value).

Furthermore, it displays ok until I adjust the ChartScrollBar, and THEN it shifs to the right and shows 1/2 the graph with negative blank space.

What's up with this strange behavior? Why is the axis getting shifted left so much? It displays a big chunk of negative space with no data there, and it's missing an equivalent amount of space on the right-hand side where there IS data. It's as if the SetMinMax function is being called with (-full_len/ScrollFactor_spin.Value, full_len-(full_len/ScrollFactor_spin.Value)) rather than (0,n).

I have tried a bunch of different things, and have been unable to come up with any combination of Chart.BottomAxis.SetMinMax and ChartScrollBar1.Min and .Max settings that correct this problem.

MegaSoftware · Post by **MegaSoftware** » Thu Dec 11, 2003 9:18 pm

After further investigation, it appears that the ChartScrollBar object ignores the Automatic setting of the chart it's attached to and changes the Min and Max properties so they're [-n,n] where n = (ChartScrollBar.Max-ChartScrollBar.Min)/2.

I set up a ChartScrollBar1Change handler and tried resetting the values for the Chart's Min/Max properties there (where it is clear they have been changed), but this has no effect on the display of the chart.

-David

ps: this is in V5.02.

MegaSoftware · Post by **MegaSoftware** » Wed Dec 17, 2003 1:58 am

I'm waiting to hear if there is a way to fix this problem...

We have data that has no negative values, but the scrolling mechanism I've gotten to work insists on showing a bunch of negative axis space.

How can this be remedied? Is there another way to handle this kind of zoom/scrolling?

Marjan · Post by **Marjan** » Sun Dec 21, 2003 9:33 am

4205487 wrote: The problem is that when the spinner's value > 1, this always repositions the graph so rather than going from 0..10,000 it goes from some '-x' to 10,000-x, where 'x' appears to be something like (full_len / ScrollFactor_spin.Value).

Hmm.. If you use TChartScrollBar component, then you should not change any of it's properties. They are set internally once connected to chart (axis). All this is done in TChartScrollBar.RecalcPosition method. Normally when you scroll/zoom chart the RecalcPosition will be called and chart scroll bar position and thumb size will be sed.

If you have TeeChart source code then you can check how setting scroll bar position and thumb size is implemented. If not, send an email to marjan@steema.com and I'll email you chart scroll bar unit.

Alternatively, if the TChartScrollBar implementation does not work as you expect, you can use regular TScrollBar and do the necessary modifications manually: set then min, max, position, thumb size i.e use code you used to set TChartScrollBar properties.

MegaSoftware · Post by **MegaSoftware** » Wed Jan 14, 2004 7:16 pm

Ok, I modified the code so I am not changing any of the TChartScrollBar's properties. I get the same behavior. I've sent you an email with three screen shots to show what's happening.