Problem in NaN data handling

[Nitin]

Hi TeeChart Team,

I am having problem in handling NaN data.
I have been provided a double array of huge data (2.5 million data points).
But i have few NaN data points in it. Now i am not able to plot it.
In TeeChart, i think making the color as transperent or SetNull is the only way to handle it which inturns makes the color as transparent.
Moreover it modifies the actual data which is absolutely unacceptable.
But for that i have to modify the NaN data point and provide a valid value to it.
But i cant modify the actual data as this data is used by many other components in my application where they need to check NaN data.
So, i have to make a copy of the double array and modify the copy so that the data is set to null.
But making a copy of the data is absolutely useless and very costly operation in this scenario as i am having huge data.
Please let me know how i can handle NaN data without modifying the actual data and without making a copy of the data.

[Narcís]

Hi Nitin,

Setting a point to be null using SetNull doesn't modify it's data. SetNull method only sets a point color to be Color.Transparent. Also, it's not necessary to copy the data, you can loop through your dataset, check if values are NaN and place a value you choose for it, for example:

Code: Select all

			double[] y = { 0, 1, 2, Double.NaN, 4, 5, Double.NaN, 7, 8, Double.NaN };

			points1.DefaultNullValue =  0;
			points1.TreatNulls = Steema.TeeChart.Styles.TreatNullsStyle.DoNotPaint;

			for (int i = 0; i < y.Length; i++)
			{
				if (Double.IsNaN(y[i]))
				{
					points1.Add(points1.DefaultNullValue, Color.Transparent);
				}
				else
				{
					points1.Add(y[i]);
				}
			}

In the example above y array would be your dataset.

[Nitin]

Hi,
We cant follow the approach you are suggesting as in that case it will make a copy of the data and will keep it in memory.
Also the size of each data point is much more than normal double value.
We have to handle 2.5 million data points at a time and imagine if all the points are getting stored in memory directly. System will be out of memory.
I am following the below approach -

Code: Select all

double [] YValues = new double [2500000];
double [] XValues = new double [2500000];

//Populating the double arrays with data

//Actually, i dont need to populate the double array here. 
//One out side component will create the double array and will give it to me
//I will be just using the array without modifying it.

_FastLine.XValues.Count = XValues.Length;
_FastLine.XValues.Value  = XValues;
_FastLine.YValues.Count = YValues.Length;
_FastLine.YValues.Value  =  YValues;                  //Assigning value like this does not make any copy.
                                                                       //So we know how much memory will be used
                                                                       //in that case it is
                                                                       // 2500000 * 8 * 2 = 40 MB.

//Setting Null for the NaN data indices
//_Tag.NullIndexes is a list of indexes for NaN data.
//that also is given to me by outside component.

for (i = 0; i < _Tag.NullIndexes.Count; i++)
{
    _FastLine.SetNull(_TagInfo.NullIndexes[i]);    //This call modifies the actual data. That is the problem.
}

TeeChart.Series.Add(_FastLine);

[Narcís]

Hello Nitin,

As I told you before, SetNull does not modify the code. Here's SetNull implementation

Code: Select all

		/// <summary>
		///Sets the specified series point to a null (transparent) point. 
		/// </summary>
		public void SetNull(int valueIndex)
		{
			SetNull(valueIndex, true);
		}

		/// <summary>
		/// Toogles between null (transparent) and regular point.
		/// </summary>
		/// <param name="valueIndex">Point index</param>
		/// <param name="isNull">If true, transform to null point, otherwise transform to normal point.</param>
		public void SetNull(int valueIndex, bool isnull)
		{
			Colors[valueIndex] = isnull ? Color.Transparent : Utils.EmptyColor;
		}

As you can see it only changes point color to be transparent. This is also demonstrated by this code:

Code: Select all

			double[] y = { 0, 1, 2, Double.NaN, 4, 5, Double.NaN, 7, 8, Double.NaN };

			points1.DefaultNullValue = 0;
			points1.TreatNulls = Steema.TeeChart.Styles.TreatNullsStyle.DoNotPaint;

			for (int i = 0; i < y.Length; i++)
			{
				if (Double.IsNaN(y[i]))
				{
					points1.Add(5);
					points1.SetNull(i);
				}
				else
				{
					points1.Add(y[i]);
				}
			}

If you associate a ChartController to the chart, open the editor and go to the data tab you'll see null points still have y=5. Same as when assigning arrays directly to the series:

Code: Select all

			double[] x = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
			double[] y = { 0, 1, 2, Double.NaN, 4, 5, Double.NaN, 7, 8, Double.NaN };

			points1.DefaultNullValue = 0;
			points1.TreatNulls = Steema.TeeChart.Styles.TreatNullsStyle.DoNotPaint;

			points1.XValues.Count = x.Length;
			points1.XValues.Value = x;
			points1.YValues.Count = y.Length;
			points1.YValues.Value = y;

			for (int i = 0; i < y.Length; i++)
			{
				if (Double.IsNaN(y[i]))
				{
					points1.SetNull(i);
				}
			}

[Nitin]

Thanks Narcís.
I just saw that. SetNull is not modifying the data.

[Avijit]

Hi,
I was using the approach you are suggesting here in this post.
I am able to handle NaN data.
But problem is performance.
I have a data file with 20 data series and each series having 25000 data points.
Out of the 25000 data points, 80% of the data points are NaN data.
Now when i am flowing the below approach, it becomes a huge performance hit as i have to iterate through all the data points and make them null.
Is there any way i can set null for a range of data?? Or any other suggetion to overcome that problem??

Regards,
Avijit

[Narcís]

Hi Avijit,

You could try filling series' color array directly as it's done with X and Y values using points1.Colors array. Remember that to set a null point you need to set it transparent (Color.Transparent).

Hope this helps.