Mathematical question

[jbrohan]

Hello

I need to find the three slopes of a rising step shaped curve as shown in the large xy graph. This is for a conductivity program where sedimentation rates are measured as conductivity.

My approach is to find the best polynomial fit in the range of orders 2 to 12 by picking the lowest mean standard error. This seems to be an order 8 curve in my initial test cases.

I then do a simple differentiation (subtract previous from current value through the best fit array). Repeat the differentiation again on the first derivative array for a second derivative.

I had expected the maxima of the first derivative to be the middle of the straight sectors. However the 8th order function is a little lumpy and the second derivative has some unwanted bumps which might hide the maxima.

Can some mathematically inclined member point me in a better direction. I'm after the three straight lines that best fit the three segments of the main curve.

Thanks

John

[Kurt Friday]

Hi John

I did a Savitzky Golay smoothing filter some time ago which should help you obtain a smooth 2nd derivative, it might be worth having a play with.

http://www.sciware.com.au/freesoftware/index.html

Cheers

Kurt

[Francois Normandin]

QUOTE(jbrohan @ Jan 29 2008, 03:06 PM)

Can some mathematically inclined member point me in a better direction.

If you have an 8th-degree polynomial fit, why not do a 2nd derivative on the algebraic solution? It all depends on how good is the fit compared to original signal, but algebraic solution wouldn't have bumps...

y = ax^8 + bx^7 + cx^6 + dx^5 + ex^4 + fx^3 + gx^2 + hx + i

dy/dx = 8ax^7 + 7bx^6 + 6cx^5 + 5dx^4 + 4ex^3 + 3fx^2 + 2gx + h

d(dy)/dx^2 = 56ax^6 + 42bx^5 + 30cx^4 + 20dx^3 + 12ex^2 + 6fx + 2g

Use an algorithm to find the zeroes of your second derivative (inflection points), then keep only those values whose first derivatives are positive. (i.e. where the slope stops increasing and starts decreading). After finding "x", the slope can be found by solving dy/dx for this value...

Hope this helps!

[hfettig]

Hi John,

I don't quite understand why you are trying to fit an 8th order polinomial to a step function.

As I understand it you are looking for the slope of the "bottom" of the step, the "riser" of the step, and the "top" of the step.

Personally I would split the raw data points into three or five groups, depending on which works better. I would calculate the "height" of the step as maximum minus minimum of the the raw data. Naxt I would split the data into three groups: minimum to min + 10% of height, min + 10% to max - 10%, and max - 10% to max. Next I would do a simple linear fit on the three sets of data to get the three slopes.

If the curvature at the top of the bottom and top of the step introduces to much of an error into the slopes, consider splitting the data into five groups: min to min + 5%, min + 5% to min + 20%, min + 20% to max - 20%, max - 20% to max - 5%, and max -5% to max. In this case, discard the data in sets 2 and 4 and do a linear fit on sets 1, 3, and 5.

Splitting into five sets probably gives you a more accurate result. You can then also parametrise the 5% and 20% split points, and vary the values to see which gives you the smallest deviations on you slopes.

Hope that helps.

Cheers,

Heiko

[Anders Björk]

Inspecting your graph would it not be more resonable to use a function that goes to constant when x goes to inf? Eg. a+b*tanh(c*t) or similar.

[Dirk J.]

Isn't there a theoretical curve you could fit?

I mean, it looks like a 'simple' second order step resonse (...)

QUOTE(jbrohan @ Jan 29 2008, 09:06 PM)

Hello

I need to find the three slopes of a rising step shaped curve as shown in the large xy graph. This is for a conductivity program where sedimentation rates are measured as conductivity.

[Francois Normandin]

QUOTE(Dirk J. @ Jan 30 2008, 04:42 AM)

Isn't there a theoretical curve you could fit?

I mean, it looks like a 'simple' second order step resonse (...)

Good idea Dirk... it could be as simple as a RLC type circuit equivalent. More info on the system that produces this step response would be required.

[jbrohan]

My thanks to everybody who wrote in with suggestions. It is so helpful to be part of this community. On a personal note, by son who is studying calculus in school was surprised by my approach to differentiating a line (just take the previous from the current right through the points), but I'll stick with this as it allows smoothing the result before using it.

Here is the approach that I used and the nicely fitting lines it produces over the test set.

John

The outline of the algorithm used to computer these three lines is as follows.

1) determine the bottom and top of the step.

a) do several generalized curve fittings, orders 2..12 and pick the best fit (often order 7 or 8). Draw this line in red

b) differentiate this line, smooth the differential and take the differential again the max and min of the (smoothed) second derivative represent the inflection points.

2) Compute the line for the step riser.

a) Compute the x midpoint and the x range of the riser.

b) select the real data points that lie within the middle half of this range. The idea here is to avoid the inflection points which will tend to make the regression line rotate clockwise.

c) do a linear regression on these points.

d) compute the points on the line +5 and -5 x points and draw the line. The +-5 is just a number to get the line drawn visually beyond the clutter of the graph.

3) Compute the line for the lower step.

a) Take the first 75% of the points between zero and the inflexion point diccovered in step1; compute the linear regression.

b) Draw the line to inflection point +5.

4) Compute the upper step.

a) Similar to the lower step, but instead of using 75% of the range from the end of the graph to the upper inflection point, skip the same number of points as was used in the lower step. This is because the upper step is often longer than the lower one, as there is no urgency to stop the experiment.

[Yuri33]

You're original data looks remarkably like the kind of data you get for enzyme kinematics-related research. They use a special function called the Hill equation to characterize the behavior of this relation. There are several ways to fit the data, most involving taking the logarithm of the data before fitting. A quick google seach found this white paper on one particular approach for fitting the Hill equation appropriate for coding into a computer program. Once the equation is fit, you can easily calculate the slope based on the fitted Hill coefficient.