Nonlinear curve fits needed

[kriesel]

Originally Posted by ewmayer

Ken, the curves in your PDF, did those try to fit an imagined discontinuity at the right end? As described below, we only need to fit the interval x = [1,99] for set1 and x = [1,59] for set2; x < 1 and x greater than those are irrelevant for the present problem.

Here's what happened; no sleep last night, got a 3 hour nap this morning for the spring-forward day, rushed set 2 in my morning, and izarc, view text file (part of it), select all, copy, paste in the right place in a worksheet duplicated from set1 and its 1-99 x values. So the discontinuity was real, because sheet two contained set2 and part of set1 in the fitting range. I didn't see the input data set, just blind copy and paste in. Apparently there was an empty line that overwrote set1's point 61.

Sorry to contaminate the thread like that.
Anyway, for other reasons, I had recently updated to Libre Office 7.1. Its Calc fortunately for this case includes polynomial trendline fitting, and allows multiple trendlines for a single data series. It appears to allow order from 2 up to the number of data points minus 1.

So here is a revised set2 pdf with quadratic through fifth-order fits.
I think it might be simpler to use a table lookup. There's still about a 1.8% discrepancy worst case at fifth order polynomial. So, also a sixth order fit attached.

Originally Posted by ewmayer

..Any help in form of candidate fit functions would be appreciated! Datasets attached...

I'll provide a few things that have helped me:

CurveExpert Professional gives a good bang for your buck, has a large number of curve-fitting routines, provides correlations..etc.. and you can "roll your own."
Some of my data sets run into the millions of data points and instead of fitting a single curve, there exist multiple curves. Identifying the parameters of a single curve within that cloud of data will allow you to deduce a general equation and possibly provide insight into associated equations. No curve-fitting software (that I'm aware of in the public domain with the exception of one algorithm I came upon at NSERC in 2006) exists that will analyze, segregate and interpret your data unless you delve into bioinformatics, spectroscopy/spectrometry and interferometry. There's a great deal more to this, from free excel "add-ins" that allow up to 256 digits per cell (comparable to some spread-sheet formats in some CAS's) to scaling these numbers down to where they become a picture.

I'm attaching a small "dated" data set of about 30,000 points which has been completely determined in terms of the equations that govern this system. The leftmost panel is the 200X vertical expansion of the rightmost panel of columns D through I. These points all represent the factorization of the same number relative to certain changed parameters. There exist alternative and equally intriguing graphs of the same. My point here is to obtain the smallest and largest possible expressions of your data in as many different ways as possible. Next, a pencil and piece of paper, as well as a nice clear spot on your desk where you can bang your head are all that are required.

I don't know what equation(s) generated your numbers but if it's possible to experiment to obtain different values under different constraints perhaps you may perceive your data differently. Good luck.

[kriesel]

Originally Posted by jwaltos

Next, a pencil and piece of paper, as well as a nice clear spot on your desk where you can bang your head are all that are required.

and it explains the headache.

[ewmayer]

Table-lookup param-selection is coded up. Of the various curve-fits tried, polynomials seem ill-suited since the whole point of curve-fitting is to achieve significant data compression, by accurately approximating a large dataset or complicated function via a 2-or-3-parameter function. Functions y = a/x^b + c (a,b,c > 0) seem most promising in the present case - one could probably refine that by replacing x with, say, (x + x^n) to correct [under|over]shooting at [smaller|larger] x, but there seems little point in expending much further effort, since in the end one needs to make a discrete choice among the circled data points, and the table-lookup lends itself well to that. Thanks to all for playing!

[greenskull]

Originally Posted by ewmayer

y = a - b*ln(x)^c, manually play with c
y = a*x^b + c, play with b
y = a*exp(b*x) + c, play with b .

Any help in form of candidate fit functions would be appreciated! Datasets attached.

None of the proposed models will give a good correlation.
Best 3-parametric model for proposed both datasets is y = a+(b*x/(c+x))

Set1:
a = 9,443007248007786E-01
b = -4,046604855620805E-01
c = 3,275195144970118E+00
Standard Error : 2,381512846725821E-04
Correlation Coefficient : 9,999910234077588E-01

Set2:
a = 8,951441384977271E-01
b = -4,061285741069697E-01
c = 3,463710165474020E+00
Standard Error : 1,523876677383889E-04
Correlation Coefficient : 9,999974855245053E-01

Or a slightly worse two-parametric model y = 1+(b*x/(c+x))
Set1:
b = -4,572079209563643E-01
c = 2,533136419817537E+00
Standard Error : 3,541549267449146E-03
Correlation Coefficient : 9,979705677579319E-01

Set2:
b = -5,011796232803167E-01
c = 2,083968117529065E+00
Standard Error : 7,730940811472742E-03
Correlation Coefficient : 9,932660341445618E-01