Non Linear trend Fit



Here you can download the source code and some binaries for performing a best fit of continuous line segments to time series.

The problem is described in:

Tomé, A. R. and P. M. A. Miranda, Piecewise linear fitting and trend changing points of climate parameters. Geophys. Res. Lett., 31, L02207, doi:10.1029/2003G019100, 2004.

and in

Tomé, A. R. and P. M. A. Miranda, Continuous partial trends and low-frequency oscillations of time series. Nonlinear Processes in Geophysics, 12, 451–460, 2005. SRef-ID: 1607-7946/npg/2005-12-451 European Geosciences Union (http://www.nonlin-processes-geophys.net/12/451/2005/npg-12-451-2005.pdf)

Contents

Non Gui application

Gui Application

Download Gui Application

A IDL Implementation







Non Gui application:

Fortran Source code:


Main File

Some utilities (from John Burkardt, FSU)

Required Lapack and Blas Routines

Binaries:


Linux (you probably need to install g95)

Windows (g95+MinGW statically linked executable, you needn't to install anything)



The executables were made with the command:

g95 -O3 -ffloat-store -ffast-math -o Non_linear_trend_fit Non_linear_trend_fit.f90 sleast.f JonhBukkardt.f90

If you want to use the Intel Fortran (free in Linux for non commercial use and ~50% faster than g95) you must include at compilation time the flag -fp_port to avoid an infinite loop. The infinite loop occurs in LAPACK routines because they depend on rounding effects to determine machine precision, in g95 this is avoid by the flag -ffloat-store.

To know how to use please run

Non_linear_trend_fit -H

in a shell (unix or in a msdos prompt in windows). Please be aware that the command line flags are always upper cases.

The output of this program is a file named Non_linear_fit_trend.out were the solution obeying the imposed conditions is at the end of the file.

For example using the test data file Lisboa.dat,

Non_linear_trend_fit -L7 -M15 -A0.01,1. Lisboa.dat

The -A0.01,1. flag means that there are accepted only solutions with a minimum trend change of 0.01/datapointsinterval whenever there are simultaneously a sign change at that particular breakpoint and solutions with a minimum trend change of 1./datapointsinterval at breakpoint when there is no sign change at that particular breakpoint.

a Non_linear_fit_trend.out file is created starting with:

**********************************************************************
 **********************************************************************
 Best fitting of  nl  continuous line segments to time series       ***
 Code by António Tomé 2005                                          ***
 *                                                                  ***
 invoked command:
 Non_linear_trend_fit -L7 -M15 -A0.01,1.  Lisboa.dat
Month day  year     time
 **********************************************************************
 **********************************************************************


and ending with:
****************************************************************
 --------------------------------------------------------------
 ----------------------Final Results --------------------------
 --------------------------------------------------------------
 ______________
 |serie number | 1
 ______________

++++++case without any condition at breakpoints+++++++

 number of breackpoints found 5 from an allowed maximum of 5
 abcissa, bestfit, partial trend
 1901.000 20.05313 0.04197171
 1918.000 20.76665 -0.02572577
 1933.000 20.38077 0.08554583
 1948.000 21.66397 -0.05317098
 1972.000 20.38787 0.05020165
 1987.000 21.14089 0.003215047
 2002.000 21.18912
 rsquare residual sum= 18.34024
 rsquare, coeficient of multiple determination= 0.4424179
 Minimum distance between breakpoints 15.00000
 Minimum trend change at breakpoint 0.04698660
 L^2*Minimumchage/Standarddeviation 24.80931
  ----»case obeying the imposed condition at breakpoints«----
 number of breackpoints found 4 from an allowed maximum of 5
 abcissa, bestfit, partial trend
 1901.000 20.05365 0.04187382
 1918.000 20.76550 -0.02536218
 1933.000 20.38507 0.08417692
 1948.000 21.64773 -0.05036350
 1971.000 20.48936 0.02869985
 2002.000 21.37906
 rsquare residual sum= 18.74946
 rsquare, coeficient of multiple determination= 0.4299664
 Minimum distance between breakpoints 15.00000
 Minimum trend change at breakpoint 0.06723600
 L^2*Minimumchage/Standarddeviation 35.11160
 ******************LINEAR FIT SOLUTION *****************************
 Value at 1901.000 20.48409
 Value at 2002.000 21.18454
 trend value 0.006935216
 rsquare residual sum= 28.63689
 rsquare, coeficient of multiple determination= 0.1294441





Do to its modular approach were the best solutions for 1 breakpoint till a maximum allowed number of breakpoints are searched, and because, when the maximum number of breakpoints are large, the computer time can be appreciable, the output file keeps two partial results for each number of breakpoints, from 1 till the maximum allowed by the data length and the minimum trend interval (-M ) flag or lesser if you use the (-B) flag. The best fit for that number of breakpoints without any condition at breakpoints and the best fit obeying the imposed trend condition for minimum change at breakpoint (-A) or (-C) flag. At the end the best two solutions in these conditions for all the possible breakpoints number are presented. For complementariness the best linear fit solution is also presented. The solutions are in three column form. The first column is the x breakpoint position, the second column is the y value of the fit at breakpoint, and the third column is the partial trend before the breakpoint. Please look at the previous pictures and at the values of the output file. The Upper figure is the best fit obeying the imposed conditions and the lower one is the best unconstrained fit for a maximum allowed number of breakpoints (five is this case)



Gui Application



To easy the utilization of this methodology I developed a GUI interface, based on DISLIN, free for non commercial applications in Linux and Windows using free compilers, and in the Fortran free compiler g95 (linux binary) and MinGW+g95 (windows binary). This interface gives, not only the result of the non gui application, but also the graphic output of the result, the “detrended” function, or the residuals if you prefer that name (scatter plot). It also gives you the option of solving only the linear problem which consists in obtaining the best fit when the breakpoint positions are imposed.




APLICATION MAIN WINDOW

Here you choose the data file (column wise, in Fortran style, columns separated by commas or spaces).

The first lines to ignore in the datafile. The Missing values (lines with this y value will be ignored)

The column of the x's values and the column of the y' values. You may edit the file and you may choose an editor for that. This file options are also available in other windows).

Then you choose the problem you want to solve:

Non Linear: the breakpoints are evaluated

Linear: you impose the breakpoint positions.

After solving one of the problem in this window you will have the possibility to see the results and save them in a postscript file or in a column wise spreadsheet file easy to import to any spreadsheet application.

NO LINEAR PROBLEM WINDOW

Here you may choose the parameters for solving the nonlinear problem:

The minimum number of data points between two consecutive breakpoints.

The minimum number of data points before the first breakpoint and after the last breakpoint. This three variables must be integers values.

The minimum trend change at breakpoints when there is no sign change and when there is a sign change of the partial trends.

You can still impose a maximum number of breakpoints (integer value).

After clicking in RUN the evolution status is displayed in the drawing Windows. Here the main difference between the windows and Linux version.

In windows the process is on background and you keep access to the interface. In Linux you loose access to the Gui interface.

NO LINEAR PROBLEM WINDOW 2

After running the result is plotted in the drawing widget. Three types of plots can be plotted. The solution as requested (the condition box below the drawing widget). A solution of the unconstrained problem (no conditions at breakpoints) with the number of breakpoints found for the solution (CB_Bestfit) button bellow the drawing widget, are still the best unconstrained solution up to a maximum number of breakpoints allowed (MB_Bestfit button bellow drawing widget).

This and other partial results are all obtained when the program runs so you do not need to do three runs for seeing this results. You only need to choose the wanted option in the radio buttons bellow the drawing widget and click in redraw.

The results are saved in the file:

Non_linear_fit_trend_gui.log

LINEAR PROBLEM WINDOW

Here we solve the linear problem.

One give the imposed breakpoints separated by commas or a file name with breakpoints.

First line number n of breakpoints and the next n lines with the breakpoints values: e.g.

---------------------------

3

1910.

1945.

1975.

_ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

The solution is practically instantaneous.


PLOT DETAILS WINDOW

Here we can choose various values for plotting the graphics.



Many of this values are automatically generated whenever the fitting program (linear or non linear) runs.

Those values can be changed here. Redraw will replot with the new values.

OTHER OPTIONS WINDOW

This window is accessible only through the plot details windows and deals only with plot line and symbols proprieties.

As happens with the plot details windows these options are only relevant if one desires a postscript output.





GUI APPLICATION

Fortran Source Code


LINUX

Main Program

Widget Subroutines

Some Utilities (from John Burkardt, FSU)

Required Lapack and Blas Routines

WINDOWS

Main Program

Widget Subroutines

Some Utilities (from John Burkardt, FSU)

Threading interface

Required Lapack and Blas Routines

Binaries


Linux (not statically linked, you need to install DISLIN). You should have no difficulty.

Windows (statically linked but for the thread, so you need also to put the following dll pthreadGC2.dll library in the directory you put the executable. This library is part of the phreads-win32 project.

Excel macros to import the output of the application in the data format and to automatically build the graphics in excel: Linear Problem and Non Linear Problem.





The Linux binary was compiled with the command:

dlink -a Nonlineartrends sleast.f NLwidgets.f90 JonhBukkardt.f90 -O3 -ffloat-store -ffast-math

The windows binary was compiled:

g95 -o Nonlineartrends.exe -static -mwindows -O3 -ffloat-store -ffast-math Nonlineartrends.f90 sleast.f thread_test.o NLwidgets.f90 JonhBukkardt.f90 libpthreadGC2.a dismg7.a -luser32



Be aware that the sources are different in windows and in Linux versions, in spite having the same name. That was not originally intended, the main reason I choose Dislin to share the same free interface, but I got greedy and in spite of g95 being a single thread compiler I've done a C interface to create a child process when running the non linear problem, because it can be a time consumed process and in this way the gui interface stays active, instead of inaccessible which could give the impression, special in windows, that the application had crashed. Unfortunately, the X libraries complains when the child process try to access the gui interface. The same doesn't happens in windows.

If you want to use the Intel Fortran (free in Linux for non commercial use and ~50% faster than g95). In linux the Dislin is also free for this compiler (the same does not happen with the window version) you must include at compilation time the flag -fp_port to avoid an infinite loop. The infinite loop occurs in LAPACK routines because they depend on rounding effects to determine machine precision, in g95 this is avoid by the flag -ffloat-store.

To easy the use of this application with excel we made two little macros to import the data output and automatically draw graphics.

I'm not a windows user so the way I put this macros visible was in tools -> macros -> Vb editor import file e voilá!

Then tools--> run macros.

If you want to see an example application of the methodology (allowing only one breakpoints) to have a new look at some spatial features of the global spatial distributed temperature field in the last 52 years. Click at:

P. M. A. Miranda, A. R. Tomé (2005), “Large scale spatial structure of observed temperature trends”, Proc. 16th Conference on Climate Variability and Change, American Meteorological Society.

http://ams.confex.com/ams/Annual2005/techprogram/paper_84378.htm

http://ams.confex.com/ams/pdfpapers/84378.pdf



Please feel free to email me any kind of question you may have regarding the code and its installation.



An IDL implementation



Dr. Michael Jarisch, from the Physics Department of the Faculty of Mathematics and Natural Sciences, Wuppertal University.

Has adapted the original code, from the GRL paper, and made a IDL procedure to find the breakpoints whenever there is trend sign change.

You can download it here.



Any questions about that procedure should be directly asked to Michael jarisch@uni-wuppertal.de.


Dr. Michael Jarisch

Faculty of Mathematics and Natural Sciences

Physics Department

Wuppertal University

Gauss-Strasse 20

D-42097 Wuppertal

Germany

Tel: +49-202-439 2779

Fax: +49-202-439 2680





Last modified 20 June 2005

© António Rodrigues Tomé, UBI, Portugal.



Please feel free to email me any kind of question you may have regarding the code and its installation.