Multilayered Neural Networks in Approximation of Macroeconomic Dependencies
- Details
- Category: Articles
Internet version without in-text references, based on: Paweł Rośczak, Multilayered Neural Networks in Approximation of Macroeconomic Dependencies, [In:] Some Aspects of Computer Science, Eds. D. Rutkowska, J. Kacprzyk, A. Cader, K. Przybyszewski, Academic Publishing House EXIT, Warsaw 2011, pp. 378–389, http://www.rosczak.com/index.php/en/macroeconomic/, 2012-01-26.
Abstract
The text undertakes the problem of implementing feedforward, multilayered neural network in macroeconomics. Neural methods are used to approximate macroeconomic dependencies and correlations usually being researched by econometric models. What the paper describes is the key differences between training the neural network and building the econometric model. The text also refers to the results and properties of the neural network system built on the basis of EMIL, i.e. the existing econometric macro model of Sweden economy. It shows that the neural network can be an additional useful tool in researching unclear, nonlinear economic issues.
1. Introduction
Econometric model is a quantitative reflection of economic phenomena or dependencies existing in the real world. Basic requirement for every model is an essential similarity to the selected features of the original. Constructing a classical econometric model involves several standard steps including: selection of explanatory and explained variables, determination of the mathematical form of a model, estimation of parameters and statistical analysis of the similarity between the created model and reflected phenomenon. Building traditional equations model is the best solution when the structure of researched issue is well known and the internal dependencies are relatively simple. However, when the knowledge about modeling dependencies is little, or when there is a suspicion that correlations have a very complex form, there arises a need to apply other tools. Such tools are artificial neural networks. Neural methods do not require any assumptions on the mathematical form of modeling correlation and are able to approximate complex nonlinear dependencies.
Economic dependencies are usually nonlinear. Many macroeconomic issues contain different rival theories leading to different structural models. This forms grounds for the use of neural networks which have an additional advantage in macroeconomics. Neural networks do not have strict limitations on the minimal size of data sample. This is an important feature because macroeconomists usually have short data samples with a significant random disturbance. Training the neural network for the approximation of economic dependencies differs from constructing the econometrical model in several steps.
2. Feedforward Multilayered Neural Network
Neural network is a useful tool for function approximation. Dependencies and correlations between macroeconomic variables can be treated as unknown stochastic functions which the neural network should be able to approximate. Approximation is based only on a data sample. A feedforward multilayered neural network is the most suitable network type for the approximation tasks.
The fundamental component of neural calculus is a neuron – an element performing simple mathematical transformation. A neural network is a set of interconnected neurons. The neurons are grouped in layers. Every layer usually consists of neurons of the same type. A multilayered network contains an input layer, one or two hidden layers and an output layer. The input layer does not perform any mathematical operations and is meaningless for the calculus. The hidden layers have neurons with linear summation function and nonlinear activation function. Neurons from output layer are the same, except that they can have identity transformation as an activation function. The term “feedforward” means that an input signal goes thorough the network from the input to the output layer, and the network outputs are not connected to network inputs.
For the purpose of performing further analysis and presenting an application example, a feedforward multilayered neural network with one or two hidden layers will be used. The given network has neurons with sigmoid function as an activation function in all of the neurons including the output layer. The neuron transformation is shown in Equation 1.
Equation 1.
y = 1 / (1 + exp(– x w) )
y – neuron output.
x – row vector of neuron inputs.
w – column vector of neuron weights.
3. Removal of Dummy Variables
The first phase of building neural network designed for the approximation of macroeconomic dependencies is determination of its inputs and outputs. This task is similar to the building of econometric macro model and the sets of explanatory and explained variables for the existing equation model can be used. The next phase requires defining the mathematical form of equations for econometric models. This step in not applicable to the neural network because neural methods can approximate nonlinear function, and the creator cannot influence the hiperplane surface represented by network weights.
Neural network automatically defines the shape of implemented function during training. The given properties reject the possibility of analyzing network parameters stability and adding dummy variables applied in econometric models. It is impossible to find parameters defined as structural in the neural architecture because the overall form of neural function is the resultant of many weights interaction. Separate weights influence on the output is practically impossible to interpret. The neural network does not need dummy variables because it performs nonlinear approximation and is able to learn even few observations which differ from the rest of data sample significantly. Dummy variables need to be removed from the data sample of the econometric model applied in the neural network training.
4. Removal of Deterministic Trend
Many macroeconomic variables reveal a tendency towards changes in their values which are unidirectional and stable in time. This phenomenon is called a deterministic trend and it leads to building simple trend models. Multilayered neural network performs interval approximation. It is not able to approximate infinite deterministic trend and should not be used with variables which include the trend. During training the neural network adjusts its weights to the given empirical observations, which means that the neural approximation is accurate only in an interval designated by data sample. Additionally, outputs of multilayered networks belong to a strictly defined interval and cannot exceed the interval limits.
From the statistical point of view output and input variables should be stochastic stationary, i.e. generated by stationary stochastic process. A week stationary should be satisfactory. Probability distributions of the week stationary variables have their basic characteristics: expected value, variance and covariance, independent of time.
Deterministic trend can be removed from variables using common filters. First differences and change rates of variables are the most popular for such methods. Building successful neural networks for the application in macroeconomics requires additional step to ensure that the input and output variables are free form deterministic trend.
5. Rescaling of inputs and outputs variables
Neurons in multilayered neural network have nonlinear activation function, often called a squashing function. Neuron activation function generates outputs only from a given interval. Typical sigmoid function takes values form 0 to 1, excluding boundaries. This means that for a very large absolute value of inputs, the first derivative of activation function converges to zero. Such phenomenon can lead to substantial problems during training based on gradient methods. This includes Error Back Propagation and all modifications of the algorithm. Variability of macroeconomic quantities usually does not cover the given interval and the usage of row data series in the neural network training is very risky. Solution to the problem is rescaling the data set to match an appropriate interval.
Rescaling operation is required for both: inputs and outputs variables. All neurons in the hidden layers, and sometimes in the output layer, have a nonlinear activation function. If the squashing function exists in the output layer, than the neural network generates its outputs only from strictly limited range; for sigmoid function it is (0; 1). Outputs of such a network need to be rescaled in the manner opposite to input values. Even if the output layer has an identity activation function, rescaling of outputs should improve the training process.
Rescaling can be counted as the next additional step in building the neural network designed for the approximation of macroeconomic dependencies.
6. Evaluation of Neural Networks Generalization and Fitness
During training the neural network adjusts its weights to the presented observations. Error measures based on training data set do not provide reliable information about network generalization. Training can lead to overfitting, which means that a neural network adjusts its weights to the observations from data set but is not adjusted to the approximated dependencies. In this case the values of error measures are very small in the data set presented after training and very large in the observations removed from training set. Overfitted neural network is not able to generalize information form the training set and is useless.
To avoid overfitting and to calculate reliable error of the neural network, data set needs to be divided into three subsets: training, test and validation. Training set is used while adjusting network weights. Error calculated form the test set informs when training should be stopped to avoid overfitting. Error measures based on validation set provide reliable information about networks generalization, i.e. fitting to the approximated macroeconomic dependencies.
The sizes of each set are a compromise between searching for the most representative data set for evaluation purposes and the most informative data set for calculating neural weights.
7. Neural Networks System Based on Sweden Economy Model EMIL
Neural version of EMIL model was built as an example of neural network approximation of macroeconomic dependencies. EMIL is an econometric macro model of Sweden economy. Creators of EMIL are Prof Jan B. Gajda from the University of Łodź (Poland) and Prof. Claes-Håkan Gustafson from University Örebro (Sweden). EMIL belongs to the category of structural models and is based on the current macroeconomic theories.
Multilayered neural networks were simulated using software written by the author in Java. Data sample consisted of 34 annual observations from the years 1966 – 1999. The whole data set was divided into training, test and validation subsets. Validation data include 5 observations, i.e. 15% of the sample. Training set contains about 45% and test set contains about 40% of total observations. Training process makes use of adaptive momentum algorithm based on Back Propagation, and the Chan – Fallside method.
EMIL model contains four functions which are approximated by four different neural networks. Different, arbitrarily chosen network structures were used. Each network was trained multiple times, always starting from the random initial weights. Networks with the best generalization abilities were selected to represent four EMIL functions.
7.1. Supply Change Function
The neural network with 6 neurons in the first hidden layer and one neuron in the output layer proved to be the best for the approximation of supply change function. Equation 2 shows details of the supply change function.
Equation 2.
ΔGDP = f1 (ΔINFLEXP, Δ2GDP-1, ΔCAP-1, ΔGO, ΔEXCHREA, ΔGDPFOR, (MS / MS-1) – 1, (P / P-1) – 1, ΔINVINV)
Δ – first difference symbol (change).
Δ2 – second difference symbol.
-1 – lag symbol (subscript).
GDP – Gross Domestic Product in constant prices from the year 1991 (Mrd. SEK).
INFLEXP – expected inflation.
CAP – capital stock in constant prices from the year 1991 (Mrd. SEK).
GO – government spending in constant prices from the year 1991 (Mrd. SEK).
EXCHREA – real exchange rate of the Swedish currency relative to basket of currencies (DEM, GBP and USD).
GDPFOR – foreign GDP, calculated as weighted average of real GDP in Germany, Great Britain and USA.
MS – money supply M1, i.e. physical currency and demand accounts.
P – prices, i.e. consumer price index CPI (1991 = 100).
INVINV – inventory investment in constant prices from the year 1991 (Mrd. SEK).
Out of sample prognoses for the supply change neural network are shown in Table 1. Values of prognosis are relatively close to empirical observations.
| Observation no. | Year | ΔGDP observation | ΔGDP prognosis |
|---|---|---|---|
| 2 | 1971 | 10.00 | 9.37 |
| 5 | 1974 | 34.00 | 29.88 |
| 17 | 1986 | 30.00 | 25.38 |
| 19 | 1988 | 31.00 | 34.33 |
| 29 | 1998 | 45.00 | 41.50 |
7.2. Inflation Rate Function
The best approximation of inflation rate function was also provided by the neural network with 6 neurons in the first hidden layer and one neuron in the output layer. Details of inflation rate function are shown in Equation 3.
Equation 3.
(P / P-1) – 1 = f2 ( (WIND / WIND -1) – 1, (PROD / PROD -1) – 1, (GDP / GDP -1) – 1, (EXCHNOM / EXCHNOM -1) – 1, (PFOR / PFOR -1) – 1), INFLEXP)
WIND – real industry hourly wage (SEK).
PROD – work productivity, i.e. global production to work time ratio.
EXCHNOM – nominal exchange rate of the Swedish currency relatively to basket of currencies (DEM, GBP and USD).
PFOR – foreign prices, i.e. foreign CPI calculated as weighted average of CPI in Germany, Great Britain and USA (1991 = 100).
Out of sample prognoses for the inflation rate neural network are shown in Table 2. Values of prognosis are very close to empirical observations. The researched macroeconomic dependency was approximated with high accuracy.
| Observation no. | Year | (P / P-1) – 1 observation | (P / P-1) – 1 prognosis |
|---|---|---|---|
| 23 | 1991 | 9.41 | 9.40 |
| 12 | 1980 | 13.70 | 13.83 |
| 5 | 1973 | 6.76 | 6.86 |
| 17 | 1985 | 7.46 | 7.74 |
| 30 | 1998 | -0.09 | -1.08 |
7.3. Net Export Change Function
Larger network with 12 neurons in the first hidden layer and one neuron in the output layer proved to be the best for the approximation of net export change function. Details of the net export change function are shown in Equation 4.
Equation 4.
ΔNETEX = f3 (ΔEXCHREA, ΔGDPFOR, ΔGDP, Δ(WIND / P) )
NETEX – net export in constant prices from the year 1991 (Mrd. SEK).
Out of sample prognoses for net export change neural network are shown in Table 3. Values of prognosis are close to empirical observations, but the approximation accuracy is lower than for the inflation rate network.
| Observation no. | Year | ΔNETEX observation | ΔNETEX prognosis |
|---|---|---|---|
| 2 | 1968 | -3.00 | -4.62 |
| 9 | 1975 | -14.00 | -11.81 |
| 20 | 1986 | 0.00 | -4.43 |
| 15 | 1981 | 21.00 | 13.71 |
| 11 | 1977 | 14.00 | 17.99 |
7.4. Labor Hours Change Function
The best approximation of labor hours change function was also provided by the network with 12 neurons in the first hidden layer and one neuron in the output layer. Details of labor hours change function are shown in Equation 5.
Equation 5.
ΔLHRS = f4 (Δ(WIND / P), ΔGDP, ΔPRIRRAT)
LHRS – labor hours in millions.
PRIRRAT – private to total GDP ratio, private sector production to global production ratio.
Out of sample prognoses for the labor hours change neural network are shown in Table 4. Values of prognosis differ considerably from empirical observations.
| Observation no. | Year | ΔLHRS observation | ΔLHRS prognosis |
|---|---|---|---|
| 25 | 1991 | -146.1 | -119.02 |
| 13 | 1979 | 36.52 | 54.13 |
| 29 | 1995 | 125.23 | 80.19 |
| 19 | 1985 | 93.92 | 10.14 |
| 20 | 1986 | 36.53 | 45.73 |
7.5. EMIL Applet
Neural networks system for EMIL model is available on the Internet as Java applet. It consists of a neural network simulator, input and output data set used during the research, set of forms for free data entering, message boxes with explanation written in English and Polish and a graph module (see Figure 1).
Figure 1. EMIL Java applet (http://www.rosczak.com/emil/)
Graph window uses the open source Java library JFreeChart ver.1.0.0. Graph displays two dimensional representation of dependency between the output variable and the given input variable with fixed values of the remaining inputs (see Figure 2).
Figure 2. Graph window for EMIL applet
8. Conclusions
Traditional econometric macro models require clear assumptions on the mathematical form of equations. Modeling correlations also needs to be relatively simple. Neural network is able to approximate economic dependencies without the knowledge of correlations or dependencies forms. It can be applied to the complex, unclear macroeconomic problems as an additional analysis tool.
Training neural network differs from building econometric macro model. When neural methods are applied on the basis of an existing model and its data, several additional steps must be taken. There is no reason for using dummy variables in the neural calculus. If such variables are part of the base model, their should be ignored in the neural implementation. Multilayered neural network performs an interval approximation which implies that the network is not able to learn a deterministic trend. If such phenomenon exists in data sample, it needs to be removed, e.g. using the first differences. Input and output variables almost always require rescaling to facilitate or to improve the training process of the neural network.
Training of the neural network often leads to overfitting. Overfitted network is useless with the data from an outside training subset. This defect is not considered after building an econometric model. To avoid overfitting, data samples is divided in two or three parts to protect and test the network against the defect.
In order to verify the theoretical assumptions a neural network system was built based on the Sweden macro model. It contained four feedforward multilayered neural networks corresponding to the four functions of EMIL model. Networks were successfully trained and tested against overfitting. Neural network which approximated inflation rate function has revealed the smallest error in the training data and generated a very accurate prognoses compared with out of sample observations. Neural methods and econometric models are hard to compare but the described results prove that the neural network approximates the macroeconomic dependencies with significant accuracy and can be a useful tool in this field.
References
Duch W. (Ed.), Korbicz J. (Ed.), Rutkowski L. (Ed.), Tadeusiewicz R. (Ed.), 2000, Neural Networks, Nałęcz M. (Ed.), Biocybernetics and Biomedical Engineering 2000, Vol. 6, EXIT Publisher, Warsaw, Poland (in Polish).
Gajda J. B., Gustafson C. H., 1999, EMIL, An Econometric Macro Model of Sweden, Working paper no. 7, Örebro University, Örebro, Sweden.
JFreeChart, http://www.jfree.org/jfreechart/, 2003-07-01.
Rośczak P., 2004, Adaptative, Momentum Back Propagation Algorithm, In: Trzaskalik T. (Ed.), Methods and Applications of Operational Research '04, Karol Adamiecki University of Economics, Katowice, Poland, pp. 323–334 (in Polish) (also at http://www.rosczak.com/aebp/, 2007-04-01).
Rośczak P., 2003, Implementation and Application of Multilayered Neural Network in Macroeconomic Modeling, MA Theses, University of Łodź, Łodź, Poland (in Polish) (also at http://www.rosczak.com/mlp/, 2007-04-01).
Rośczak P., Model of Sweden Economy EMIL in the Form of Neural Network, http://www.rosczak.com/emil/, 2007-04-01.
Sarle W. S. (Ed.), 1997-2002, Neural Network FAQ (newsgroup comp.ai.neural-nets), SAS Institute, ftp://ftp.sas.com/pub/neural/FAQ.html, 2003-07-01.
Siedlecki J., (Ed.), 2001, Application of Artificial Neural Networks in Economic Modeling, Wrocław University of Economics Publishing House, Wrocław, Poland (in Polish).
Zieliński J., (Ed.), 2000, Intelligent Systems in Management, Theory and Practice, WNT, Warsaw, Poland (in Polish).
