Forecasting accuracy: a comparative study between artificial neural network and autoregressive model for streamflow

Received Feb 16, 2020 Revised Apr 21, 2020 Accepted May 16, 2020 Estimating the reliability of potential prediction is very crucial as our life depended heavily on it. Thus, a simulation that concerned hydrological factors such as streamflow must be enhanced. In this study, Autoregressive (AR) and Artificial Neural Networks (ANN) were used. The forecasting result for each model was assessed by using various performance measurements such as Mean Absolute Error (MAE), Root Mean Square Error (RMSE), Mean Absolute Percentage Error (MAPE), Mean Forecast Error (MFE) and Nash-Sutcliffe Model Efficiency Coefficient (CE). The experimental results showed the forecast performance of Durian Tunggal reservoir datasets by using ANN Model 7 with 7 hidden neurons has better forecast performance compared to AR (4). The ANN model has the smallest MAE (0.0116 m/s), RMSE (0.0607 m/s), MAPE (1.8214% m/s), MFE (0.0058 m/s) and largest CE (0.9957 m/s) which show the capability of fitting to a nonlinear dataset. In conclusion, high predictive precision is an advantage as a proactive or precautionary measure that can be inferred in advance in order to avoid certain negative effects.


INTRODUCTION
Forecasting is a common analysis of trends that been employed in the process of creating a prediction of the future based on past information and present data. Thus, throughout modern life forecasting plays a very important role [1]. Various methods for prediction were developed and used in different fields, such as the stock market [2][3][4][5], electrical load forecasting [6][7], economic forecasting [8][9], medical [10][11] and many more.
Streamflow forecasting is one of the earliest forecasting problems that drew the attention of the scientist. As we begin to grasp the value of estimating streamflow for livelihoods around the stream since the earliest times. In fact, there was a record of the flow level of the River Nile dated around 3000 B.C., therefore the ancient Egyptians annual peak stream level from 3050 B.C. till 2500 B.C. also were discovered [12]. This is one of the longest recorded time series of a natural phenomenon. It isn't just used to characterize the streamflow issue itself yet in addition as a benchmark time series to study and scrutinize completely different forecasting algorithms [13]. In point of fact, improved forecasting accuracy can help the operation of the reservoir system. The adoption of the appropriate forecasting techniques in the hydrological forecasting can help to predict and ease the system in the future because of forecast accuracy improves. Research has found that successful forecasting is dependent on the accurate estimation of the model. Thus, all forecasting techniques including qualitative and quantitative approaches are designed to produce an accurate model. As predicting the future values was very important for environmental protection and flood control. Therefore, data-driven modelling technique has gained popularity in the hydrology field [14]. According to [15], various techniques that have had been adopted to predict the streamflow values such as Artificial Neural Networks (ANN), Stepwise Multiple Linear Regression model (SWMLR), Zero Order Forecasts (ZOF) and etc.
Other than that, Regression Analysis including Simple Regression, Multiple Regression, and Autoregressive (AR) models, has frequently used to forecast future values of streamflow [16]. For instance, [17] demonstrated streamflow forecasting for one month ahead appears more flexible in Autoregressive Integrated Moving Average (ARIMA) and Seasonal ARIMA (SARIMA) models. It showed that the application of ANN obtained a closed fit during the calibration period and outperformed a conventional model of the Winnipeg Flow Forecasting System [18].
Using Box-Jenkins variations, AR model was commonly used to forecast annual flows. While [19] have also demonstrated that ARMA was able to forecast hydrological data such as river flow. This was due to the fact that the data sets used in the study fulfilled the ARMA assumptions which were linear and stationary. Be that as it may, time series with a timescale that not exactly a year for the most part gives a strong seasonality. For monthly or quarterly-monthly streamflow, it was proposed utilizing SARIMA, Deseasonalized ARMA (DARMA) and Periodic ARMA (PARMA) as these models believed able to forecast better compare with others. While [20] forecast the monthly inflow of the Dez dam reservoir using ARMA and ARIMA by increasing the parameter to four to increase the forecast accuracy. Previous study by [21] stated that pattern of runoff forecasting in the United States based on the relative error is SARIMA. While SARIMA (0,1,4) (1,0,1) 12 model with the maximum R 2 and minimum Mean Biased Error (MBE) was capable of long-term average runoff forecasting for all over the United States.
Artificial Neural Networks were generally utilized in a wide range of fields, for example, digital image [22], fault detection [23], gold price forecasting [24] and many more. It has picked up the enthusiasm since rediscovery and popularization of the backpropagation algorithm by Rumelhart and McClelland in 1986. It also accepted as a tool for modelling the complex hydrological data since it is applicable to deal with complicated problems through pattern recognition methodology [25]. The use of ANN approach in water resource problems also has gained more and more popularity due to the complex interrelationships that the system may be nonlinear and multivariate [26]. An interesting advantage of ANN is that they do not need to provide any clear description of the structure they are predicting [27].
Different types of ANN approach have been applied in forecasting the hydrological knowledge [28]. For example, [29] forecast the daily discharge of river basin using Adaptive Neuro-Fuzzy Inference Systems (ANFIS) and it appears more accurate compared to ANN and Multiple Nonlinear Regression (MNLR). Researcher confirms the ability of forecasted 1-day ahead of streamflow using ANN was better than predicted with AR model. This approach was said to be a useful tool in solving a specific problem in hydrology. While [30] has shown that the Radial Basis Function Neural Network (RBFNN) is better in quality and can provide high accuracy and reliability for daily streamflow forecasting, but is less commonly used than the Feed-Forward Back-Propagation Neural Network (FFNN) forecasting model.
Streamflow forecasting using ANN methodology showed a performance that was comparable to the traditional or conceptual technique [31]. The preliminary research discussed the utility of ANN to forecast hydrological variables over the past year, and the subsequent analysis demonstrated the effectiveness of Multilayer Networks with back-propagation training models over conventional statistical techniques. Besides, ANN with proper data pre-processing will have a better performance than the ARIMA model [32]. Therefore, the main focus of this paper is to compare the accuracy of the forecasting method between ANN model and AR in prediction the future values of streamflow. It is expected that this study will contribute to this growing area of research by comparing the forecasting models even though the forecasting performance of the streamflow might be different between methodologies.

RESEARCH METHOD
This section will discuss briefly on the case study, reseach methodology and evaluation performances, used in the research.

Study area
A daily timescale dataset of streamflow for the Durian Tunggal Reservoir was used in this study.

Box-Jenkin approach
The AR model commonly used to forecast annual flows. The AR (p) forecasting model defines as: where t y is the predicted value in the period t in the term of the first p data in time series; i  is coefficients associated with each previously observed value and t  is a normal white noise process with zero mean and variance. In this study, AR models were fitted to the streamflow data.

ANN approach
Artificial Neural Network is a nonlinear application that used to pre-processing the inputs and outputs. Most of the suggested neural network structure for modelling is MLP which is a feed forward network. Usually, the structure consisting of three connected layers of neurons. The number of neurons in the input and output layer is specified by the problems to which the network is applied. The layers are composed of input as the input data, hidden layer as data processing and output as: where is the vector of weights, is a vector of inputs, is biased, and is non-linear activation function. Before applying ANN, [33] suggested to rescale the data in the range of [0,1] by using min-max normalization.

Evaluation of model performance
A prediction error is calculated in order to evaluate the adequacy of each model in terms of how well the model forecast. Therefore, five types of measurement error are used: where is the ith actual observation for the constituent being evaluated, ̂ is the ith simulated value for the constituent being evaluated, ̅ is the mean of the observed data for the constituent being evaluated, and n is the total number of observations. The lower the MAE, RMSE, MAPE, and MFE and the bigger the CE the higher the accuracy of the forecast. Figure 2 shows the data from 1 st June 2008 until 31 st May 2014 were split into training and testing datasets. The changes in the volume of Durian Tunggal streamflow were plot against time to identify the pattern. However, there is no obvious pattern existed such as trend, cyclic or seasonal pattern. The Box-Jenkins model is applied to stimulate Durian Tunggal streamflow. Since the time series is stable, AR models are determined. From the training model, six alternative models were selected. Alternative model AR (3) with the smallest AIC, BIC values and AR (6) with the largest correlation was chosen as a possible tentative model for forecasting. Next, the Box-Jenkins model procedure resulted in 6 alternative AR models. The inputs of the forecasting measurement for alternative AR models are stated in Table 1. The error measurement of these alternative AR models is compared to each other. Hence, MAE, RMSE, MAPE, MFE, and CE were used to measure the error. Based on the error measurement of alternative testing AR models, it is concluded that AR (4) is the best model.  Next, the datasets of Durian Tunggal streamflow are tested by using ANN model. By following ANN procedure, the datasets were normalized in the range of [0,1] with the Min-Max Normalization formula. Several alternative models are obtained to determine the best forecasting model. The ANN that has been used is Multilayer perceptron (MLP) which has input, hidden layer and output. Therefore, the ANN model will capture and determine all the information of the data by moving forth and back between the layers. The data lagging technique also being applied in this model to compliment the daily data. Hence, seven models are built with ten different hidden neuron numbers and 10 times cross-validation. The desired model is determined based on the smallest forecasting performance error.

RESULTS AND DISCUSSION
For further analysis, a comparison of the error measurement of the Box-Jenkins model and ANN model were conducted. From the input data result of the error measurement of AR and ANN models in Table 2, it compares the model that has a better forecast performance based on MAE, RMSE, MAPE, MFE, and CE. The final finding shows that ANN (7-7-1) model has the smallest values of MAE (0.0116 m 3 /s), RMSE (0.0607 m 3 /s), MAPE (1.8214% m 3 /s), MFE (0.0058 m 3 /s) and largest CE (99.57% m 3 /s) among the others. Therefore, model ANN (7-7-1) is declared as the best model followed by ANN (4-4-1) and ANN (5-5-1). These models can be defined as the second and third best approaches and have outperformed AR models. Figure 3 (a -c) illustrates the 1:1 plot line to examine the agreement level between the values of the chosen models. Based on the figures, it can be observed that ANN models able to predict accurately with slight over-forecast as the data points were located slightly above the equal 1:1 line. Overall, ANN has overforecast the Durian Tunggal streamflow values with 0.58%, 0.66% and 0.7% for ANN architectures of 7-7-1, 4-4-1 and 5-5-1, respectively. Meanwhile, Figure 4 (a-c) shows a comparison of the observed and the predicted values. Most of the predicted data points in Figure 4

CONCLUSION
The purpose of this study was to use ANN and Box-Jenkins to develop two different models. The model was then compared against each other. In this analysis, ANN model was presented even with difficulties throughout deciding the appropriate model input, hidden layer and learning rate. Thus, the result shows that with lower error performance, the ANN model performs better than AR approach in forecasting streamflow and it confirms the ability of this approach in providing a useful model in solving stream flow problems with nonlinear and stationary datasets. Although the findings of the studies are very encouraging and may help to improve decision-making in the development of flood protection systems or water resource initiatives, the theoretical models still need to be improved before they can be implemented.