Numerical study of the speed’s response of the various intelligent models using the tansig, logsig and purelin activation functions in different layers of artificial neural network

Today's world is no longer that of yesterday, the pace with which we live and also the speed is enormous and rapid, that overnight we discover the appearance of new technologies and solutions in all the fields, in particular, that of scientific research. Artificial intelligence plays the main role. Predicting the behavior of new materials using artificial neural networks (ANNs) has become a frequently adopted solution by researchers today. The performance of neural networks depends mainly on the activation functions used. This work was designed to mainly study the impact of these functions on the response speed of an ANN in general, and particularly on the model we are working on to predict the thermomechanical behavior of innovative materials. By using tansig, purelin and logsig in a feed forward back propagation training by Levenberg-Marquardt algorithm, we were able to generate 9 models. For each of these models, we were interested in analyzing the speed’s response of the network and studying its regression. Thus, this work was able to show us that choosing the right neuron activation function from one layer to another can clearly influence the performance of the results. Depending on the problem studied, the desired objective and the chosen architecture, the activation function can radically change the result and provide us with the expected efficiency.


INTRODUCTION
Materials science has always been one of the flagship strands of scientific research because the use of matter is found in all areas of human life. It was able to follow and take advantage of the solutions offered by technology through the different eras. At the beginning the goal of the researchers was to discover the elements and materials existing on earth, to study them, to analyze them and to optimize their applications. More recently, their objective is to go beyond the existing and innovate new non-existent materials [1], model them, study them and analyze them to use them for very specific applications; robotics, aviation, automotive, food and others. The combination of two materials whose properties we already know can give us a new one that is more efficient than the other two in terms of characteristics.
At the same time, artificial intelligence (AI) that was designed to mimic human intelligence and go beyond human ability, reasoning and logic represents the future in every way. AI can be used in classification, optimization also in prediction and this in different fields; in medicine for example to predict  [2], in the field of quality to predict the results of an audit control [3] or in materials science, to predict the properties of innovative materials, bio composites, in our case. AI has already shown its ability to give satisfactory results in predicting the elasticity of a biocomposite to a certain degree of percentage of the mixture and this using artificial neural networks [4], [5]. The performance of an artificial neuron network (ANN) is linked to several parameters, the architecture of the model chooses, number of neurons in each layer, the weight of the bonds, but particularly it is linked the activation function of neurons. An activation function is defined as a mathematical function allowing the passage of information from one neuron to another [6]. In the same neural network, we can find more than an activation function because the latter can be different from one layer to another.
This work aims to study the response of the neural network of choice using three types of activation functions popularly used in AI, namely tansig, logsig and purelin. By using a single architecture with these three activation functions, 9 models will be generated and studied to analyze them in order to allow us to converge towards the fastest and most precise result, and therefore lead us to the most optimum model to predict the behavior of innovative materials.

Dataset and model building 2.1. Dataset
AI relies on the results of the latest research using different analytical techniques to arrive at the desired results. So, to train an ANN you need to have a dataset with the data as input and the desired goal as output as information. The objective of training a neural network is to find the relationship between the input and the target provided in the dataset, so that when asked to predict a result, it will be able to provide us with the desired information with a negligible degree of error [7]. Our dataset represents the results from the use of a semi-analytical method called Mori Tanaka [8]. It is made up of two inputs representing the elasticity of the two components of the new material, and the output representing its elasticity.

Architecture and training function
In this step we have to choose the architecture of the neural network to build the 9 models to study. This process consists in defining the topology of the network, therefore the choice of the number of hidden layers and the number of neurons in each layer. Then choose the algorithm to use to train the network.
In our study, the choice fell on a network of three layers as shown in Figure 1. An input layer, an output layer and a hidden layer. The number of neurons in the input layer is identical to the number of input variables of our dataset which is 2 as well as the number of neurons in the output layer is 1 representing the target of our datasast. Regarding the hidden layer, our choice fell on 15 neurons. This choice was not arbitrary, it was after several trials and studies that we made this choice [9]. So, our network topology is 2-15-1. Since this is a regression problem, we opted for the feed forward back propagation trained with Levenberg-Marquardt (ML) algorithm since the latter was able to prove its efficiency by combining it with the feed forward back propagation [10].

Activation functions
At the same time, the activation functions (AF) used in the hidden layer and in the output, layer is tansig, logsig and purelin [11]. These three functions allowed us to build 9 models to study. In every three models, we set the activation function of the hidden layer and we modify that of the output layer, and so on for the other two functions used. In Table 1 we find the representation of the models with the activation functions used in the two hidden layers and the output one.
− Logsig: log sigmoid transfer function. It is defined as, − Purelin: linear transfer function is typically used for function approximation and regression tasks. It is defined as,

RESULT AND DISSCUTION 3.1. Intelligent model's number 1, 2 and 3
This is a 2-15-1 topology feed forward back propagation trained with a ML. In these three models we used and fixed tansig as the activation function of the hidden layer [12], and we changed that of the output layer in each model. Table 2 shows us the responses of the network studied in relation to these activation functions. The training of the latter has been done in such a way that at each training epoch we save the trained model by performing a check to avoid overfitting [13]. Among these three models, the most suitable combination giving a faster result is that of the third model, followed by the combination of tansig with tansig. A representation of the evolution of the regression coefficient over time for the three models in question clearly shows us that the fastest model is also the one which converges quickly to 1 [14]. The regression coefficient is one of the important indicators proving the performance of an artificial neuron network [15], the closer this coefficient is to 1, the more the network is said to be efficient. Figure 2 illustrates the result clearly. And from Figure 2, we can deduce that when we are looking for regression coefficients close to 1 exhibiting more efficiency, the Tansig/Tansig function quickly leads to this result.

Intelligent model's number 4, 5 and 6
This is a 2-15-1 topology feed forward back propagation trained with an ML. In these three models we used and fixed logsig as the activation function of the hidden layer [16], and we changed that of the output layer in each model. Table 3 shows us the responses of the network studied in relation to these activation functions. The training of the latter has been done in such a way that at each training epoch we save the trained model by performing a check to avoid overfitting. Among these three models, the most suitable combination giving a faster result is that of the sixth model.  Like wise for these three models using logsig in the hidden layer, it was necessary to follow the evolution of the regression coefficient of each model as a function of time. The representation of the latter in Figure 3 shows us that the three models do not converge. This is due to the use of the logarithmic function in the hidden layer. This activation function was able to provide us with results, but it showed us that its use in the hidden layer will not be able to provide us with effective results.

Intelligent model's number 7, 8 and 9
This is a 2-15-1 topology feed forward back propagation trained with an ML. In these three models we used and fixed purelin [17] as the activation function of the hidden layer, and we changed that of the output layer in each model. Table 4 shows us the responses of the network studied in relation to these activation functions. The training of the latter has been done in such a way that at each training epoch we save the trained model by performing a check to avoid overfitting. Among these three models, the most suitable combination giving a faster result is that of the ninth model.

Fastest model validation
In this part, we will proceed to the validation of the fastest model, and this by asking it to predict the behavior of a new material. Polypropylene reinforced with 15% biofiller [18]. The ANN was trained by testing and error checking in such a way as to see the minimum squared error along with the maximum regression value. Using 1,000 repeated epochs of training, the regression produced by our model is shown in the Figure 5. The regression coefficient of polypropylene [19] reinforced at 15% with a choosen bio load is 0.99 and that converges to 1 and signified that our model is efficient and performant [20]- [25].

CONCLUSION
In this article we worked on an artificial neural network that meets the need to predict the properties of a new material, to study its response by playing on the activation functions of neurons. The activation function of a neuron is a very important parameter that influences the results. Choosing the right activation function clearly leads to effective results. The aim was to study the behavior of a feed forward back propagation which consists of three layers. Using tansig, purelin and logsig in activating neurons from both hidden and output layers, we were able to build 9 models that we studied. The speed of response as well as the regression of each of this model was our goal. We were able to show that in the prediction of the characteristics of biocomposites, the use of the tansig/tansig function allowed us to have a good regression coefficient close to 1 more quickly than using the other combinations of the functions of activation. The activation function remains a mechanism by which neurons process and transmit information through the different layers. For the tansig, the processing and the transmission are done in a wide interval which allows a fast learning to converge towards a better regression.