Two Phase Multivariate to Multivariate Time Series Forecasting Using Self-attention Convolutional Autoencoder and Temporal Convolutional Network

2.1 Convolutional Autoencoder

CAE is a variant of neural network aimed at reconstruction and feature extraction. Convolutional Neural Network (CNN) can dynamically generate key features from input images due to its local receptive field, weights sharing and pooling characteristics (Zhao et al., 2017). Autoencoder (AE) can transform feature vectors into abstract feature vectors that can learn inter-relationship from high dimensional data space to low dimensional data space (Yan and Han, 2018). In a previous study, Zheng et al. (2014) proposed feature learning technique for MTS data by separating MTS into UTS and using deep CNN to derive good classification results. This technique, however, has a major limitation that inter-relationship between UTS cannot be incorporated. To overcome this shortcoming, inspired by CNN and AE for their feature extraction characteristics, we explore these frameworks for MMTS forecasting. An aggregate of two models, CNN and AE, CAE extracts representative features of the local map of the given input. Different from the previous study, instead of separating MTS into UTS, we jointly use MTS for feature extraction. With weight sharing structural characteristics of CNN, CAE has fewer training parameters, higher training efficiency, and avoids over-fitting (Wu et al., 2021).

In the proposed method, CAE is used to extract non-linear relationships of multiple variables of MTS data. Adding local convolutions with AE, CAE can encode the original input by extracting the local features of the input into a feature map. Then convolutional decoder decodes the extracted feature map into the original output. Due to the local feature capturing characteristics, multiple filters are used in the same convolutional layer of CAE to extract diverse features. To elaborate, MTS inter-relationship extraction is illustrated in <Figure 1>.

Figure 1.

Architecture for MTS Inter-relationship Feature Extraction

The universal convolutional operation of CAE can be expressed as Equation (2):

W_c and b_c are the weight parameters. h(ㆍ) is the activation function and * denotes dot product of the convolutional layer and the pixels. The c-th filter scans the input matrix Xⁱ and produces Z_c, output of the convolutional operation.

After the convolutional operation, an activation function of Parametric Rectified Linear Unit (PReLU) is utilized. PReLU is an activation function that generalizes the traditional rectified unit with a slope for negative values. PReLU enables different types of nonlinearities for different layers and is expressed as Equation (3):

$$ T_c=W_c^{\text{*}}{X}^i+b_c$$ is the c-th channel input to the activation function h(ㆍ) and p_c is the coefficient for the control of negative part of the activation function. If p_c becomes 0, the activation function becomes ReLU. When p_c is a learnable parameter, Equation (3) is PReLU (He et al., 2015).

2.2 Self-attention

CNN and its local convolutional kernel limit the convolutional operation to capture only local information of the kernel (Wu et al., 2021). <Figure 2> represents the convolutional kernel of CNN. If the kernel size becomes 1 x 3, the result is only dependent on the three values of the receptive field. To secure a wider information, convolutions are stacked several times, or the size of the data is reduced through the pooling layer. However, widening the receptive field is not efficient as the depth of the network must become deep enough to secure a sufficient size of the receptive fields. This method is not a sufficient solve to the non-local dependency problem.

Figure 2.

Figure of CNN Convolutional Kernel

Wang et al.(2018) proposed a non-local neural network, a type of self-attention module, that is efficient and can be well integrated into the existing deep learning architectures. By implementing the self-attention operations, the proposed model can capture the non-local dependencies by only the interactions between any two positions. Furthermore, analyzing the attention map created from the self-attention module enables cause analysis after MMTS forecasting. The self-attention module for the proposed method is depicted in <Figure 3>.

Figure 3.

Self-attention in the Proposed Method, 1×1 Convolution is Applied to the Input to Create Query, Key, and Value. The Final Output is Added to the Input, Similar to the Residual Connection

The residual style of the self-attention module output is expressed as Equation (4), shown below.

The output feature map of the self-attention O_c∈R^1×W×C is added to the original input Z_c∈R^1×W×C. 1, W, and C represent the height, the width, and the channel of the output of the convolutional operation and the output feature map. The residual style of the attention feature enables self-attention to be inserted in the last and the first layer of the convolutional layers of the encoder and the decoder of CAE. Self-attention module is calculated by the following Equation (5):

z_i and z_j are feature maps of a certain convolutional layer and k(ㆍ), and q(ㆍ) are 1x1 convolutions. T transposes the variable. Applying 1×1 convolutions on the feature maps reduce the number of channels and computation cost (Wu et al., 2021). The attention weights are calculated by normalizing the output score of Equation (5). The attention weights are calculated by following Equation (6):

Next, the attention weight is calculated with the input feature map to produce the attention applied output feature map O_c. Equation of the output feature map is presented below in Equation (7):

v(ㆍ) is 1×1 convolution.

Considering Equation (5) is computed based on the relationships between feature maps of the convolutional layer, the self-attention module can mitigate local dependency issues.

2.3 Temporal Convolutional Network

TCN is a variant of convolutional architectures for sequential data, which includes causal convolutions and dilated convolutions. In a previous study such as Bai et al. (2018), TCN architecture was experimented against traditional recurrent architectures across comprehensive sequence modeling tasks. In the previous study it was concluded that TCN outperforms generic LSTMs and GRUs.

For MMTS forecasting it is required to reproduce the original MTS input from the condensed UTS representation. Thus, an accurate forecast ability of the condensed UTS representation must be a foreground operation. However, LSTMs and GRUs exhibit vanishing/exploding gradient problems, which we deemed inappropriate for accurate forecasting of the condensed UTS representation.

TCN lists several advantages compared to these models. One of the major advantages of TCN is its ability to stabilize gradients. TCN can avoid vanishing/exploding gradients as it has a backpropagation path different from the temporal direction of the sequence (Bai et al., 2018).

Before explaining the model structure of TCN, we define the nature of sequence modeling task of the condensed UTS representation, where the condensed UTS representation $$ X^{\text{'}}=\left\{x_0^{\text{'}},\cdots ,x_t^{\text{'}}\right\}$$ is trained to predict the expected output of $$ x_{t+1}^{\text{'}},\dots ,x_{t+a}^{\text{'}}$$. a is the range of future time steps. The core of the sequence prediction lies in relying only on the given past input sequence of $$ x_0^{\text{'}},\dots ,x_t^{\text{'}}$$ and not on the future input sequence of $$ x_{t+1}^{\text{'}},\dots ,x_{t+a}^{\text{'}}$$ to predict the output $$ x_{t+a}^{\text{'}}$$. The general function is as Equation (8):

r(ㆍ) is TCN model.

Causal convolutions satisfy the aforementioned point where the nature of time series data, the future output at time t + a must satisfy the ‘causality’ of relying on only the present and the past, not the future data points. TCN uses 1-D convolution with zero-padding with ‘kernel size -1’ to achieve convolution only with elements from time t and earlier in the previous layer (Bai et al., 2018). However, 1-D convolutions and causal convolutions require many layers and large filters to capture distant time events. To solve the inefficiency of requiring many layers and large filters to capture the distant time events, TCN utilizes dilated convolutions. Advanced by the wavenet, dilated convolutions increase the receptive field by the order of magnitude with low computation cost (Oord et al., 2016). Dilation filter size is as Equation (9):

Dilation size of layer l is d_l, filter size is p, and dilation rate is d. Inserting zeros in the convolutional kernels, convolution is applied with d - 1 skipped inputs. The architecture of TCN, causal convolution and dilated convolution, is shown in <Figure 4>. By stacking several dilated convolution layers, TCN can achieve larger receptive fields with fewer layers compared to original convolution operations.

Figure 4.

Basic Structural Explanation of TCN

3.1 Phase Ⅰ

The proposed method utilizes SACAE to capture inter-relationship, the non-linear correlation existent in the input MTS data. SACAE encoder is consisted of multiple 1-D convolutional blocks and self-attention modules. Flatten layer is added in the final operation to flatten the contiguous range of dimensions into simple vector output.

MTS data is inputted into SACAE as channels while each 1-D convolution operation has kernel size of three, stride size of one, and padding size of one. By doing so, SACAE can preserve the overall length of the subsequent outputs after each convolutional block operations. The output which is the same length as the input enables TCN in Phase Ⅱ to forecast the next time step. This will be further discussed in the next section.

As data is passed along each convolutional block of SACAE encoder, the number of channels is exponentially decreased to extract abundant features of the original MTS input data. Each convolutional blocks of SACAE encoder decreases the number of channels of its input. Created channels then extract abundant information between individual variables of input MTS data. For example, data created from the same machinery of a manufacturing process. While CAE is able to capture the non-linear correlations of MTS data in the local field, the non-local dependency issue makes it difficult to extract the global dependencies. To mitigate such problem, self-attention is added to the last and the first layers of CAE. Now, SACAE calculates the importance of elements in the feature map and captures the global dependencies. We include self-attention in the deeper layers of convolutional blocks to capture global information as shallow convolutional layers are prone to capture local features of the input.

3.2 Phase Ⅱ

TCN is composed of causal convolutions and dilated convolutions. Causal convolutions extract relevant historical information and forecasts the future output while dilated convolutions facilitates computational efficiency. In a previous study, TCN has been proven to outperform baseline architectures of sequence modeling tasks (Bai et al., 2018). In the proposed method, a baseline TCN with few changes has been used. The role of TCN is to forecast the next time steps of the encoded UTS representation of the original MTS data. The module predicts the next time steps of the encoded data based on the multiple temporal convolutional layers. TCN is composed of several temporal block layers with number of hidden channels in each layer either 25 or 50. The structure of TCN is presented in <Figure 3>. We increase the dilation rate d exponentially with the rate of 2ⁱ. Thus, according to the number of layers, the filter may cover the whole input sequence.

SACAE decoder reconstructs the forecasted time steps of the UTS representation data back to the original input data. Through analysis of self-attention block of the decoder, cause analysis of the effects of the input data to the forecasted outputs is possible. SACAE decoder is similar to the structure of SACAE encoder. Multiple convolution blocks and self-attention layers are embedded. While the overall structure of both encoder and decoder is similar, SACAE decoder utilizes transposed 1-D convolutions.

4.1 Datasets

The proposed method was verified on three real world datasets, namely CNC (Computer Numerical Control) (Sun, 2018), C-MAPPS (Turbofan Engine Degradation Simulation) (Saxena et al., 2008), and PMM (Predictive Maintenance Modelling dataset) (Zonta, 2020). CNC is milling machine experiment dataset in the System-level Manufacturing and Automation Research Testbed (SMART) at the University of Michigan. C-MAPPS is engine degradation simulation dataset collected by NASA. PMM is telemetry reading and error identification, maintenance, and failure dataset created by Microsoft. Data was collected from a CNC machine for variations of tool condition, feed rate, and clamping pressure. The data was collected from 18 experiments with 100m/s sampling rate from four motors in the CNC milling machine. The proposed method was experimented on six variables of motor x and y as variables from motor z and spindle showed no significant variance in data. The data from C-MAPPS is collected from simulation of commercial turbofan engine data. Sampling was done at 1 Hz and consist of 30 engine and flight condition parameters. The proposed method was experimented with FD002_RUL data on fourteen parameters with 249 train time length and 20 test time length. PMM data was constructed for predicting problems caused by component failures. The data consist of eight variables including error identification time points and component maintenance time points. Experiment was conducted on 234 train length and 20 test length data.

<Figure 6> illustrates the relationships between six variables of CNC dataset through a pair grid plot. The six variables exhibit non-linear correlation between each other. Variables such as x1_currentfeedback, x1_outputvoltage, and y1_outputvoltage exhibit clearer non-linear correlation with other variables in the dataset.

Figure 6.

Non-linear Correlation Pair Grid of Six Variables in the Dataset and an Example of Explicit Non-linear Correlation of Variables x1_outputvoltage and y1_commandvelocity, Elaborated in <Figure 6>

To illustrate that x1_outputvoltage and y1_commandvelocity show clear non-linear correlation, we fitted a fourth order polynomial function as shown in <Figure 7> (Left) and witnessed that two variables do indeed inherit a non-linear correlation. However, fitting a linear regression to two variables in <Figure 7> (Right) do not properly represent the relationship between two variables. Drawing a conclusion that two variables inherit a complex non-linear correlation.

Figure 7.

Fitted Image of Fourth order Polynomial Regression of variables x1_outputvoltage and y1_commandvelocity (Left) illustrates a clear non-linear correlation between two variables while linear regression (Right) does not properly represent the relationship between two variables

4.2 MMTS Forecast

In this study, we compare the proposed method with baseline sequence modeling benchmarks. Statistical, machine learning, and deep learning methods are compared with the proposed method. Models used to compare are ARIMA, SVR, and LSTM, baseline sequence models well known in each method. The results suggest that the proposed method outperforms the baseline sequence modeling architectures in capturing the high dimensional relationships of the variables.

Prediction accuracy was determined using MMAE, Equation (10), MMSE, Equation (11), and MR², Equation (12). These evaluation metrics are commonly used performance metrics for time series forecasting and are modified for MMTS forecasting.

n is the number of samples in the test set and m is the number of variables in the test set, $$ {\overline{y}}^i$$ is the mean yⁱ of values, $$ y_t^i$$ is the actual output variable for i^th variable at time t and $$ {\widehat{y}}_t^i$$ is the predicted output for i^th variable at time t.

Shown in <Table 1>, the proposed method shows superior forecast capability and generally achieves better performance compared to other forecast baseline models. As expected, the baseline deep learning model performs similar to the baseline statistical model. We expect that the performance difference will exponentially increase with the increase in volume of data.

Table 1.

Forecasting Performance of Baseline Models and the Proposed Method

The proposed method exhibits strong performance improvement in C-MAPPS dataset. We conclude this is because of the ability of the proposed method to effectively learn the abundant non-linear correlations existent in C-MAPPS dataset. <Figure 8> illustrates the overall correlation between variables of C-MAPPS and PMM datasets. C-MAPPS with fourteen variables, except for sensor 15, show relatively high correlation values to other variables. On the contrary, PMM datasets, with about half the number of variables to C-MAPPS, do not show clearer correlations within the dataset.

Figure 8.

Pearson Correlation Plot of C-MAPPS (left) and PMM (right) Dataset

The major advantage of the proposed method is its ability to learn inter-relationships, the non-linear relationship inherent in high-dimensional MTS data.

We now discuss the results of each model on forecasting individual variables to compare the ability of the proposed method to reflect the non-linear correlations. These results are presented in <Table 2>.

Table 2.

Individual Mean Squared Error (MSE) Score on CNC Dataset Variables to Evaluate Models on Reflecting Non-linear Correlation

Prediction accuracy was determined by using mean square error (MSE), Equation (13). Compared to MMSE evaluation metric, MSE measures the prediction loss of individual variables in CNC dataset.

n is the number of samples in the test set, y_t is the actual output variable and $$ {\widehat{y}}_t$$ is the predicted output variable at time t.

<Table 2> represents MSE scores of the individual variables of CNC dataset forecast results of models. As shown in <Figure 6>, three variables x1_currentfeedback, x1_outputvoltage, and y1_outputvoltage of CNC dataset show strong explicit non-linear correlations with other variables. As an example, variables x1_outputvoltage and y1_commandvelocity has a complex correlation of a combination of linearity and non-linearity. Accordingly, the proposed method shows lowest MSE scores on all corresponding three variables. Also, shows the lowest MSE score on variable x1_currentfeedback, which has the clearest non-linear correlations amongst the six variables. The result of this experiment and the result of C-MAPPS dataset can conclude that the proposed method has strength in learning the non-linear correlations in MTS data, whereas other baseline models do not.

4.3 Cause Analysis

Besides forecasting MMTS, the proposed method also provides insights for cause analysis. For domain such as Prognostics and Health Management (PHM) in manufacturing, cause analysis is equally as important as forecasting future values. In real world applications, cause analysis of MTS can aid engineers to discover root cause of accidents and save time and effort.

The self-attention map of SACAE decoder is used for cause analysis. The self-attention map mitigates the effect of long-term dependency problem and creates weights that help a model focus on important time steps in input data. It takes i number of inputs and outputs i number of outputs. The inputs will interact with each other to decide which time point is important and needs more attention. In our case, the self-attention focuses on time points that are crucial in forecasting the last 20 degradation points until failure of PMM dataset.

As shown in <Figure 9>, PMM dataset contains error identification and maintenance time points. These time points indicate the points in input data where error has been identified and the points where major component maintenances have been required. There are fifteen error identification time points and nine component maintenance time points. <Figure 9> shows visualization of original input data of PMM dataset with error identification, maintenance time points, and predicted time points.

Figure 9.

Visualization of PMM Original Dataset with Error Identification (red), Maintenance Time Points (blue) and Predicted 20 Time Points (orange)

The proposed method’s diagnostic performance on PMM dataset is demonstrated on <Figure 10>. The self-attention module in SACAE decoder is used for cause analysis map. The cause analysis map indicates the time points the proposed model references strongly to forecast the last degradation time points leading up to the major failure time point. The row of cause analysis map indicates the last 20 degradation time points, and the column indicates the input time points from start to up to 234 time points. The darker columns of the cause analysis map indicate time points the self-attention module judged is important for forecasting the last 20 degradation time points.

Figure 10.

Original Cause Analysis and Zoom-in Cause Analysis Map of PMM dataset. Zoom-in Cause Analysis Map shows 85 to 110 time points of the Original Cause Analysis Map. The red and blue dotted lines indicate major events in the machine history. The cause analysis map of SACAE decoder focuses on these events or time points leading up to these events

As shown in <Figure 10>, the cause analysis map and the zoom-in version demonstrate the ability to locate crucial points in the time map of the input data. The red dotted lines indicate the error identification points, and the blue dotted lines indicate the maintenance time points of the input data. The maintenance time points are considered more crucial time points than the error identification time points as major defect has occurred which led to actual maintenance actions. The zoom-in version shows the self-attention map from 85 to 110 time points. All 20 prediction times focus strongly on time points 89 to 91, time points 93 to 95, and prediction time points 16 and 17 also focus on 103 to 105 time points. This indicates the model captures important time points of the input data to forecast the last degradation time points of PMM dataset. The cause analysis map has captured most major events of both error identification and maintenance time points. Especially, 7 out of 9 maintenance time points have been captured by the cause analysis map.