The Experts Method for the prediction of big data streams of energy flow
We have developed a method, called Experts Method, to forecast the evolution of a multivariate set of time series, of big dimension, and with partially censored data. It has been applied to the data provided by the H2020 Big Data Horizon Prize 2017 (http://ec.europa.eu/research/horizonprize/index.cfm?prize=bigdata).
The data subject to the forecast are energy flow-related measurements over N=1912 high-tension lines registered over two years (called target data). The lines have been anonymized with respect to location, thus they are provided as purely temporal data. The energy flows are registered every 5 minutes for one year, but since the data recording can be switched off for some (random) period of time, some of the time series show missing data. The data are divided into subsequent files, where each column of the files contains the observations of one high-tension line in the time period [t,t+dt]. The length dt of the periods of observation can be different from file to file (it anyway is always bigger or equal than 1 hour). This setting emulates the fact that in real applications data are recorded continuously by sensors, but they can be passed to the statistical analysis during different times, either regularly or when the measurements overcome a fixed threshold, or some extra control is planned and an immediate forecast of the future behaviour is needed.
The Experts Method is thus suitable to predict the evolution of a multivariate set of data streams of big dimension, where the observed streams are given during different random times.
The proposed method is supervised, thus the target data have been divided into a training set, to set up the method, and an adapt set, on which the method is tested. Anyway, differently from deep learning methods based on neural networks, our method does not need a huge training set to be trained.
The advantage of this method, if compared with classical multivariate time series analysis, is that it can be applied also when the time series column order is reshuffled, from time to time, in the collected dataset.
The method is based on the definition of a set of “experts”, which are portions of the training set of the considered time series which best fit (using an L2 norm in a Sobolev space) the data immediately preceding those to be predicted. A suitable combination of Singular Value Decompositions is used to filter out the noise, and provide robust predictions.
For further details see:
- Aletti, M. Bellan, A. Micheletti, The Experts Method for the prediction of periodic multivariate time series of high dimension. In Smart Statistics for Smart Applications : Book of short papers SIS2019, G. Arbia, S. Peluso, A. Pini, G. Rivellini, Pearson, 2019, pp. 643-648