CC BY
11
СРАВНИТЕЛЬНЫЙ АНАЛИЗ МОДЕЛЕЙ ГЛУБОКОГО ОБУЧЕНИЯ ДЛЯ ПРОГНОЗИРОВАНИЯ ВРЕМЕННЫХ РЯДОВ НА ДАННЫХ КРИПТОВАЛЮТЫ
SOLANA С ИСПОЛЬЗОВАНИЕМ DARTS
Ал-Хаидари Хазим Хамид Абду1, магистрант Аль-Шаибани Ескандер Тахер Саиф2, магистрант Аль-мактари Мохаммед Ахмед Саиф Хайдар2, магистрант Аль-Хаити Абдулвасеа Нассер Хамид Мубиль2, магистрант ^Университет науки и технологий МИСИС ^Российский университет дружбы народов (Россия, г. Москва)
DOI:10.24412/2500-1000-2024-9-2-77-86
Аннотация. В статье рассматривается сравнительный анализ нескольких моделей глубокого обучения для прогнозирования временных рядов на данных криптовалюты Solana с использованием библиотеки Darts. Исследование оценивает точность прогнозирования шести моделей: Block RNN, N-BEATS, N-HiTS, RNN, TCN и TFT, используя как эмпирический, так и количественный подход. Модель Block RNN продемонстрировала наилучшую общую точность, достигнув наименьших показателей ошибок, за ней следовали N-BEATS и TCN. Модели N-HiTS и TFT оказались менее точными из-за их высокой сложности и относительно небольшого объема данных. Однако дальнейшее обучение модели N-BEATS привело к значительным улучшениям, продемонстрировав её потенциал в улавливании долгосрочных трендов на волатильных криптовалютных рынках. Это исследование предоставляет ценные рекомендации по выбору моделей глубокого обучения для прогнозирования в таких динамичных условиях.
Ключевые слова: глубокое обучение, временные ряды, прогнозирование, криптовалюта Solana, Darts, модели RNN, N-BEATS, волатильные рынки.
Cryptocurrencies, like Solana, have gained massive popularity in recent years. Unlike traditional currencies, cryptocurrencies are digital and decentralized, meaning they are not controlled by any government or central authority. While this offers many benefits, it also makes the prices of cryptocurrencies highly volatile, with sudden rises and falls being common. This volatility poses a challenge when it comes to predicting future prices or trends.
In the field of data science, time series forecasting is a technique used to predict future values based on past data. In the case of cryptocurrencies, forecasting can help traders and analysts make more informed decisions. However, due to the unpredictability of the crypto market, choosing the right model for forecasting becomes critical.
Deep learning models have shown great promise in improving time series forecasting, especially with complex and volatile data like cryptocurrency prices. In this study, we use
the Darts library, a powerful tool for time series forecasting, to compare several deep learning models on Solana cryptocurrency data, including Block RNN, NBEATS, NHiTS, RNN, TCN, and TFT models [1]. The goal is not to predict future prices but to determine which model can best respond to the sharp trends and fluctuations in the crypto market.
1. Data Collection and Preprocessing
The data for Solana (SOL) was collected using the Yahoo Finance API [2] via the yfinance library, which provided historical price data with an hourly frequency over a two-year period. This dataset included various metrics, but for this analysis, the focus was on the closing price of Solana. The hourly frequency of the data was chosen to capture the high volatility characteristic of cryptocurren-cy markets, providing a detailed view of price changes over time.
In the preprocessing phase, the collected data was cleaned and prepared for analysis.
- TexHuuecKue HayKU -
Missing values in the hourly time series were handled using the forward-filling technique, ensuring a continuous dataset with no gaps. The data was then organized into two main columns: datetime, representing the date and time of each observation, and SOL, representing the closing price. This structured dataset was essential for building accurate forecasting models.
The prepared data was split into training and testing sets, with 90% allocated for training and 10% for testing [3]. To enhance the forecasting models, covariates were introduced to provide additional context for the time series. These covariates included temporal attributes such as month, year, day, day
of the week, week, and day of the year, normalized to a range between 0 and 1. This normalization was achieved by dividing each attribute by its respective maximum value or typical range, ensuring that all covariates were on a comparable scale and facilitating the model's ability to learn from both past and future temporal patterns.
The graph below visualizes the Solana price data over time, highlighting the volatility observed in the dataset. The sharp fluctuations in the price indicate the high volatility typical of cryptocurrency markets, which poses a significant challenge for forecasting models.
Pic. 1. SOL Price Over Time
2. Evaluation Methods
In evaluating the performance of the forecasting models, an empirical approach was initially used to analyze the visual outputs of the models. This involved inspecting the graphs to qualitatively assess how well each model captured the trends and fluctuations in the Solana price data. However, to provide a
more objective assessment, several quantitative metrics [4] were employed to measure the accuracy and effectiveness of the models.
First is the Mean Squared Error (MSE) which measures the average squared difference between the actual and predicted values. A lower MSE indicates better performance.
n
MSE=1\ (y£ - 3>i)2 nZ_i ¿=1
Second metric is the Mean Absolute Percentage Error (MAPE) which represents the
average percentage difference between the actual and predicted values, providing an in-
tuitive interpretation of the error.
MAPE =
100%
n
Next is the symmetric Mean Absolute Percentage Error (SMAPE) which is a modified
I
¿=1
Wi - Vi
yi
version of MAPE that accounts for both actual and predicted values symmetrically.
SMAPE =
100%
n
I
¿=1
(M + I&D/2
These metrics along with the empirical approach were used to compare the effectiveness of each model in predicting the future prices of Solana cryptocurrency.
3. Models Training and Evaluation
The Block RNN model is a type of recurrent neural network designed to handle sequential data by capturing temporal dependencies through its recurrent structure. The core idea of Block RNN is to use recurrent layers to process input sequences, allowing the model to learn and remember patterns over time. This model employs a series of recurrent blocks that process the data in a step-by-step manner, updating its internal state based on previous time steps. By leveraging this architecture, Block RNN aims to predict future values in a time series based on historical information.
The performance of the Block RNN model was evaluated using three key metrics. The Mean Squared Error (MSE) was 121.51, indicating the average squared difference between the actual and predicted values. The Mean Absolute Percentage Error (MAPE) was 6.91%, reflecting the average percentage error, while the Symmetric Mean Absolute Percentage Error (SMAPE) was 7.19%, providing a balanced measure of accuracy. The graph below illustrates the model's predictions. It shows that the Block RNN model's forecast is largely horizontal with some noisy variations, failing to capture the underlying trends of the data effectively. This pattern suggests that the model may not be adequately learning the temporal dynamics of the Solana price series, resulting in suboptimal performance in trend prediction.
Pic. 2. Block RNN Model Predictions vs. Actual Solana Prices
- TexHuuecKue HayKU -
The N-BEATS model is a deep learning architecture designed specifically for time series forecasting [5]. Unlike traditional models, N-BEATS does not rely on recurrent or convolutional structures but instead uses a stack of fully connected feed-forward neural networks to capture temporal dependencies. The model operates by learning patterns from historical data through a series of blocks, each contributing to the overall forecast. N-BEATS employs a unique approach by generating forecasts directly from the input time series, aiming to capture complex patterns without the need for complex recurrent or convolu-tional layers.
The performance of the N-BEATS model was assessed using several metrics. The Mean Squared Error (MSE) was 127.92, reflecting
the average squared error between the actual and predicted values. The Mean Absolute Percentage Error (MAPE) was 7.03%, indicating the average percentage deviation of the predictions from the actual values. The Symmetric Mean Absolute Percentage Error (SMAPE) was 7.32%, providing a balanced view of the prediction accuracy. The graph below demonstrates that, unlike the Block RNN model, N-BEATS shows some ability to align with the actual price movements, with moments where both the actual prices and predictions rise and fall together. However, while this indicates an improvement, the model still struggles to fully capture the trends and fluctuations in the Solana price data, suggesting room for further enhancement in trend prediction accuracy.
— actual
forecast
27 29 31 Sep 0(3 06
datetime 2024-Sep
Pic. 3. N-BEATS Model Predictions vs. Actual Solana Prices
The N-HiTS model is an advanced time series forecasting model that extends the idea of neural network architectures by combining hierarchical attention mechanisms with neural network layers [6]. It aims to capture complex temporal dependencies and patterns by leveraging hierarchical structures to process different levels of data granularity. The model uses attention mechanisms to focus on relevant
parts of the time series, allowing it to make more informed predictions based on historical patterns.
The performance of the N-HiTS model was evaluated using several metrics. The Mean Squared Error (MSE) was 528.30, indicating a higher average squared error between the actual and predicted values compared to other models. The Mean Absolute Percentage
Error (MAPE) was 12.43%, reflecting a significant average percentage deviation from the actual values. The Symmetric Mean Absolute Percentage Error (SMAPE) was 13.49%, further highlighting the model's accuracy issues. The graph below shows that the N-HiTS model's predictions are notably poor, with
170
results that are not even close to the actual values. Unlike the Block RNN and N-BEATS models, the predictions of N-HiTS fail to capture any meaningful trends, resulting in a performance that is considerably worse and highlighting the need for further improvements in its forecasting capabilities.
Pic. 4. N-HiTS Model Predictions vs. Actual Solana Prices
The RNN (Recurrent Neural Network) model is designed to handle sequential data by maintaining an internal state that evolves over time. This architecture allows the model to capture temporal dependencies and learn patterns from historical data sequences. RNNs process data step-by-step, updating their internal memory based on previous inputs, which helps them make predictions about future values by leveraging learned temporal patterns.
The RNN model's performance was assessed using three key metrics. The Mean Squared Error (MSE) was 259.66, indicating the average squared deviation between the
actual and predicted values. The Mean Absolute Percentage Error (MAPE) was 9.38%, showing the average percentage error in the predictions. The Symmetric Mean Absolute Percentage Error (SMAPE) was 9.96%, providing a balanced measure of accuracy. The graph below demonstrates that while the RNN model's predictions exhibit minimal noise, they still fail to align closely with the actual values. This suggests that while the model produces stable predictions, it struggles to accurately capture the underlying trends and fluctuations in the Solana price data, resulting in less effective forecasting.
160
155
150 -
145 -
140 -
135 -
130 -
125 -
лЛ - actual forecast
——---- - forecast
27
29
31 Sep
datetime
03
05
2024-Sep
Pic. 5. RNN Model Predictions vs. Actual Solana Prices
The Temporal Convolutional Network (TCN) model is a deep learning architecture that leverages convolutional layers to process sequential data for time series forecasting [7]. Unlike recurrent models, TCNs use dilated convolutions to capture dependencies across various time scales efficiently. This design allows the model to learn complex temporal patterns by applying convolutional operations over the entire sequence, aiming to improve forecasting accuracy and capture long-range dependencies within the data.
The performance of the TCN model was evaluated using key metrics. The Mean Squared Error (MSE) was 152.29, indicating the average squared deviation between the
actual and predicted values. The Mean Absolute Percentage Error (MAPE) was 7.82%, reflecting the average percentage error in the predictions. The Symmetric Mean Absolute Percentage Error (SMAPE) was 8.12%, providing a balanced measure of accuracy. The graph below demonstrates that while the TCN model produces predictions with relatively low error metrics, it fails to follow the trends of the actual data effectively. The predictions exhibit a pattern that does not align well with the actual price movements, indicating that the model struggles to capture the underlying temporal dynamics of the Solana price series despite its advanced architecture.
160 -
155 -
150 -
145 -
140 -
135 -
130 -
— actual
forecast
27
29
31 Sep
datetime
03
OS
20 2 4-Sep
Pic. 6. TCN Model Predictions vs. Actual Solana Prices
The Temporal Fusion Transformer (TFT) model is a sophisticated time series forecasting architecture that combines attention mechanisms with temporal convolutions to handle complex temporal patterns and longrange dependencies [8]. The TFT model uses attention layers to focus on relevant parts of the sequence and fusion mechanisms to integrate different types of input data, such as historical values and covariates. This approach aims to enhance the model's ability to capture intricate patterns and make accurate predictions based on both past data and future inputs.
The performance of the TFT model was evaluated using several metrics, revealing the poorest results among the models tested. The Mean Squared Error (MSE) was 654.07, re-
flecting the highest average squared deviation between the actual and predicted values. The Mean Absolute Percentage Error (MAPE) was 12.45%, indicating a significant average percentage error in the predictions. The Symmetric Mean Absolute Percentage Error (SMAPE) was 13.62%, providing a balanced view of prediction accuracy. The graph below illustrates that the TFT model's predictions not only fail to follow the actual trends but also show a sudden drop in performance. The model's forecasts are notably inaccurate, with substantial deviations from the actual price movements, highlighting its difficulties in capturing the temporal dynamics of the Solana price data and its overall poor performance in trend prediction.
Pic. 7. TFT Model Predictions vs. Actual Solana Prices 4. Results Discussion
The results of the models were compared based on the performance metrics, as summarized in the table below.
Table 1.
Model MSE MAPE SMAPE
Block RNN 121.51 6.91% 7.19%
N-BEATS 127.92 7.03% 7.32%
NHiTS 528.3 12.43% 13.49%
RNN 259.66 9.38% 9.96%
TCN 152.29 7.82% 8.12%
TFT 654.07 12.45% 13.62%
From the table, it is evident that the Block RNN model consistently produced the best results, outperforming all other models in terms of MSE, MAPE, and SMAPE. N-BEATS and TCN followed closely, but their accuracy was slightly lower. The N-HiTS and TFT models, however, performed poorly, likely due to their higher complexity and the relatively small dataset. Despite this, our empirical analysis suggests that N-BEATS has more potential, as it better captures underlying trends. Therefore, we have decided to continue training it further to optimize its performance.
5. Additional Training of N-BEATS Model
After conducting additional training on the N-BEATS model, we observed improvements in its performance metrics, as indicated by the results: a Mean Squared Error (MSE) of 37.33, a Mean Absolute Percentage Error (MAPE) of 3.33%, and a Symmetric MAPE (SMAPE) of 3.41%. These values demonstrate that the additional training has slightly enhanced the model's accuracy, making it more capable of capturing general trends in the data. The forecast plot reflects this improvement, showing that the model effectively tracks the overall downward movement of the actual data.
Pic. 8. N-BEATS Model Predictions (with further training) vs. Actual Solana Prices
Despite these advancements, the N-BEATS model still struggles to fully capture sharp fluctuations, especially during periods of high volatility. This is evident in moments where the forecast deviates from the actual data, particularly around significant price drops or sudden spikes. These challenges are common in machine learning models applied to financial time series, particularly when the dataset is relatively small. However, the slight reduction in MAPE and SMAPE suggests that the model's generalization has improved, resulting in more consistent and reliable predictions.
The post-training results confirm the potential of the N-BEATS model for further applications. While the improvements are incremental, they provide a foundation for future refinement. Possible next steps include fine-tuning hyperparameters such as learning rates, hidden layers, and block sizes, which may further enhance the model's accuracy. Additionally, incorporating more external features such as volume data, macroeconomic indicators, or technical analysis signals could help the model better anticipate market shifts and improve its performance during periods of volatility.
Conclusion
In this comparative analysis of deep learning models for time series forecasting on Solana cryptocurrency data, the Block RNN model emerged as the top performer across all key metrics, including MSE, MAPE, and SMAPE. N-BEATS and TCN followed closely, offering comparable results with slightly higher error rates. On the other hand, the N-HiTS and TFT models underperformed, likely due to their greater complexity and the relatively small dataset, which limited their ability to generalize effectively.
Despite the overall performance of the Block RNN model, additional training of the N-BEATS model proved to be a valuable exercise. Post-training results demonstrated noticeable improvements, reducing the model's MSE to 37.33, MAPE to 3.33%, and SMAPE to 3.41%. These enhancements indicate that N-BEATS is better suited for capturing longer-term trends, although it continues to struggle with high-volatility periods.
The findings suggest that, while simpler models like Block RNN can provide solid baseline performance in cryptocurrency forecasting, more complex models like N-BEATS have the potential to deliver superior results with further refinement. Future research should focus on fine-tuning hyperparameters,
incorporating additional covariates, and ex- casting potential of deep learning models in ploring larger datasets to maximize the fore- highly volatile markets like cryptocurrencies.
References
1. Papers with Code. Time Series Forecasting. Electronic resource. URL: https://paperswithcode.com/task/time-series-forecasting. Date of access: 23.08.2024.
2. Yahoo Finance API. Electronic resource. URL: https://developer.yahoo.com/api. Date of access: 23.08.2024.
3. David G. L. Medium. Five Methods for Data Splitting in Machine Learning. Electronic resource. Date of access: 23.08.2024.
4. Bruce P., Bruce A. Practical Statistics for Data Scientists. Released May 2017. Publisher: O'Reilly Media, Inc. ISBN: 9781491952962.
5. Dg Oreshkin B.N., Carpov D., Chapados N., Bengio Y. N-BEATS: Neural Basis Expansion Analysis for Interpretable Time Series Forecasting. ICLR 2020 Conference Blind Submission. Published: 20 Dec 2019, Last Modified: 03 Apr 2024.
6. Challu C., Olivares K.G., Oreshkin B.N., Garza F., Mergenthaler-Canseco M., Du-brawski A. N-HiTS: Neural Hierarchical Interpolation for Time Series Forecasting. Submitted on 30 Jan 2022 (v1), last revised 29 Nov 2022 (this version, v6).
7. Bai S., Kolter J.Z., Koltun V. An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling. Submitted on 4 Mar 2018 (v1), last revised 19 Apr 2018 (this version, v2).
8. Shazeer N. GLU Variants Improve Transformer. 12 Feb 2020. Arxiv.
COMPARATIVE ANALYSIS OF DEEP LEARNING MODELS FOR TIME SERIES FORECASTING ON SOLANA CRYPTOCURRENCY DATA USING DARTS
Al-Haidari Hazim Hamid Abdu1, Graduate Student Mohammed Ahmed Saif Haidar Al-maqtari2, Graduate Student Eskander Al-shaibani2, Graduate Student
Al-haithi Abdulwasea Nasser Hamid Moqbel2, Graduate Student 2Peoples' Friendship University of Russia 1National University of Science and Technology MISiS (Russia, Moscow)
Abstract. This research presents a comparative analysis of several deep learning models for time series forecasting on Solana cryptocurrency data, using the Darts library. The study evaluates the performance of six models, Block RNN, N-BEATS, N-HiTS, RNN, TCN, and TFT, using both empirical and quantitative. The Block RNN model demonstrated the best overall performance, achieving the lowest error rates, while N-BEATS and TCN closely followed. N-HiTS and TFT models struggled with higher complexity and the relatively small dataset, leading to poor performance. However, further training of the N-BEATS model resulted in significant improvements, demonstrating its potential in capturing long-term trends in volatile cryptocurrency markets. This study provides valuable insights for selecting deep learning models suited to forecasting in such dynamic environments.
Keywords: Deep learning, time series, forecasting, Solana cryptocurrency, Darts, RNN models, N-BEATS, and volatile markets.
