Bridging the Gap Between Machine and Human in Stock Prediction: Addressing Heterogeneity in Stock Market

Accurately predicting stock prices remains a formidable challenge in financial markets. Traditional predictive models often aggregate data from multiple companies, failing to account for the unique characteristics of each firm, which can hinder the model’s ability to identify company-specific patterns. Moreover, existing research on stock price prediction frequently trains and tests models within the same group of companies, neglecting to assess their generalizability on ‘Out-of-Sample’ companies. This study addresses these limitations by employing BERT to encode business descriptions into vectors, capturing the distinctive attributes of each company. We further enhance the predictive modeling framework by developing features that describe the percentage change of existing indicators, adding significant novelty to the existing research. Additionally, we apply a Restricted Boltzmann Machine (RBM) for dimensionality reduction after the BERT encoding process. In our approach, both the technical indicators and the vectorized descriptions are treated as distinct elements within the transformer encoder. By integrating these representations, our model is better equipped to differentiate between firms and recognize their individual patterns. The proposed model demonstrates superior performance over baseline models, particularly when tested on ‘Out-of-Sample’ companies, highlighting its ability to learn, understand, and analyze company-specific descriptions for more accurate predictions. This research offers novel insights into addressing the heterogeneity in stock price prediction.