A Data mining algorithm to analyse stock market data using lagged correlation

This paper develops an algorithm for predicting the market direction more accurately when two stocks are strongly correlated to each other with a lag of K number of trading days. The forecasting horizon is the lag; therefore this method is suitable for short term capital gains when the correlation is strong.. This will identify the stocks that are closely related, display the daily price movements and its direction side by side and forecast the direction of the price movement for the dependent stock as well as clearly showing the applicable lag. To test the effectiveness of the method, the most correlated stocks were found and prediction of the direction of the price movements made for 3 different dates for training the model. For each date actual data were then used to verify the accuracy of the prediction. In the testing and verification stage the model predicted the direction of the movement of the stock prices accurately 67% of the time. A generic algorithm is specified so that an automated data mining process can be developed. This algorithm takes into consideration the market-wise analysis performed, varying the lag from a lower limit to an upper limit as specified by the user, calculating the correlation coefficient for each independent stock and all other dependent stocks in the market, selects the pairs of stocks where the correlation coefficients are above a user specified range and lists the stocks data graphically side by side for easy comparison. The primary motivation of this paper is threefold. First, this research examines and analyses the use of market-wide lagged correlation analysis as a forecasting tool. Specifically the ability of one stock to predict the future usually short term future trends of a closely correlated another stock. Second, this paper endeavours to determine the feasibility and practicality of using lagged correlation analysis as a forecasting tool for the individual investor. Finally this paper specifies the general algorithm for the process so that it can be automated in a data mining technique In summary, the paper finds ways for the investor to reduce the short term risk of investing in the share market.