Adaptive multi-column multi-stage machine learning pipeline for predicting stock price by solving a nonlinear optimization program

Abstract

Predicting a stock’s next day price is a challenging task that researchers have long been attempting to address. Machine learning algorithms have shown to be successful at predicting next day price of a stock. A single machine learning model, however, suffers from limited stability and predictive power. Parallel models bring together a diverse set of learners to address these limitations. Predictions from individual diverse learners are typically merged into the final estimate using either a majority vote or simple or weighted averaging schemes. These schemes are simple and intuitive but static and sensitive to output from a single model that often leads to under-performance specifically for problems of a very dynamic nature, such as stock market prediction. To tackle these limitations, we present a new adaptive multi-column multi-stage pipeline, a novel technique to blend multiple individual learners by (1) calculating optimal column weights for the parallel models in the first-stage by solving a nonlinear minimization program (quadratic program) and (2) training a separate neural network based weight prediction module to estimate the optimal weights. We use Support Vector Regressiors (SVRs) as the multiple columns or multiple individual learners in the first-stage. We show the effectiveness of the proposed technique by attaining solid predictive performance on 8 stocks of National Stock Exchange, India. The proposed method outperforms other methods for 7 out of 8 stocks in the study.