The scope of this project will be to explore the possibility of projecting future performance of the S&P 500 Index. Being able to make a confident prediction of future performance is crucial in making financial investments. Furthermore, producing a model that can accurately predict how a certain index will perform will also give good insight as to how well many of the stocks within that index will perform. The goal of the project will be to use a Long Short-term Memory (LSTM) recurrent neural network (RNN) architecture to perform technical analysis on past performance to predict future performance.
A very important disclaimer is that there are a multitude of factors that ultimately affect the performance of the stock market that are outside the analysis of past performance. These can include economic factors such as interest rates and economic outlook or company news and performance for a specific stock. That being said, past performance and current market trends (technical analysis) can serve as a very powerful indicator of how a stock will perform in the future. The very basic idea behind technical analysis is to analyze several key numbers of a stock from a previous trading period. These include previous high, previous low, previous open, and previous close. These four numbers are used to form candlestick graphs, which are a graphical depiction of each of the values for a chosen time period (days, weeks, months, and so forth). These form a basis of an intuition into how a stock may perform in the future. On top of this prediction, other indicators and trends are used to validate the prediction. These can include trends formed from moving averages, volume indicators, or relative strength index (just to name a few).
At the base of the problem, a prediction needs to be made based on a time series data set. Because of the time series nature of the data, the solution will need to be able to remember relative older data in order to make a forecast on future performance. Some of the simpler classifiers such as linear regression and k-nn risk overfitting the data to dates and months and thus forecasting the exact same trends seen in the same months of past years. LSTM networks work well in this type of problem because important information can be retained for long periods of time. Each unit of a LSTM network is made up of a cell. The cell works by using three types of gates to regulate information stored in a cell (input gate, output gate, and forget gate). The output of each cell then contributes to the final decision of the network.