Stock Prices Don’t Predict Stock Prices

LSTM Tomfoolery

As deep networks become more ubiquitous and easier for software developers to implement, many of them will undoubtedly use the familiar training grounds of stock price prediction to sharpen their LSTM skills. I’ve already seen several articles come about that show how to predict future asset prices by feeding in historical asset prices into LSTM and then producing a nice chart of how perfectly the predictions match up with reality.

As I’m sure you could have guessed, though, these are all a bit of a farce. We all know that no matter how good you are, you’re not predicting the market with 90–100% accuracy, no matter how you define accuracy. Stock and crypto markets are too dynamic, and as a society, we’re just not that good yet.

Typically, what I’ve seen done is that authors building a process whereby they take some previous number of prices for an asset, sometimes performing clever transformations on those prices (i.e. taking logs, lognorms, scaling those prices, or turning those prices into percentage return values), feeding that sequence through an LSTM, asking the LSTM to predict the next price, then shifting their window forward by a day to rinse and repeat.

When they’re done, they come up with nice plots that show that they were almost perfectly able to predict an asset’s price movement. What they don’t show, however, is that their LSTMs basically just get really good at using the previous day’s price (or some combination of the previous day and the recent history of stock prices prior to that day) to act as a stand-in for the next day’s price.

The accuracy of the next day’s price isn’t perfect, but if you keep predicting prices in this manner and plot them out, to the naked eye, it will look as if these prices are almost exactly predictive of reality.

In reality, you can’t trade these prices on a day to day basis because the per-day accuracy is not good enough to make money, and since these LSTMs aren’t predicting forward looking predictions more than one day in advance (typically), you can’t trust these networks to predict the longer-term price movements for a stock.

Of course, none of this is surprising. If you’re interested in learning more about how this process plays out specifically with respect to LSTMs, this article is a good one to read:

Efficient Markets

Of course, for those that have studied the equities market for long enough know that this is entirely predictable. The idea of an “efficient market” has become extremely popular, and though I don’t believe that the entire stock market is fully efficient, I do think that the effects of previous price are entirely non-predictive of future results. And I believe that there’s sufficient evidence to back this up.

In fact, here’s a great quote from the article I mentioned above that highlights how I feel about using stock prices to predict stock prices:

Princeton University economist Burton Malkiel, who argues in his 1973 book, “A Random Walk Down Wall Street,” that if the market is truly efficient and a share price reflects all factors immediately as soon as they’re made public, a blindfolded monkey throwing darts at a newspaper stock listing should do as well as any investment professional.

I’d strongly encourage everyone that’s interested in learning more about finance to familiarize themselves with the weak, semi-strong, and strong versions of the efficient market hypothesis.

Note again that I’m not saying I believe that any of these are true, but I have seen enough to believe that at least the weak version has some credibility to it, and I think the semi-strong version might as well.

In essence, what those theories claim is that you’re not going to be making money any time soon by using previous stock prices to predict future stock prices. Note this may not be true in the case of HFT, that’s an area I haven’t looked too deeply into quite yet, but I wouldn’t be surprised if one day these theories catch up to that industry as well.

Investing Properly

So if you can’t use previous prices to predict future stock prices, what may be some other methods to invest?

Well, you could, and should, invest in passive index funds if you have no idea what you’re doing or if you care to grow your investments slowly and steadily over the long run, letting compound interest (in the form of re-invested dividends) and massive economy growth work for you.

But if you’re in the camp of those that believe actively managed investments have the potential to outperform passive investing, you have a few other options.

One thing that’s quite common is to be a fundamental investor. The idea here is that you conduct an inordinate amount of due diligence and research into a company to understand its true business, and you allocate your funds to overweight those companies that have the best future outlook based on your research.

Fundamental investing goes hand in hand with value investing, which is a method whereby you attempt to determine the true value of a company’s stock and invest in that stock when it has dropped sufficiently low enough to provide an investment with a high probability of a “good” return, whatever that may mean for you.

Other methods include quantitatively based methods like stat arb, and more recently, machine learning methods.

At Apteo, we combine machine learning with fundamental investing in an effort to increase the likelihood that we can objectively find investments with high probabilities of return, and over the course of the foreseeable future, we plan to use machine learning to manage a portfolio that has been composed of these high-probability investments as well.

And guess what, we use LSTMs.

LSTMs — But Not The Way You May Think

Say what?

Earlier in this article, I claimed that LSTMs can’t be used to predict stock prices by feeding in prior stock prices.

I maintain that claim.

But if that’s the case, how are we using LSTMs?

Well, I mentioned earlier that we focus on using machine learning to scalably and automatically analyze the core metrics of companies’ businesses. We’re obviously looking at a variety of quantitative metrics from their reports and from the larger market, but we’re also looking at subjective analysis and opinions that outsiders produce.

There’s a huge amount of very smart people out there analyzing and conducting due diligence on a large variety of companies. These people can produce very well thought out reports (and sometimes reports that have no thought behind them). Traditionally, it’s been hard to scalably read and understand all of the reports out there.

So what we do is we use LSTMs to help us analyze the content of these reports. Essentially, we’re taking the automation of fundamental analysis one step further by incorporating methods to parse and quantify those subjective analyses that may have been hard to incorporate at scale before the advent of deep networks, at least at a very accurate level.

We believe that these reports, quantified and properly analyzed in the context of historical company movements, and in conjunction with other important financial metrics, can produce insights at scale. Our initial results are positive, and we’re using these results to create sample portfolios that we can track over time, while also backtesting these portfolios to see how well our networks would have done over the course of time.

Notice that throughout all of this, we put virtually no importance on the use of previous stock prices in our analysis. That’s not to say that we don’t use this information, or that we won’t rely on this sort of information more in the future. Rather, it’s to say that we have a belief that in the long run, stocks move according to the dynamics of the markets, and the markets, in the long run, rely on standard metrics like a company’s health, economic fundamentals, and sound reasoning, and we’re automating the process of leveraging all of this information to make better investments.

Source: Deep Learning on Medium