Why is it hard to predict stock prices even when buy orders continue? The market's 'long memory' revealed by the Hurst exponent
Buy, buy, buy, and buy again.
When looking at the execution history, there are times when trades in the same direction continue repeatedly.
If buy orders are this concentrated, it is natural to think that more buy orders will follow.
In fact, it has been confirmed that financial market orders possess a property called ‘long memory,’ where past trading directions are related to future trading directions.
However, there is something strange.
Even though orders have persistence, stock price changes do not show such clear persistence.
Even if the direction of the next order is predictable, it does not mean that the next price movement can be easily predicted.
This seemingly contradictory phenomenon has become an important theme in market microstructure research.
Last time, we considered information asymmetry behind orders through PIN.
This time, we will examine how those orders are connected over time.
The central focus will be the Hurst exponent.
What is the Hurst exponent? The strength of the market’s ‘memory’
The Hurst exponent is a statistical indicator used to analyze the persistence of time-series data.
It is generally represented by the symbol H.
Typical interpretations are as follows:
-
H = 0.5: The baseline where no long-term persistence is observed
-
H > 0.5: A state with persistence
-
H < 0.5: A state with anti-persistence
Anti-persistence is a property where relationships that cancel out the direction are more likely to appear than fluctuations continuing in the same direction.
A little caution is needed here.
Just because H is 0.5 does not mean that all autocorrelation is absent. Also, just because H exceeds 0.5 does not necessarily mean that future stock prices will rise.
The meaning changes significantly depending on what data the Hurst exponent is calculated for.
For example, when examining the trading direction of orders, buy is recorded as ‘+1’ and sell as ‘-1’.
Buy: +1
Sell: -1
By analyzing this sequence of signs, one can investigate the persistence where buys call for more buys, and sells call for more sells.
On the other hand, if the Hurst exponent is calculated for price returns, the persistence of price changes is analyzed.
Even with the same Hurst exponent, the information that can be read differs if the measurement target is different.
And the Hurst exponent does not tell you the time when an individual trend will end.
It merely represents statistical properties over a certain observation period.
‘Orders remember the past’ discovered by a 2004 study
In 2004, Fabrizio Lillo and J. Doyne Farmer published a paper titled ‘The Long Memory of the Efficient Market’.
The subject of the study was order data from the London Stock Exchange (LSE).
They focused on the correlation between the signs of buy and sell orders over time.
Normally, in a time series where the influence of the past fades quickly, autocorrelation decreases rapidly as the time interval increases.
However, the autocorrelation of order signs decayed very slowly.
The paper reports that the autocorrelation generally follows the following relationship:
C(k) ∝ k^(-0.6)
C(k) represents the autocorrelation of order signs separated by k intervals.
This decay pattern corresponds to a Hurst exponent of approximately 0.7.
In other words, there was a relatively strong persistence in the direction of orders.
This does not mean that “there is a 70% probability that the next order will be a buy order after a buy order.”
A Hurst exponent of 0.7 is a value related to the persistence of a time series, not direct win rates or the probability of the next order occurring.
Nevertheless, the discovery that the order flow is not a simple random sequence was highly significant.
This is because it indicates that the behavior of market participants is not independent of past orders.
Why do orders in the same direction continue?
Imagine the trading of institutional investors here.
Suppose an asset management firm wants to buy a large quantity of shares.
If they execute the entire quantity as a market order at once, they might consume the sell-side order book sequentially, causing the purchase price to rise significantly.
Therefore, they split large orders into smaller ones and execute them over time.
Such large parent orders are called metaorders.
Child orders split from the parent order are sent to the market repeatedly.
For example, if an asset management firm is executing buy orders over several hours, orders in the same direction will be observed repeatedly in the market during that time.
In 2005, Lillo, Szabolcs Mike, and Farmer published “Theory for Long Memory in Supply and Demand.”
This study theoretically explains the mechanism by which the split execution of large orders creates long-term correlation in order signs.
Long memory does not necessarily mean that many investors are reacting to the same news.
It is possible that a single large trading intention is manifesting as small orders over a long period.
If orders are predictable, why are prices hard to predict?
This is where it gets most interesting in this article.
Let’s assume we can predict the tendency for buy orders to continue.
And let’s assume that each individual buy order always pushes the price up by the same amount.
In that case, if you knew that buy orders would continue, it should be easier to predict price increases.
However, the actual market is not that simple.
Lillo and Farmer’s research shows that fluctuations in trade size and liquidity act compensatorily against the long memory of order signs.
Furthermore, in 2004, Jean-Philippe Bouchaud, Yuval Gefen, Marc Potters, and Matthieu Wyart studied the nature of price changes using data from the Paris Stock Exchange.
Their paper, “Fluctuations and Response in Financial Markets: The Subtle Nature of “Random” Price Changes,” analyzes the interplay between persistent market orders and limit orders that suppress price changes.
When buy orders continue in the market, participants providing liquidity also change their behavior accordingly.
If orders become easier to predict, the price impact per order may change.
A 2014 study by Taranto, Bormetti, and Lillo also reports the phenomenon that in directions where order flow becomes more predictable, the probability of individual trades moving the price decreases.
Even if buy orders continue, it does not mean that those buys push the price up with the same force every time.
The persistence of order direction and the persistence of price changes are separate phenomena.
This distinction provides a clue to understanding why order flow has long memory while price returns show relatively random fluctuations.
The Hawkes process captures the mechanism by which orders chain together
There is another way to study the chain of orders from a different perspective than the Hurst exponent.
That is the Hawkes process.
The Hawkes process is a model in which events that occurred in the past influence the rate of subsequent event occurrences.
For example, when a certain type of order occurs, it becomes more likely that the same type of order or related events will occur for a while.
This property is called self-excitation.
In financial markets, it is used to analyze phenomena where order arrivals and price changes are concentrated in time.
In 2014, Emmanuel Bacry and Jean-François Muzy published “Hawkes Model for Price and Trades High-Frequency Dynamics.”
In this study, trade arrivals, price changes, and the mean-reverting nature of price changes are expressed using a multivariate Hawkes process.
Furthermore, in 2015, Bacry, Iacopo Mastromatteo, and Muzy published a review summarizing the application of Hawkes processes in the financial field.
The Hurst exponent and the Hawkes process analyze slightly different subjects.
The Hurst exponent is an indicator used to examine the persistence of a time series.
On the other hand, the Hawkes process models how past events influence the rate of future event occurrences.
Note that just because there is self-excitation does not necessarily mean that long memory exists.
There are also Hawkes models where the influence of the past disappears in a short time.
Also, just because a buy order occurs does not mean the next one will be a buy; it is necessary to estimate the influence on sell orders and price changes as well.
While the two are not the same, there is room to combine them as a way to examine order flow from different angles.
Long memory is also being studied in Japanese stocks
There is a study that cannot be overlooked when applying this theme to Japanese stocks.
In 2023, Yuki Sato and Kiyoshi Kanazawa published “Inferring Microscopic Financial Information from the Long Memory in Market-Order Flow: A Quantitative Test of the Lillo-Mike-Farmer Model.”
It was published in “Physical Review Letters.”
The study uses data from the Tokyo Stock Exchange from 2012 to 2020.
What is noteworthy is that they analyzed special data that includes information to identify trading accounts, not just the buy/sell direction of orders.
The researchers examined the behavior of participants who split orders for execution and verified the relationship between individual traders’ order splitting and the long-range correlation of order signs appearing in the entire market.
As a result, they reported that the quantitative predictions of the Lillo-Mike-Farmer model proposed in 2005 are consistent with actual TSE data.
Important empirical results regarding the long memory of order flow have been obtained in the Japanese market as well.
A research theme called the “Hurst Exponent Calendar” for Japanese stocks
From here on, these are application ideas based on existing research.
What would we see if we estimated the Hurst exponent by dividing it into various times of the year?
In the Japanese stock market, there are times when the behavior of trading participants is prone to change, such as fiscal year-ends, consecutive holidays, and the year-end/New Year period.
For example, the following questions can be considered.
Does the persistence of order signs change around the March fiscal year-end?
Do differences appear in the Hurst exponent along with changes in liquidity on trading days around Golden Week?
Are different order-splitting patterns observed during the year-end/New Year period or periods when financial results announcements are concentrated compared to normal trading days?
As a way to analyze such questions, a “Hurst Exponent Calendar” that organizes the Hurst exponent by date or event can be considered.
Of course, it is not guaranteed that the Hurst exponent will be high in a specific month.
When actually investigating, it is necessary to consider the influence of trading volume, volatility, day of the week, and trading hours.
It is important to collect sufficient data for each event and confirm whether it is reproduced over multiple years.
Rather than simply looking for days with a high Hurst exponent, analyze under what conditions the order behavior of market participants changes.
Therein lies the interest as a research theme.
How to investigate the Hurst exponent with actual order data
So, what preparations are necessary to actually analyze the long memory of order flow?
First, prepare order data or trade data that can distinguish between buy and sell directions. When using trade data, identify whether the buyer or the seller actively executed the trade.
Create a time series by assigning +1 to buyer-initiated trades and -1 to seller-initiated trades.
If the trade direction is not recorded, there are methods to estimate it from the relationship between the trade price and the quote. However, since misclassification affects the results, accuracy verification is necessary.
Next, examine the autocorrelation of the order signs.
For example, check how much orders in the same direction are related when widening the interval between orders, such as 1, 10, or 100 orders ago.
Then, estimate the Hurst exponent.
Estimation methods include R/S analysis and DFA (Detrended Fluctuation Analysis).
What is important here is not to calculate the Hurst exponent just once and be done with it.
Changing the observation period or estimation method may change the results.
Since biases in intraday trading or changes in market conditions can create apparent persistence, careful comparison is required.
In practice, it is easier to organize the verification in the following order.
First, confirm the persistence of order signs.
Calculate the autocorrelation and Hurst exponent, and compare them with data that has been randomly shuffled.
Next, compare with price returns.
Investigate whether the same persistence exists in price returns during periods when order persistence is strong.
Finally, re-verify with a future period.
Check if the same relationship can be confirmed in data not used for estimation, and consider spreads and commissions when using it for trading strategies.
Furthermore, by combining this with order book depth or order flow imbalance, you can also analyze the relationship between persistence and price impact.
Note that in decentralized over-the-counter markets like FX, it is impossible to reproduce the order direction of the entire market using only tick data from a specific dealer.
In this analysis, it is particularly important to know which market and which trades the data represents.
Summary: The market remembers more than just prices
When looking at market charts, we inevitably focus on price fluctuations.
However, behind the scenes, many participants are splitting orders, providing liquidity, and trading while reacting to past orders.
Lillo and Farmer’s research showed that order signs have long-term persistence.
Subsequent research has also revealed the mechanisms behind the split execution of large orders and changes in market liquidity.
And in 2023, important research was published in the Japanese stock market verifying the relationship between order splitting and long-range correlation.
Understanding the Hurst exponent is not a shortcut to predicting the future of prices.
Rather, it is an entry point for thinking about why order persistence does not directly translate into price persistence.
When considered together with OFI, Kyle’s lambda, and PIN, which have been covered in previous articles, different aspects of the market gradually connect.
Which way are the orders biased?
How much does that order move the price?
How much informed trading might be included?
And how long does the order bias persist?
These are separate questions, but they are interrelated.
Next time, we will cover the ‘square-root impact law,’ which deals with the non-linear relationship between order volume and price impact.
When you place a large order, does the impact on the price increase in proportion to the order volume?
This is an important theme when considering market liquidity and execution costs.
Reference Papers
-
Lillo, F., & Farmer, J. D. (2004). The Long Memory of the Efficient Market. Studies in Nonlinear Dynamics & Econometrics, 8(3).
https://doi.org/10.2202/1558-3708.1226 -
Bouchaud, J.-P., Gefen, Y., Potters, M., & Wyart, M. (2004). Fluctuations and Response in Financial Markets: The Subtle Nature of “Random” Price Changes. Quantitative Finance, 4(2), 176–190.
https://doi.org/10.1088/1469-7688/4/2/007 -
Lillo, F., Mike, S., & Farmer, J. D. (2005). Theory for Long Memory in Supply and Demand. Physical Review E, 71, 066122.
https://doi.org/10.1103/PhysRevE.71.066122 -
Tóth, B., Eisler, Z., Lillo, F., Kockelkoren, J., Bouchaud, J.-P., & Farmer, J. D. (2012). How Does the Market React to Your Order Flow? Quantitative Finance, 12(7), 1015–1024.
https://doi.org/10.1080/14697688.2012.690886 -
Taranto, D. E., Bormetti, G., & Lillo, F. (2014). The Adaptive Nature of Liquidity Taking in Limit Order Books. Journal of Statistical Mechanics: Theory and Experiment, P06002.
https://doi.org/10.1088/1742-5468/2014/06/P06002 -
Bacry, E., & Muzy, J.-F. (2014). Hawkes Model for Price and Trades High-Frequency Dynamics. Quantitative Finance, 14(7), 1147–1166.
https://doi.org/10.1080/14697688.2014.897000 -
Bacry, E., Mastromatteo, I., & Muzy, J.-F. (2015). Hawkes Processes in Finance. arXiv.
https://arxiv.org/abs/1502.04592 -
Lillo, F. (2021). Order Flow and Price Formation. arXiv.
https://arxiv.org/abs/2105.00521 -
Sato, Y., & Kanazawa, K. (2023). Inferring Microscopic Financial Information from the Long Memory in Market-Order Flow: A Quantitative Test of the Lillo-Mike-Farmer Model. Physical Review Letters, 131, 197401.
https://doi.org/10.1103/PhysRevLett.131.197401
*This article is an introduction to financial market research. The analysis examples and research ideas in the text do not indicate the effectiveness of any specific trading strategy or future profits.
■ Hurst exponent: Measures the persistence of price fluctuations
Efficiency Ratio: Measures the linearity of price movements
ADX: Confirms the strength of a trend
3-color line: Green = trending environment, Orange = ranging environment, Gray = judgment pending
Information panel: Displays each value, environment judgment, and upward/downward direction. Includes an ON/OFF button
A reference standard that considers the bias of finite samples associated with R/S estimation, rather than simply comparing the Hurst exponent to 0.5