The article below, What Can Replace the Efficient Markets Theory?, is the work of FT columnist Jonathan Davis. The article considers the efficacy of the efficient markets hypothesis (EMH) - which continues to be taught widely in business schools despite meaningful empirical support. From this base, the article goes on to discuss the work of Professor Andrew Lo and, specifically, the adaptive markets hypothesis which he has presented as a viable alternative to the seemingly unmovable dinosaur of investment finance, the EMH.

What can replace efficient markets theory?

The most interesting thing about the efficient markets hypothesis is not whether it is valid or not – clearly it is not – but how it has managed to remain so influential for so long. At a recent conference in London on the subject, organised by the CFA Institute, Professor Andrew Lo of Massachusetts Institute of Technology offered the audience a simple explanation: “physics envy”.

This was a reference back to the early inspiration of the Nobel economics laureate Paul Samuelson, who set out to find for economics a set of fundamental laws that would do for the dismal science what Newton’s laws of dynamics had done for physics, and from which a rigorous general theory with practical uses could subsequently be developed.

Using such building blocks as utility theory, equilibrium and the principle of no arbitrage (“no free lunches”), this led Mr Samuelson and his many successors to develop what we have come to know as the discipline of microeconomics that is universally taught to every finance and economics student at university and business school. The efficient markets hypothesis and the notion that stock prices follow a random walk are offshoots of this approach.

The attempt to bring order and an overarching theoretical framework into analysis of the seemingly unruly behaviour of financial markets was a temptation that has for years proved too great for academics (and many market participants) to resist, but it has turned out to be a long and largely fruitless journey.

The problem of course, as Prof Lo has helped to demonstrate with his empirical studies of the random walk, is that the financial markets simply don’t lend themselves to deductive theory as well as the physical world. If a theoretical approach is not firmly grounded, it is not surprising that the predicted consequences that flow from it should fail to show up consistently in the way that investors and markets actually behave.

Behavioural finance has grown to become a popular alternative approach precisely because it does appear to explain more clearly how investors, individually and collectively, appear to act.In Prof Lo’s words: “Economic systems involve human interactions, which almost by definition are more complex than interactions of inanimate objects governed by fixed and known laws of motion.”

The real beauty of the efficient markets hypothesis, and the explanation for its longevity in the face of consistent empirical evidence that it is invalid, surely lies in its beguiling simplicity. As the future is uncertain and many of the key variables that concern investors cannot be predicted with confidence, a theoretical structure that appears to offer a way to live with uncomfortable reality has obvious attractions.

Prof Lo’s own response has been to develop what he calls the adaptive market hypothesis, which seeks to draw on the insights of neuroscience and evolutionary biology. The hypothesis aims to create a framework that seeks to relate the behaviour of financial markets to a number of different factors, including the emotional condition of market participants at different points in time and the current balance of advantage between competing groups of market participants. “Market efficiency,” he says “cannot be evaluated in a vacuum, but is highly context-dependent and dynamic, just as insect populations advance and decline as a function of the seasons, the number of predators and prey they face, and their abilities to adapt to an ever-changing environment.”

What is at work in financial markets, he believes, is a Darwinian process of “survival of the richest”. The implications of this approach are interesting. One is that the relationship between risk and return will not be stable over time, which seems right both intuitively and empirically. Another is that, rather than markets becoming steadily more efficient over time, as early proponents of the EMH proclaimed, this world is one in which new profit opportunities will continue to emerge at a constant rate.This is the engine that provides the continuing incentive for active managers to remain in the market.

But they will need to be innovative and adaptive to changing market conditions if they are to remain successful, Prof Lo argues. One-trick ponies risk going out of business before their kind of market next comes around.Most important of all, investors cannot rely on the comforting message of the efficient market hypothesis that all you need to do to obtain an expected return is to take the appropriate level of risk.

The biggest problem with this new approach, as with all alternatives to EMH, including behavioural finance, is that it doesn’t give investors a simple metric for understanding what to do.Its great merit, however, is that it appears to relate to the complex and uncertain world that we all actually inhabit, something the efficient markets hypothesis has never done.