CC BY
134Analysis of Investors' Strategies Using Backtesting and DEA Model*
Dina NASRETDINOVA
Financial University, Moscow dnd17@mail.ru
Darya MILOVIDOVA
Financial University, Moscow milo. dariya@gmail. com
Kristina MICHAILOVA
Financial University, Moscow kmikhaylova26@gmail.com
Abstract. This paper analyzes the efficiency of the rules described by famous "investment gurus". We backtested 30 strategies over the period of 20 years using monthly data from USA stock market and scored their comparative characteristics using DEA model. Although strategies vary in historical performance, 11 strategies managed to beat benchmarks over the long term. Most efficient strategies according to DEA appear to be Graham, Lian, Zweig, Siegel strategies.
Аннотация. Статья анализирует эффективность правил, на которых основаны стратегии профессионалов в сфере инвестиций. Мы провели бэктест 30 стратегий за 20 лет, используя месячные данные американского фондового рынка и DEA-модель, оценили их сравнительные характеристики. Хотя стратегии и варьируются исторически, 11 стратегий превзошли эталонный индекс в долгосрочном периоде. Наиболее эффективными стратегиями, согласно DEA, являются стратегии Грэхема, Льяна, Цвейга и Сигеля.
Key words: DEA model, backtesting, strategy, benchmark, return, risk, DEA-score.
INTRODUCTION
Technical analysis and technical trading rules draw a lot of attention for last several decades just because of simplicity of making historical backtests. While many non-professional investors and almost all professionals are using fundamental analysis as their primary tool for long-term investments, little has been done so far to question empirically performance of fundamental trading rules. How these rules performed in the past, relative to each other and to the benchmark? Do "investment gurus" really add value with their strategies, or they just monetize on bestsellers, describing it? In our research we examine this question by formalizing fundamental trading rules, or "screens", and then performing historical backtest. Further we develop and apply DEA methodology to select rules that are most efficient based on wide variety of efficiency measures, as many investors use different approaches to select best screening rules.
Our results could be interpreted two-ways. If there is significant divergence between results reported or observed in reality and results obtained in backtest, it may mean that either the author is not telling the whole truth about his stock selection methodology, or his stock picking skill is weaker than his marketing team. Results from DEA also proves this paradox. 5 strategies out of 30 were inefficient.
STOCKSCREENING
While there are two basic fundamental investing styles: growth and value, many of stock screening strategies have both a value and a growth component. One of the most ineffectual aspects of choosing a stock investment strategy is the historical performance. The historical performance often assessed by backtesting.
Stock screening, i.e. applying sets of filter rules to fundamental parameters of wide universe of equities
* Анализ стратегий инвесторов с помощью использования бэктеста и DEA-модели.
to select stocks in portfolio has been widely applied by practitioners.
Many sets of filters have been proposed. However, some of them apply rules and philosophies that contradict each other.
We have used strategies of B. Graham, D. Dre-man, J. Neff, W. Buffett, P. Lynch, K. Fisher, M.Zweig, J. O'Shaughnessy, J. Greenblatt, J. Piotroski, I. Kahn, A. Nutt, W&E Schloss, J. O'Neil, and others in our research to backtest it and estimate its weaknesses and strengths.
All strategies were backtested over the 20 years period (1993-2013) with monthly rebalancing. Screening universe was comprised of all members of Russell 3000 Index, NASDAQ Composite Index, S&P 1500 Composite Index. Performance of almost all strategies (except strategies with "absolute income") were recorded against benchmark.
We considered strategies by their performance.
Lian's strategy focuses on early growth industries or fallen angels out of favor sectors. Its screen returned 6947,94% vs S&P 500 returned 419.83% which beats the benchmark. The mean active return is around 16% which is the best result among other strategies. Sharpe ratio is also high which shows us that this strategy is not risky. Lian's symmetric return distribution function has high kurtosis. It also has fat tails, mostly in positive end of distribution curve showing that strategy has a lot of profitable deals (Figure 1).
Further research of profitable strategies showed Fisher's strategy for technology industry. Portfolio returned 1635.86%. Fisher started to trade since 2002. The strategy has been growing steadily, but from 2003 to 2007 there was a period of stagnation, i.e. the
strategy had no income. During the crisis, the strategy has fallen relatively deep. But it has recovered quickly. Sharp ratio indicated positive value. However, the standard deviation was high. Hence, we can claim that the strategy has a high risk. It is obvious from the FisherTech strategy's return distribution function that it implies unexpected figures. Kurtosis is sharp and extremely high. It has long positive and negative tails, however, skewness is positive, which proves the high profit of this strategy. Figure 2 represents the full information.
Nevertheless, another growth screen based on his approach did not beat benchmark. Fisher screen has delivered a return of 19.26%. Thus, this strategy doesn't work. Moreover, the value of Sharpe ratio is 0,06, which is the lowest compared to rest of strategies. Furthermore, the skewness has high negative value.
According to Browne's Screen, which is represented by Figure 3, this portfolio returned 1412.14%. Thus, the strategy beats the benchmark. It has a high value of Sharpe ratio. Moreover, it has standard deviation of 22,39 which is not the highest value comparing to other strategies. Brown's strategy has symmetric return distribution function. However, it has fatter positive tail. Therefore, it makes strategy profitable.
Another relevant example of profit-making strategy is Piotroski's method which focuses on the stocks whose B/M ratios were in the top 20 percent of the market. He wanted a firm's ROA to be positive. Piotroski's portfolio returned 970.43%. Screen beat the benchmark. During the crisis the maximum of drawdown was reached. This strategy has recovered quickly and continued stable growth. The Sharpe ratio is relatively low. In addition, it has the highest standard deviation among all strategies. Piotroski's
-0.2 0.0 0.2
Returns
Figure 2. Return distribution function. Fisher's strategy.
Figure 3. Return distribution function. Browne's strategy.
strategy shows symmetric return distribution function. Slightly left-skewed, but tail in positive, end is fatter which brings high return figures (see Figure 4).
Graham's screen returned 940.64%. During times of crisis market was shrinking much faster than the strategy did. While the market was slowly recovering, Graham's strategy returns were steadily growing. On average, strategy selects 15 stocks with moderate turnover ratio. This strategy is not very risky which is proved by its Sharpe ratio of 0,43. Graham's strategy revenue distribution function is symmetric, but with fatter positive tail, which makes the strategy profitable. In general, from figure it is evident that it is steady going. Information about Graham's strategy results is given in Figure 5.
Schloss's rules are based on Graham's strategy but with insignificant changes. He chooses companies with real assets with little or no debt and stocks that were selling below their book value. Figure 6 represents Schloss's screen portfolio results. It returned 840.56%. In general, the strategy beats the benchmark. During the crisis there was a drawdown. Sharpe ratio is higher compared to the previous one. A careful analysis of Schloss's return distribution function implies evidence of its reliability. Function is symmetric and has fat tail in positive side.
O'Neil identified stocks based on at least 25% current quarterly earnings per share, P/E in the range of 20 to 45. His strategy returned 758.03% during the whole period. The semivarience has one
Figure 4. Return distribution function. Piotroski's strategy.
Figure 5. Return distribution function. Graham's strategy.
of the highest values of strategies we considered. This strategy has a low risk due to relatively high Sharpe ratio and low standard deviation. O'Neil's strategy return distribution function is very close to "ideal" Gauss distribution function. It has normal, flatter than other strategies' figures kurtosis and fat tails both in positive and negative sides (Figure 7).
Greenblat's Screen implements the Magic Formula value investing strategy. It is based on buying 20-30 "good, cheap companies". In Magic Formula he averaged a 17-year annual return of 30.8% and beats the S&P 500.96% of the time. In practice we obtained the following results. Greenblatt's portfolio returned 573.81%. Thus, this strategy in theoretical and practical terms beat the market. During the cri-
sis was the deep drawdown. But the strategy quickly recovered and continued stable growth. Closer look at strategy (return distribution function) explains high returns. This figure is symmetric; a lot of deals with positive returns appear (Figure 8).
Buffet's approach is solely based on stocks overall potential as a company. The screen showed the 569.74% return. From 2001 to 2006 (within 5 years) strategy performed poorly; during the crisis screen returns began to fall before benchmark and fell fairly deeply, but recovered much faster than the market. This strategy appears to have quite large periods of stagnation, followed by periods of sharp recoveries. Standard deviation equals to 26,47, which is one of the highest results obtained. Buffet distribution fig-
-0.2 -0.1 0.0 0.1 0.2
Returns
Figure 7. Return distribution function. O'Neil's strategy.
Returns
Returns
Figure 9. Return distribution function. Buffet's strategy.
Figure 10. Return distribution function. EveiUard's strategy.
ure is symmetric with long tails. Kurtosis is high, tail to the right is fatter which, gives positive return in general. Results are represented in Figure 9.
Eveillard's portfolio consists of securities whose intrinsic value and long-term potential outweighs market risk. Screen has delivered a return of 503.71%. During the crisis was the deepest drawdown, but the strategy quickly recovered and continued stable growth. The density graph of this strategy is symmetric. As we see from this figure it has fat right tail which indicates its profitability (see Figure 10).
Lynch's strategy for all stocks returned 414.67%. In the whole, this strategy cannot beat the market. From 1999 to 2007 strategy rose more sharply than bench-
mark and beat it. But during the crisis it fell then slowly recovered and in 2012 fell again. It has negative skew-ness in return distribution function. But fat tails on the right end shows strategy's profit. Figure 11 gives the illustrated results.
In addition Lynch proposes two approaches - one for fast-growers, the other for slow-growers.
Slow-growers strategy focuses on the large/aging companies growing only slightly faster than the economy as a whole, but often paying regular dividends. Screen returned 256.90%. Strategy started to trade since 1995; during 1995 and 2001 performance was stagnant, while the market grew steadily. Starting from 2003 to 2008, strategy returns were growing and in 2008 it was able to beat
Figure 11. Return distribution function. Lynch's strategy.
Figure 12. Return distribution function. Lynch's strategy (slow-growers).
benchmark. During the crisis the market dropped deeper than slow-growers strategy. And then, when the market grew stronger and began to gain momentum, this strategy fell and remained in the period of stagnation. Lynch slow strategy's return distribution (see Figure 12) function has asymmetric figure. It has positive skewness. But right tail fatter than left, which leads to profitability of strategy. However, with reference to another Lynch's approach called fast-growers (Figure 13) we see more symmetric figure with flatter kurtosis.
Fast-growing strategy's screen focuses on the small, moderately fast-growing companies bought at
a reasonable price. Lynch's screen has 215.19%. The strategy for the entire period fluctuates between periods of stagnation and growth. During the crisis, the strategy has fallen much deeper than the market, but at the same time when the market fell steadily, the strategy of fast-growers grows steadily. Performance during 2009-2011 was better than the benchmark. But then, as the market broke sharply upwards, it began to fall and leave in a period of stagnation.
Lynch - Stalwarts strategy implies focusing on large companies that are still able to grow, with annual earnings growth rates of around 10%-12%. Lynch's slow-growers portfolio returned 79.65%, while the
S&P 500 returned 419.83% during the same period. This strategy does not beat benchmark.
O'Shaughnessy's screener is a combination of two models: a momentum/earnings growth-focused method called "Cornerstone Growth" and a value-focused method called "Cornerstone Value".
In his Cornerstone Growth approach, he chooses companies that have market capitalization of at least $ 150 million, price - sales (P/S) ratios below 1.5. Finally O'Shaughnessy ranks companies for highest relative price strength over the previous year and chooses the top 50. According to equity screen of this approach, we have the following results. In general, for all the period screen does not
beat the benchmark. O'Shaughnessy Cornerstone Growth portfolio returned 174.21%. Sharpe ratio is the lowest of other strategies. Return distribution function of this approach is symmetric. Fatter right tail made strategy such profit. Kurtosis is relatively flat (see Figure 14).
Cornerstone Value is a five criteria large-cap dividend yield-focused value screen outlined in James O'Shaughnessy's work. His work showed that a large-caps stock portfolio with above average stock liquidity and cash flow per share which was ranked for high dividend yields performed the worst results over the long term. According to his work, this value strategy has 0% of return, compared to 419.83% for the S&P
Returns
Figure 13. Return distribution function. Lynch's strategy (fast-growers).
500 Index. For all time, the strategy has not selected any stock. Hence, this strategy does not work.
In addition Dreman's investment approach which is based on interpreting market psychology and using value measures to pick stocks that are out of favor with the market performed with 0% return. Dreman claims that he invests in out-of-favor stocks, often in out-of-favor industries, that he identifies using relatively straightforward formal criteria.
Moreover, P. Lynch's strategy for financial companies does not work. Financial screen has 0% return.
Further research in this area may include other general ratios, indicators and graphs. Capture ratio stands for analyzing strategies' behavior relatively to market's behavior. Capture ratios divided into down-market and up-market ratios. The up capture ratio should be greater than 100%, which would indicate that during periods when the market is up, the investor, on average, did even better. The higher the up capture, the better strategy is. Alternatively, down capture ratios should be less than 100%, meaning that when the market went down the investor caught only a fraction of the losses. The lower the down capture, the better. Although rare, it is possible to see negative down captures, indicating that when markets are down the manager tends to be up.
The Graph 1 represents the typical display of up and down capture. The reference point is the benchmark, as noted by the crosshairs in the middle. The top left quadrant represents the ideal location. Here,
the strategies of investors are up more than the market in up periods and down less in falling markets. As we can see that Lian and FisherTech have one of the best results. The down left quadrant represents that investment strategies lag when markets are up, but hedges in down markets (such as: Hye, Siegel, Mo-bius). In the top right stands the aggressive quadrant, riding high in up markets but losing more in down markets.
The chart represents the portfolio return. First of all, we diversified our portfolio (combined our investment strategies). Portfolio diversification is the means by which investors minimize or eliminate their exposure to specific risk, minimize or reduce systematic risk and moderate the short-term effects of individual asset class performance on portfolio value. In a well-conceived portfolio, this can be accomplished at a minimal cost in terms of expected return. Such a portfolio would be considered to be a well-diversified. We see that such strategies like Fishertech, Lian, Schloss, Nutt are higher than the portfolio line.
DEA MODEL
Each investor accepts different level of risk and other factors. Efficiency of strategy can be also good indicator in making decisions. While backtesting shows us historical behavior of investors' strategies, DEA model gives information about efficiency of different
strategies in general. We analyzed strategies using DEA model to prove backtesting results, obtain new information about strategies and find other acceptable efficient ones.
DEA methodology was widely adopted in literature dedicated to performance evaluation of various classes of financial market participants (see, for example, Gregoriou, Zhu (2005); Fedorova, Didenko (2014a) and (2014b)).
The general property of various DEA techniques is that it uses minimum quantity of parameters and assumptions, independence of unit measurements, support on easily interpreted empiric results. In this regard authors offer to estimate efficiency of management companies in Russia by DEA method. If we estimate efficiency of the companies in such way, there is the following question: How to define the factors influencing efficiency activity?"
Generally, DEA is methodology which connects operational research, mathematics and economics. The DEA methodology uses mathematical programming to process empirical data on inputs and outputs of a given group of decision making units (DMUs). As a result, each DMU is assigned a value within interval (0,1]. Value 1 represents relatively efficient DMU, while the DMU with value less than 1 is deemed inefficient. In this way, the efficiency of each DMU is evaluated with respect to other DMUs. Our DMUs in this case are strategies of different investors. Thus, we will define from 0 to 1 their rate of efficiency.
Appropriate inputs and outputs are important in DEA model. Each author offers his own inputs and outputs. We took following ones: "Turnover", "Semi-variance" as inputs, "MAR", "Skewness" as outputs. Our decision is based on deep analysis of all the input/output combinations. For example, "Turnover" was taken as an indicator. Higher turnover means higher commissions and strategy will be more money-losing. Hence, with higher turnover expenses are higher and the strategy is worse.
Among 30 strategies only 4 were efficient (with 1-efficiency). As we see from Graph 2, some strategies are not included, because of 0-efficiency. The most effective ones are Graham, Lian, Zweig, Siegel (numbers 2,10,19,28).
For comparison, we made another model. We took the main indicators of risk as inputs: "Jensen.Alpha", "Beta" and "Information.Ratio", and as outputs "Turnover" and "Total Return".
Despite new parameters, the most effective strategies did not change. "Leaders" are Graham, Lian, Zweig strategies.
Lian's strategy is the best one from both results of backtesting and DEA-score. (see Figure 15). Thus, fat positive tail tells that this strategy is both profitable
Graph 2. Bar chart. Efficiency of strategies (inputs - "Turnover", "Semivariance", outputs - "MAR", "skewness").
Graph 3. Bar chart with efficiency results (inputs - "Jensen. Alpha", "Beta" and "Information.
Ratio", outputs - "Turnover" and "Total Return").
and reliable.
Comparing our DEA model results with results obtained after backtesting, we see one contradiction. Siegel's strategy was one of the "worst" strategies from backtesting results while in DEA-score it obtained 1-efficiency. We can explain it with a closer look at the data. According to distribution function, (Figure 16) strategy has high skewness by means of one deal with very high profit. Skew-ness was taken as output in our first DEA model. Skewness was significant in obtaining DEA-score. Moreover, in second DEA-model with other parameters Siegel did not obtain 1-efficiency. Hence, we can sum up that Siegel's strategy achieved good results due to high skewness. But we can not consider strategy as reliable one. Extremely high skewness makes strategy very risky and non-reliable. Hence, strategy is not satisfactory.
DEA-score indicated strategies as efficient and non-efficient. Interdependence of efficient and inefficient strategies appears to be supported by Dendrogram (Figure 17). Dendrogram reflects information of correlation between different strategies. We can observe that Graham and Lian have very high correlation which shows the identical behavior of these strategies to the changes on the market. FisherTech, Schloss, Lynch, Piotroski have the lowest correlation with other strategies. The most profitable ones we
Figure 15. Lian's strategy. Return distribution function.
Figure 16. Return distribution function. Siegel's strategy.
got from results of backtesting, FisherTech less correlated with profitable Lian, Siegel, Graham strategies. Efficient Graham and Lian strategies are highly correlated.
CONCLUSION
The results of this study suggest a number of new avenues for research. Concluding our scientific work, we can say that not all the strategies suggested by investors are profitable and reliable. DEA-score and backtesting results provided confirmatory evidence of that.
After backtesting all our strategies, we obtained the following results. Screens of Dreman, O'Shaughnessy (Value), Lynch (Financial Companies) and Siegel have 0% return vs. 419.83% for the S&P 500. That means that they have not invested in anything at any time. The strategies that do not beat the market for the whole period are K. Fisher
(Super-Stock), K. Fisher (All Stocks), Kahn, Hye, O'Shaughnessy (Growth), Lynch (Slow-growers), Lynch (Stalwarts), Mobius, Zweig. It means that these investment strategies do not work in the history. In general, these strategies exceed the market, but not much: Greenblatt, Eveillard, Kahn, Lynch (All Stocks), Lynch (Fast-growers). Finally, such strategies as Buffett, Graham, Browne, Fisher (Technology), Lian, O'Neil, Nutt, Piotroski, Schloss and W. Schloss beat the benchmark of the whole period. Most of them have high rates of return, especially Lian's.
DEA-score showed that Graham, Lian, Zweig, FisherTech strategies are efficient and reliable, and 5 strategies appeared to be completely inefficient. However, results of backtesting and DEA are not contradictory. Hence, results of one way of analysis are a ground for another. It is hoped that this study will stimulate further research in the field of investment.
Figure 17. Dendrogram.
REFERENCES
Buffett, Warren "The Essays of Warren Buffett: Lessons for Corporate America", — Cardozo Law Review's — 3d ed., 2001, 357.
Clark David, Marry Buffett "Warren Buffett and the Interpretation of Financial Statements. The Search for the Company with a Durable Competitive Advantage", — Simon & Schuster, 2002, 140.
Davis Ned "The Triumph Of Contrarian Investing — Crowds, Manias, And Beating The Market By Going Against The Grain", — The McGraw — Hill Companies Inc., 2004, 194.
Dreman David "Contrarian Investment Strategies — The Next Generation", — Simon&Shuster Inc., 1998, 478.
Fisher P. "Common Stocks and Uncommon Prof its", — New York: Harper & Brothers, 1958, 317.
Forbes Steve "Even Buffett Isn't Perfect What You Can and Can't Learn from the World's Greatest Investor", — John Wiley & Sons, Inc., Hoboken, New Jersey, 2010, 323.
Gardner D., Gardner T. "The Motley Fool Million Dollar Portfolio", — HarperCollins Publishers Inc., 2008, 265.
Graham B. "The Intelligent Investor", — HarperCollins Publishers Inc., 2003, 638.
Graham and Dodd "Security Analysis", — The McGraw — Hill Companies Inc., 2009, 818.
Greenblatt J. "The Little Book That Still Beats the Market", — John Wiley & Sons, Inc., Hoboken, New Jersey, 2010, 52.
Hagstrom R. "The Warren Buffett Way", — John Wiley & Sons, Inc., Hoboken, New Jersey, 2005, 268.
Hagstrom R. "The essential Buffett, timeless principles for the new economy", — John Wiley & Sons, Inc. (US), 2001, 230.
Klein R. G. "Benjamin Graham on Investing", — The McGraw — Hill Companies, Inc., 2009, 419.
Lynch Peter "Beating The Street", — Simon & Schuster, 1994, 170.
O'Neil William J. "How To Make Money In Stocks", — The McGraw — Hill Companies Inc., 2009.
Greg N. Gregoriou, Joe Zhu Evaluating Hedge Fund and CTA Performance: Data Envelopment Analysis Approach — Wiley, 2005.
E. Fedorova, A. Didenko. DEA — based rating of pension fund managers. "Economic Analysis: Theory and Practice" 40 (391), p. 48-57.
E. Fedorova, A. Didenko, D. Sedykh. What impacts efficiency of pension fund managers in Russia. "Financial analytics: science and experience", ISSN 2073-4484, issue No. 33 (219) — 2014 September.
