shortfall surprise

SUMMER 2007 THE JOURNAL OF TRADING 11

In this article, we use a large sample oforder executions to better understand“shortfall surprises”: the differencebetween actual and expected trading

costs. Our trading cost measure is executionshortfall, including both liquidity impact andthe opportunity cost of slow executions.1

Exhibit 1 summarizes our main findings. Inour sample, on average, only 20% of actualshortfall is predictable pre-trade. Focusing onthe non-predictable component, we investi-gate the following possible reasons for theshortfall surprise:

• Price surprise. A higher-than-expectedprice increase over the execution horizonshould result in a higher-than-expectedactual shortfall on buy orders and lower-than-expected shortfall on sell orders.We find that, on average, the price sur-prise explains 42% of the shortfall sur-prise. For orders in large-cap stocks thattake more than an hour to execute, theprice surprise explains 73% of the short-fall surprise. Splitting the price surpriseinto its alpha and market components,the alpha surprise explains 38% of theshortfall surprise, and the market sur-prise only 4%.

• Volume surprise. Higher-than-expectedvolume over the execution horizonshould result in lower-than-expectedshortfall. Surprisingly, we find that the

volume surprise explains very little ofthe shortfall surprise (only 0.3%), indi-cating that higher-than-expected volumedoes not reduce trading costs.

• Volatility surprise. Higher-than-expectedvolatility over the execution horizonshould result in higher-than-expectedshortfall. We find that the volatility surprise explains only 0.1% of the short-fall surprise.

• Spread surprise. Higher-than-expectedquoted spreads over the execution horizonshould result in higher-than-expectedshortfall. We find that the spread surpriseexplains 1.8% of the shortfall surprise.

Price surprises, therefore, are by far themost important factor explaining shortfall surprises; the volume, volatility, and spreadsurprises have little explanatory power. Thesefindings have important implications for post-trade analysis, pre-trade tools, the developmentof algorithms, and the choice of executionstrategies.

In interpreting post-trade executionquality, for example, we must somehow con-trol for the underlying execution-horizon pricemove. For pre-trade analysis, our findings sug-gest that the only way to improve the pre-tradet-cost estimates is to better predict the alpha-move over the execution horizon. Better pre-trade volatility and volume estimates will nothelp much. The same applies for algorithm

Shortfall SurprisesKWAKU ABROKWAH AND GEORGE SOFIANOS

KWAKU ABROKWAH

is an analyst, Equity Execution Strategies, atGoldman, Sachs & Co. inNew York, [email protected]

GEORGE SOFIANOS

is a vice president, EquityExecution Strategies, atGoldman, Sachs & Co. inNew York, [email protected]

development and the choice of execution strategies; onlybetter predictors of execution-horizon alpha will signif-icantly reduce execution shortfall.

In the next section, we develop our framework forexplaining the shortfall surprise, followed by a section onour data sample and the construction of our variables.The following section presents our empirical findings. Inthe section after that, we focus on our puzzling findingthat volume surprises do not affect trading costs. The nextsection elaborates on the distinction between volatilityand price surprises. We conclude with a discussion of theimplications of our empirical findings, and possible exten-sions of our analysis.

A FRAMEWORK FOR ANALYZING THE SHORTFALL SURPRISE

We begin by formally defining execution shortfall, ourt-cost measure. For buy orders, execution shortfall is theexecution price minus the prevailing mid-quote at orderarrival (strike price) as percent of strike price.2 For sell

orders, execution shortfall is the strike price minus the exe-cution price as percent of the strike price. In Exhibit 2, weintroduce a hypothetical order execution that we will usethroughout this section to illustrate our framework. Thetrader receives an order to buy 60,000 shares. The strikeprice at order arrival is $25.00. The trader executes theorder over time in three executions and the volume-weighted execution price is $25.06. The execution short-fall in this example is six cents or 24 basis points (bps).

In addition to liquidity impact, execution shortfallincludes the opportunity cost of delayed execution. Theopportunity cost arises because the price may move awayfrom the trader over the execution horizon. This pricemove has both a market and a stock-specific (alpha) com-ponent and we estimate both. Exhibit 3 shows the threecomponents of execution shortfall: liquidity impact, alphaloss, and market loss. In our example, the trader is buyingin a rising market, so the opportunity cost is positive andincreases the shortfall. If the price were falling, the oppor-tunity cost would be negative, reducing the shortfall, andpossibly resulting in negative shortfall. Unlike liquidity

12 SHORTFALL SURPRISES SUMMER 2007

E X H I B I T 1Summary of Main Findings1

1Sample period June 1 through December 31, 2006; 241,610 orders.2R-square from univariate regression of actual on expected shortfall. 3Incremental R-square from multivariate regression analysis of shortfall surprise on the five factor surprises: volatility, spread, volume, EH-alpha and EH-market.

impact, which is never negative, execution shortfall canbe negative.3

In this article, we analyze the shortfall surprise (SS)that we define as actual shortfall (SA) minus expectedshortfall (SE):

SS = SA – SE (1)

To generate the shortfall surprise, we must first esti-mate expected shortfall. Expected shortfall is the short-fall a trader can predict with only pre-trade information:when the trader receives the order, but before execution.To estimate expected shortfall, we need estimates for bothliquidity impact (LE) and the expected underlying pricemove over the execution horizon (EH-price or PE):

SE = LE + PE (2)

We estimate liquidity impact using the GoldmanSachs t-cost model.4 The model uses the following sixfactors to predict liquidity impact:

1. Stock capitalization: large-cap, mid-cap, and small-cap stocks.

2. Listing market: NYSE, AMEX, and NASDAQstocks.

3. Order size: the actual dollar value of the order executed.

4. Volatility over the execution horizon.5. Quoted spreads over the execution horizon.6. Trading volume over the execution horizon.

The first three factors (stock capitalization, listingmarket, and order size) we know with certainty pre-trade.The other factors (volatility, spread, and volume) we donot know with certainty pre-trade, and we must use theirexpected values to estimate liquidity impact (LE):

LE = L(Y, XE) > 0 (3)

where Y are the factors known with certainty pre-trade, and XE are the factors not known with certaintypre-trade.


E X H I B I T 2Execution Shortfall

We next discuss our EH-price concept. In Exhibit 4,we assume the market component of EH-price is zeroand show how we measure the stock-specific alpha moveover the execution horizon (EH-alpha). Continuing theExhibit 2 example, the trader received the order at 12:00,executed 20,000 shares immediately, another 20,000 sharesat 13:00, and the final 20,000 shares at 14:00. The exe-cution horizon is two hours (order arrival to last execu-tion), and the volume-weighted execution turnaroundtime is one hour. The execution turnaround time is theorder’s half-life, taking into account that the order execu-tion is spread over the two-hour horizon.5 We define EH-alpha as the price move over the order’s half-life, asidefrom the liquidity impact of the trade itself.

In calculating EH-alpha, it is critical to use a post-trade price after the liquidity impact of the trade sub-sides. In our example, liquidity impact subsides at 15:00,so any price after that will work. In our empirical analysis,we use the closing price to measure the actual EH-alphafor each order in our sample. This procedure assumesthat the closing price is not affected by the impact of the

trade. Our empirical analysis below confirms this is indeedthe case.

In Exhibit 4, we show how we use the closingprice to calculate EH-alpha. We first calculate the alpha-to-close. For buy orders, alpha-to-close is the closingprice minus the strike price as percent of the strikeprice. For sell orders, alpha-to-close is the strike priceminus the closing price as percent of the strike price.The alpha-to-close in our example is 40 bps. We then“allocate” the alpha-to-close to the order in propor-tion to the order’s half-life. We derive the allocationfactor (Φ) by dividing the order’s half-life (one hour)by the time from arrival to market close (four hours).6

So, in our example, the allocation factor is 1/4 and EH-alpha is 10 bps.

In practice, the underlying price move has bothmarket and alpha components:

EH-price = EH-alpha + βEH-market (4)

where β is the intra-day stock beta.


E X H I B I T 3The Three Components of Execution Shortfall

In our empirical analysis, we use S&P 500 ETFprices to decompose the underlying price move into EH-market and EH-alpha. In Exhibit 5, continuing ourexample, the market-to-close move is 16 bps and allo-cating over the execution horizon EH-market is four bps.

Combining Equations (2) to (4) and, for illustrationpurposes, using a linear version of the liquidity impactmodel L (Y, X), expected shortfall is given by:

SE = a + bY + c XE + β ME + AE (5)

where ME is the expected EH-market and AE is theexpected EH-alpha.

Suppose the liquidity impact is 10 bps. To get thetotal shortfall, we must add the EH-price move. InExhibit 5, the EH-price move is 14 bps so the totalshortfall is 24 bps: 10 bps impact and 14 bps price loss.

We next derive the shortfall surprises factor modelwe use in our analysis. In Equation 5, replacing theexpected values of the various factors by their post-tradeactual values, the actual shortfall (SA) is given by:

SA = a + bY + cXA + β MA + AA + U (6)

Actual shortfall is also influenced by other factors,(U), not included in our empirical analysis. Most of theseother factors are random. Some of them, however, maybe systematic but difficult to quantify even post-trade; fore.g., trader skill or the presence of natural counterparties.

Subtracting expected shortfall (Equation 5) fromactual shortfall (Equation 6) we get our basic shortfall sur-prises model:

SS = c (XA – XE) + β (MA – ME) + (AA – AE) + U (7)


E X H I B I T 4Execution-Horizon Alpha

1If the time of last execution is after 16:00, it is possible for the ratio to exceed 1. In these cases (129 orders), we set the ratio to 1.

Or, writing out the factors in full:

In our analysis of the shortfall surprises, we focus onthe five input surprises: 1) the volatility surprise, 2) thespread surprise, 3) the volume surprise, 4) the market sur-prise, and 5) the alpha surprise. The expected EH-marketand EH-alpha are hard to estimate and, in our empiricalanalysis, we assume they are zero. We describe the sur-prises in greater detail in the next section. Note that inthe surprises model, the factors known with certaintypre-trade (Y) drop out. Also, because of the way we con-structed EH-price, the coefficient of the alpha surprises

S = c (VOL – VOL )+s1

A E

Volatility surprise�

cc (SPREAD – SPREAD )2A E

Spread surprise�

+ c (VLM –VLM )+ (3A E

Volume surprise�

β EEH-market –EH-market )A E

Market surprise�

+(EH-alpha – EH-alpha )A E

Alpha surpri� sse

+ U (8)

in the model is one, and the coefficient of the market sur-prise is β, the intra-day stock beta.

In deriving our shortfall surprise model (8), we assumethat the liquidity impact model we use to generate expectedimpact is correctly specified. But the model may be mis-specified; we may have omitted systematic variables (U inour specification), or the model coefficients may be biased.Model misspecification creates another possible source ofshortfall surprises. In the Appendix, we discuss this issuemore formally and decompose shortfall surprises into amisspecification component and the input surprises com-ponent. In our empirical analysis, we carefully constructedour sample to minimize the risk of model misspecification.

SAMPLE CONSTRUCTION AND DESCRIPTION

Our final estimation sample consists of 241,610 clientorders executed by Goldman Sachs over seven months,


E X H I B I T 5The Market and Alpha Components

June through December 2006. To minimize the risk ofmodel misspecification, we restricted our sample to closelymatch the sample we use in estimating the Goldman Sachst-cost model. Our final sample consists of market, not-held orders, executed on an agency basis by the GoldmanSachs U.S. single-stock trading desk.7 We also aggressivelyfiltered unusual orders; we dropped odd lots, kept onlyorders in common stocks, dropped ADRs and ETFs,OTCBB and Pink Sheets, dropped crosses, dropped ordersin extreme price stocks (less than $1 or more than $150),dropped tick sensitive orders (e.g., sell short), etc. Finally,we dropped orders with data errors.

Exhibit 6 summarizes the order composition of thefinal sample. The 241,610 orders in our sample are evenlydivided between buys and sells. One quarter of the ordersare NASDAQ stocks8 and the balance are NYSE stocks.9

Of the total orders, 64% are in large-cap stocks and 9%in small-cap stocks. Only 296 orders exceed 25% ofaverage daily volume, and 89% of the orders are less than10,000 shares. The majority of the orders (92%) have anexecution half-life less than 15 minutes; 13,464 ordershave a half-life more than 30 minutes. The average stockprice in our sample is $34.

Exhibit 7 summarizes order characteristics for theoverall sample. The average order size executed is 9,420shares, ranging from 18 shares to 29 million shares. Thevalue-weighted average order size executed is 5% of ADVand the participation rate is 21%.10 The value-weightedhalf-time is 65 minutes, but the median is only 15 sec-onds and ranges from instantaneous executions to all-dayexecutions. The value-weighted average actual shortfallis 23 bps, but the median is only 2 bps. Actual shortfallranges widely from –757 bps to +1,069 bps.

Exhibit 8 confirms that, at least on average, theclosing prices we use in calculating the EH-price move arenot affected by liquidity impact. In Exhibit 8, we plot theaverage shortfall, same-day closing price and the closingprices over the next five days for all the orders in oursample. We measure all prices relative to the order-arrivalprice, and include both buy and sell orders, but flip the signof the sells. We use value-weighted averages, so large tradesget more weight in the average. The average closing priceis 26 bps higher than the strike price, but there is littlereversal over the next five days. The absence of reversalreassures us that the closing price is not affected by thetemporary liquidity impact of the trades themselves.

We next describe how we construct the shortfallsurprises and the five factor surprises we use in our analysis.

For each order in our sample, we use the Goldman Sachst-cost model to estimate the expected liquidity impact.The model is re-estimated monthly using the most recentnine months of data. For each order, we use the mostrecent version of the model as of the date of the order togenerate out-of-sample estimates of that order’s expectedimpact. For an order received on July 7, for example, weuse the model estimated with data prior to July 7. Themonthly re-estimation of the model ensures the coeffi-cients are not stale and again reduces the risk of modelmisspecification.

The Goldman Sachs t-cost model gives differentimpact estimates depending on the execution horizonspecified. In our analysis, we estimate the expected impactusing the order’s actual execution horizon: order arrival


E X H I B I T 6Order Types

Sample period June 1 through December 31, 2006.

to last execution. To generate our impact estimates, wemust also specify pre-trade estimates for volatility, quotedspreads and trading volume:

• Volatility is measured as the percent differencebetween the intra-day high and low price, adjusted

to the order’s execution horizon, by using the sameallocation factor Φ that we use in calculating EH-alpha (Exhibit 4).

• Quoted spread is the time-weighted spread over theexecution horizon, taking into account the spreadsmile (the U-shaped intraday spread pattern).11


E X H I B I T 8Closing Prices are Not Affected by Liquidity Impact

Sample period June 1 through December 31, 2006; 241,610 buy and sell orders (Sign of sell orders flipped). Value-weighted averages.

E X H I B I T 7Overall Sample Summary Statistics1

1Sample period June 1 through December 31, 2006; (241,610 orders).2Simple mean.3Value-weighted mean.

• Trading volume is measured over the executionhorizon, taking into account the volume smile (theU-shaped intraday volume pattern).

We generate pre-trade estimates for these threefactors using their median value over the prior 21trading days.

To estimate expected shortfall (Equation 5), we alsoneed pre-trade estimates for EH-alpha and EH-market.Unlike liquidity impact, where reliable pre-trade esti-mates are widely available, pre-trade estimates of EH-market and EH-alpha are difficult to find. As is typicallythe case in practice, in our analysis we assume that pre-trade, the expected EH-market and expected EH-alphaare both zero.

Exhibit 9 shows a scatter plot of actual shortfall (ver-tical axis) against expected shortfall (horizontal axis).Actual shortfall equals expected shortfall along the 45 degree line. Orders with actual shortfall exceedingexpected shortfall are above the 45 degree line and orderswith actual shortfall less than expected shortfall are below.Along the vertical, we see the large range in actual short-fall from a minimum of –757 bps to a maximum of +1,069bps. Along the horizontal, we see the range in expectedshortfall estimates from a minimum of 0.3 bps to a max-imum of 334 bps. Because we assume expected EH-market and EH-alpha are zero, the expected shortfall isalways positive (just the liquidity impact).

The number in each bubble indicates the numberof orders clustering in close proximity to that point. About


E X H I B I T 9Scatter plot of Actual Vs Expected Shortfall

Sample period June 1 through December 31, 2006; (241,610 orders).

0 50 100 150 200 250 300

210,000 orders are clustered between –60 and +35 bps ofactual shortfall and between 0 and 15 bps of expectedshortfall. The remaining orders show a large dispersion,both in the expected and actual shortfall. Our objectivein this article is to better understand this dispersion: theshortfall surprise.

Exhibit 10 summarizes the post-trade actual valuesof the five factors we use to explain the shortfall surprise.All five factors range widely, mirroring the large range ofactual shortfall. Volatility, for example, ranges from 0% to20% and quoted spreads from 0 bps to 882 bps. The actualEH-market move ranges from –119 bps to +121 bps, butaverages to zero by all measures (median, simple, andweighted means). The actual EH-alpha, however, has amuch bigger range (from –557 bps to +836 bps) and thevalue-weighted mean is 10 bps.

In our shortfall surprises model (Equation 8), theshortfall surprise depends on the five factor surprises.Exhibit 11 summarizes our estimates of the shortfall sur-prise and the five factor surprises. The first column showsthe forecasting accuracy of our pre-trade measures and givesan indication of the potential explanatory power of eachsurprise. If, for example, a factor is perfectly forecasted,then there is no factor surprise and that factor cannot explainthe shortfall surprise. We measured forecasting accuracy byestimating univariate regressions of actual values on expected

values and report the unexplained variation (one minus R-square). The unexplained variation tells us how muchof the actual variation in each variable is not explained bythe expected value of that variable.

Exhibit 11 shows that 80% of the actual shortfallvariation is not explained by expected shortfall (or con-versely, that the pre-trade estimates explain only 20% ofthe post-trade variation in actual shortfall). Expectedvolatility leaves only 23% of the variation in actualvolatility unexplained. Expected spreads and volumeleave about 50% of the post-trade variation unexplained.Since we assume they are zero, expected EH-market andEH-alpha leave 100% of the variation in the actual EH-alpha and EH-move unexplained. The EH-market andAH-alpha surprises, therefore, have great power topotentially explain the shortfall surprise. Exhibit 11 alsoshows the large range of all the surprises, another mea-sure of forecasting accuracy. In general, all five factorshave large surprises and, therefore, potentially largeexplanatory power in our shortfall surprise regressions,to which we now turn.

THE EMPIRICAL FINDINGS

Our preferred regression specification for the short-fall surprise (SS) model is:


E X H I B I T 1 0The Five Factors: Actual Values1

1Sample period June 1 through December 31, 2006; (241,610 orders).2Simple mean.3Value-weighted mean.

We experimented with many alternative specifica-tions (for e.g., with and without intercepts, different func-tional forms for the five factors, etc.) and, in all cases, theresults are similar.12

Exhibit 12 shows the results from the estimation ofour preferred specification over our whole sample of241,610 orders. The overall fit of the regression is 44%(R-square), and the coefficients of all five factor surprisesare significant and have the right sign. Higher-than-expected volatility and spreads lead to higher-than-expected shortfall while higher-than-expected volumeleads to lower-than-expected shortfall. Similarly, higher-than-expected market and alpha moves result in higher-than-expected shortfall. Our hypothesis is that thecoefficient of EH-alpha in the regression should be one,and Exhibit 12 shows that this is indeed the case; a fivebasis-point increase in EH-alpha will lead to a five

S = c + c (VOL – VOL )

+ c (S

s0 1

A E

2

Volatility surprise�

PPREAD – SPREAD )+ c logVLM

VLA E

3

A

Spread surprise� MM

+ c (EH-market – EH-m

E

4A

Volume surprise

⎛⎝⎜

⎞⎠⎟

�

aarket

+ c (EH-alpha – EH-alp

E

5A

Market surprise�)

hha )E

Alpha surprise�

basis-point increase in the actual shortfall. The coefficientof EH-market in our regression is an estimate of the short-term intra-day beta on our sample; the results in Exhibit12 suggest that this is also close to one.13

To better understand the relative importance of thefive factor surprises in explaining the shortfall surprise,we also ran a series of univariate regressions of the short-fall surprise on each of the factor surprises in turn. InExhibit 13, columns four to eight, we report the R-squaresfrom these univariate regressions. The first row shows theresults of running the regressions on all the orders in oursample. In the univariate regressions, the EH-alpha sur-prise is by far the most important factor explaining almost40% of the shortfall surprise variation. The other fourfactors have little explanatory power; EH-market explains5.8%, volatility explains 3.0%, spreads explain 1.5% and,most surprising, volume only explains 0.2% of the short-fall surprise.

Exhibit 14 shows that the covariances across thefive factor surprises, while small, are not always zero.The correlation between the EH-alpha and volatilitysurprises, for example, is 25% and the correlationbetween the volume and volatility surprises is 17%.Because the covariances across the factor surprises are notzero, the univariate regression results can be misleading.To double check, we also performed an incrementalanalysis, where we evaluate the contribution of each


E X H I B I T 1 1The Five Factor Surprises: Actual Minus Expected Values1

1Sample period June 1 through December 31, 2006; 241,610 orders.2One minus the R-square from univariate regression of actual to expected.

factor surprise by dropping it from the full multivariateregression. The overview results in Exhibit 1 are fromthe incremental approach and are similar to the uni-variate regression results.

The last three columns in Exhibit 13 show anotherexample of the incremental approach. Column 10 showsthe full five-factor regression, and the regression in columnnine drops the two EH-price surprises (market and alpha).Dropping the EH-price surprises reduces the explanatorypower of the regressions from 44% to only 4.5%, con-firming that the price surprises account for almost all ofthe explanatory power in our shortfall surprise regressions.

In Exhibit 13, rows two to four, we divide oursample by stock capitalization: orders in large-cap stocks(>$7.5 billion), mid-cap stocks, and small-cap stocks (<$1 billion). As we move from large-cap to small-capstocks, the accuracy of the expected shortfall forecasts asmeasured by the root mean-square error (RMSE) falls.14

The expected shortfall RMSE is 13 bps for orders in large-cap stocks, but 49 bps for orders in small-cap stocks, indi-cating that the shortfall surprise is higher in small-capstocks.

Our findings on the five factor surprises are similaracross stock capitalization buckets. In all three buckets,


E X H I B I T 1 2The Shortfall Surprise Multivariate Regression1

1Sample period June 1 through December 31, 2006; t-statistics in brackets below the coefficients.

E X H I B I T 1 3Shortfall Surprise Attribution: An Overview1

1Sample period June 1 through December 31, 2006.2Root mean-square error: the square root of 1/N Σ(SAi – SEi)2, where SAi is the post-trade actual shortfall and SEi is the pre-trade expected shortfall for each order.3The average executed order size in this bucket is 775 shares.

the EH-price surprise explains most of the shortfall sur-prise. The volatility and spread surprises explain more ofthe shortfall surprise in small-cap stocks than in the othertwo buckets, but the numbers are still small. In the uni-variate regressions, for example, the spread surpriseexplains 4.3% of the shortfall surprise in small-cap stocks,but only 0.1% of the surprise in large-cap stocks. Thevolume surprise explains very little of the shortfall surpriseacross all stock capitalizations.

Because we construct EH-price using closing prices,one concern is that the closing prices are affected by liq-uidity impact and therefore are not a pure measure of theunderlying price move. In Exhibit 9, we showed that, atleast on average, the closing prices in our sample are notaffected by liquidity impact. The liquidity impact of thelarger trades in our sample is more likely to affect theclosing price.15 So, in Exhibit 13, rows five to six, we fur-ther test the sensitivity of our results to the quality of theclosing price by dropping the larger trades in our sample.In row five, we drop all orders greater than 25% of ADV(297 orders) and in row six, we drop all orders greaterthan 15% of ADV (1,228 orders). In both cases, droppingthe outlier large orders does not affect our results.

In the last row in Exhibit 13, we focus on the smallorders in our sample: 177,191 orders less than 0.1% ADV(average order size 775 shares). As expected, for thesesmall orders, the shortfall forecasting error is low: theRMSE is only 7 bps compared to 23 bps overall. For thesesmall orders, the five factor surprises explain only 17% of

the much smaller shortfall surprise. But still, across thefive factor surprises, the EH-price surprise is by far themost important factor in explaining the shortfall surprise:volatility, spread, and volume surprises explain 2.3%, whileEH-price explains 14.7%.

In Exhibit 15, we focus on the time dimension oforder executions. A quick execution exposes the orderto little EH-price movement, so we expect the EH-pricesurprise to explain less of the shortfall surprise. At theother end, an order that takes several hours to execute isexposed to potentially large price moves, so we expectEH-price to be much more important in explaining theshortfall surprise. The results in Exhibit 15 confirm thishypothesis.

Exhibit 15 is divided in four panels: Panel A coversorders in all stocks, Panel B focuses on orders in large-capstocks, Panel C focuses on mid-cap stocks and Panel Dfocuses on small-cap stocks. The top row in each panelreplicates the information in Exhibit 13. The next tworows in each panel divide the orders into orders with half-life more than 30 minutes and orders with half-life less than30 minutes. The last two rows in each panel repeat thissplit, using a 60-minute cut-off.

The first thing to notice in Exhibit 15 is that theforecasting accuracy of expected shortfall falls sharply asthe execution horizon increases (column 4). In Panel A,for example, the RMSE for orders with half-life morethan 60 minutes is 89 bps compared to only 15 bps fororders with half-life less than 30 minutes. For orders in


E X H I B I T 1 4Covariance of the Five Surprises1

1Sample period June 1 through December 31, 2006; (241,610 orders).2Log of ratio of actual to expected volume.

small-cap stocks (Panel D), the RMSE for orders withhalf-life more than 60 minutes is even higher: 118 bps.The forecasting accuracy results in Exhibits 13 and 15strikingly quantify what most traders know from experi-ence: most of the shortfall surprise comes from large ordersthat take time to execute.

In Exhibit 15, column 11 shows the explanatorypower of the full five-factor regression model. Across thevarious buckets, the R-square of the full model rangesfrom 75.3% (orders >60 minutes, large-cap stocks) to8.0% (orders <30 minutes, large-cap stocks). We nextexamine the contribution of the five-factor surprises inexplaining the shortfall surprise. The last column inExhibit 15 shows that, consistent with our hypothesis,the EH-price surprise is much more important inexplaining the shortfall surprise for slow executions. In

Panel A, for example, EH-price explains 63.5% of theshortfall surprise in orders with half-life more than 60minutes, but only 6.3% of the surprise for orders withhalf-life less than 30 minutes. For orders in large-cap stockswith half-life more than 60 minutes, EH-price explainsa remarkable 73% of the shortfall surprise.

Looking at the two components of EH-price(market and alpha), the results in Exhibit 15 across allbuckets are consistent with the results in Exhibit 13: EH-alpha is by far the most important factor in explainingthe shortfall surprise. For orders in large-cap stocks withhalf-life more than 60 minutes, for example, EH-alphaexplains 65.3% of the shortfall surprise while EH-marketexplains only 12.5%. For orders with half-life less than30 minutes, EH-alpha explains 6.1% of the shortfall sur-prise while EH-market explains only 0.8%.


E X H I B I T 1 5The Shortfall Surprise Attribution: Details1

1Sample period June 1 through December 31, 2006.

The summary statistics on EH-alpha and EH-marketin Exhibit 10 provide a clue why the alpha componentdominates: EH-alpha has a bigger range (–557 bps to+839 bps) than EH-price (–119 bps to +121 bps). Outof the 241,610 orders in our sample, 3,173 orders areassociated with an EH-alpha move exceeding 50 bps (inabsolute terms), but only 407 orders are associated withan EH-market move exceeding 50 bps.

Exhibit 15, column 10, confirms the robustness ofour findings on the volatility, spread, and volume sur-prises: consistently small explanatory power across allbuckets. Finally, Exhibit 15, column three, confirms therobustness of the absence of any volume effect. Across allbuckets, the volume surprise explains less than 1% of theshortfall surprise.

Perhaps the most striking observation aboutExhibits 13 and 15 is the robustness of our main results.Whichever way we cut the data, EH-alpha is by far themost important factor in explaining the shortfall surprise;while volatility, spread, and volume surprises explain verylittle. The volume results are particularly surprising and,in the next section, we take a closer look.

THE VOLUME SURPRISE

Our most puzzling finding is that volume surprisesdo not affect trading costs. This seems counterintuitive:more volume than anticipated should surely reduce tradingcosts. This intuition, however, rests on the presumptionthat higher volume is caused by an increase in the supplyof liquidity (Case A in Exhibit 16). Our empirical find-ings suggest that an increase in volume is caused by anincrease in both liquidity supply and demand (Case C inExhibit 19): a hundred more traders supply liquidity, butanother hundred traders demand more liquidity. Anincrease in uncertainty, for example, will lead to an increasein both liquidity supply and demand. Similarly, the pop-ular “participate” execution strategies and algorithmsautomatically generate liquidity demand in proportion toincreases in liquidity supply.

In our multivariate regressions where we controlfor the variation in all five factors, all else being equal,the volume surprise reduces trading costs, but by a tinyamount; for e.g., volume twice the expected amountreduces trading costs by less than 0.3 bps. In the uni-variate regressions of shortfall surprise on volume sur-prise, a surprise increase in volume actually increasestrading costs (Case B in Exhibit 16). Equivalently, the

simple correlation between the volume and shortfall sur-prises is small, but positive, 4%. The striking differencebetween the multivariate and univariate results suggests thatother factor surprises correlated the volume surprise (fore.g., volatility and EH-alpha in Exhibit 14), and increasecosts outweighing the beneficial effects of higher-than-expected volume.

We performed extensive additional robustness testsof our finding on the volume surprise. In all cases, volumesurprises explain very little of the shortfall surprise. InExhibit 17, for example, we focused on high-volume sur-prise events: 20,116 orders in our sample associated witha volume surprise greater than one standard deviation.Again, there is little change: volume surprises explain only1.4% of the shortfall surprise and the EH-price moveexplains 60.2%.

The important implication of our volume findingsis that traders and algorithms cannot naively exploitvolume surprises to reduce trading costs. To beneficiallyexploit a volume surprise, execution strategies must dis-tinguish between liquidity supply and liquidity demandsurprises, which is hard to do. Taking advantage ofhigher-than-expected volume has one unambiguousbeneficial effect: traders fill orders faster reducing exe-cution risk.

THE VOLATILITY SURPRISE

Another empirical finding worth highlighting is thesharp distinction between the small effect on trading costsof volatility surprises and the large effect of EH-price sur-prises. The important distinction here is that volatilitysurprises are unsigned price moves while EH-price sur-prises are signed price moves, depending on the directionof the trade.16 For buy orders, EH-price is closing priceminus strike price, while for sell orders, EH-alpha is strikeprice minus closing price (see Exhibit 18).

The unsigned EH-price is just another measure ofexecution-horizon volatility: closing price minus strikeprice (as percent of strike price) for both buys and sells.Let’s call this measure of volatility EH-volatility. InExhibit 18, we compare the results of regressing the short-fall surprise on the signed and unsigned EH-price. Inthe univariate regression of shortfall surprises on signedEH-price, EH-price explains more than 40% of the vari-ation in the shortfall surprise. However, in the univariateregression of the shortfall surprise on EH-volatility (theunsigned EH-price), the unsigned EH-price explains less



E X H I B I T 1 6The Volume Surprise: Why It Does Not Matter?

E X H I B I T 1 7High Volume Surprises1

1Sample period June 1 through December 31, 2006.2|VLMA – VLME| > one standard deviation.

than one percent of the variation in the shortfall surprise. We find similar results in a multivariate regres-sion of the shortfall surprise on both the signed andunsigned EH-price: all the explanatory power comesfrom the signed price move.

What is striking about the analysis in Exhibit 18 isthat, by design, EH-price and EH-volatility are exactlythe same measure, except that one is signed based on thedirection of the trade and the other is not. The impor-tant implication of our findings in this section is thattraders and algorithms cannot exploit volatility surprisesto reduce trading costs unless they can accurately predictthe direction of the price move.17

IMPLICATIONS AND CONCLUSION

Our empirical findings have important implicationsfor post-trade execution quality analysis (EQA) and forpre-trade tool, for the development of algorithms, andfor the choice of execution strategy. We conclude with adiscussion of these implications.

Post-trade EQA

Our finding that price surprises are the main causeof shortfall surprises means that, in interpreting post-tradeEQA, it is essential to control for the underlying execu-tion-horizon price move. Our preferred way of doingthis is by reporting the EH-price move alongside theactual shortfall, or equivalently, by using the EH-price tosplit actual shortfall into liquidity impact and price loss.18

Suppose actual shortfall is 20 bps and the EH-price is 18bps: the liquidity impact is only 2 bps, so this is good exe-cution. If the EH-price is 4 bps, however, 20 bps is poorexecution (16 bps liquidity impact). We avoid using theusual VWAP benchmarks to control for price surprises,because VWAP benchmarks may be affected by the impactof the trade itself and may give misleading results.19

The importance of price surprises also suggests thatpost-trade EQA to evaluate execution strategies and algo-rithms can only be meaningfully done with a large sampleof orders. In small samples, extraneous price surprises willdominate the statistics and obscure the true performancebeing evaluated.


E X H I B I T 1 8The Volatility Surprise

1Sample period June 1 through December 31, 2006.

Pre-trade Tools and Development of Algorithms

Trading cost models are widely used to generatepre-trade cost estimates and choose the right executionstrategy. Cost models are also embedded in various algo-rithms.20 These trading-cost models have relatively lowforecasting power and model developers are trying hardto improve them. Our findings suggest that the only wayto make a significant difference in forecasting accuracy isby coming up with better forecasts of EH-price. Betterforecasts of volume and volatility, in particular, will notimprove the forecasting accuracy of trading cost models.

But how can we come up with better EH-priceforecasts? One way is to continue improving models forshort-term price forecasts. Work is being done on this frontin the context of short-term quantitative trading. Thechallenge here is that a better short-term price forecastingmodel is more valuable as a proprietary trading tool, ratherthan a widely available pre-trade tool. But the synergiesexist and the possibility of combining short-term tradingand improved execution strategies within a buy-sidetrading desk is intriguing.

Another way to improve the accuracy of shortfallestimates is by allowing users to adjust the estimates basedon the user’s view of the expected EH-price. Sell-sidetraders may have a good “feel for the market,” so allowingthem to adjust cost estimates accordingly can be useful.Similarly, buy-side traders may have a better understandingof portfolio managers’ intentions and alpha signal. TheGoldman Sachs PortX algorithm, for example, allowstraders to input their alpha-to-close expectation and usesthis input to adjust the pre-trade cost estimates, and theoptimum execution strategy, accordingly.

Our findings also have important implications for thedevelopment of dynamic algorithms that react real-timeto market changes. For a dynamic algorithm to usefullyreact to a volume surprises, it must be able to distinguishbetween supply and demand causes of the surprise, whichis very challenging. Similarly, dynamic algorithms cannotusefully exploit volatility surprises, unless they can pre-dict the direction of the market.

Choice of Execution Strategy

Our findings in this article strikingly quantify oneof the main themes in our research over the past fiveyears:21 in choosing an execution strategy that takes more

than a few minutes to complete, by far the most impor-tant factor is the expected underlying price move (EH-alpha). But is there any empirical evidence that traderscorrectly anticipate EH-alpha and adapt their executionstrategies accordingly? In Rakhlin and Sofianos [2006a],we found no evidence that buy-side traders allocatedorders across passive VWAP algorithms and more aggres-sive shortfall algorithms based on expected EH-alpha. But that article was only the beginning of an ongoingresearch effort.

If traders cannot correctly anticipate EH-alpha, thenexecution strategies should be modified to reflect this fact.Segmenting orders between passive and aggressive exe-cutions, for example, requires reliable EH-alpha forecasts.In the absence of reliable EH-alpha forecasts, adopting asingle, relatively passive, execution strategy may work best.The downside of a passive strategy is high execution risk(a large variation in actual shortfall). If execution risk isa concern, then an alternative strategy is to let the trader’srisk aversion drive the choice of execution strategy, andaccept a higher liquidity impact. Yet another alternativeis to request capital and pay a premium to completelyeliminate execution risk.

The shortfall surprises framework that we devel-oped in this article can be used more generally to quan-tify and compare the value-added of alternative executionstrategies. By analyzing shortfall surprises, instead of actualshortfall, we control for the usual factors affecting exe-cution quality and can focus on the incremental value ofadditional factors. Consider again the comparison betweenpassive VWAP and aggressive shortfall algorithms. VWAPalgorithms should outperform shortfall algorithms whenEH-price is low and the reverse when EH-price is high.Using our shortfall surprises framework, for VWAP exe-cutions, the shortfall surprise should be negative (actualless than expected) when EH-price is low and positivewhen EH-price is high and the reverse for the shortfallexecutions. In doing this comparison, we do not have tocontrol for order size, stock capitalization, or the other“usual suspects,” because expected shortfall already reflectsthese factors.

We can also use the shortfall surprises frameworkto quantify the value-added of high-touch executions.The rise of low-touch trading has created a pressing ques-tion: why pay more than 12 bps in commissions to usethe high-touch if one can execute for less than 3 bps incommissions low-touch? The answer, of course, is that fordifficult orders, high-touch executions may add value by


reducing the indirect costs of trading: liquidity impact andopportunity cost. But to decide between high-touch andlow-touch, we must explicitly quantify high-touch value-added, and determine the order difficulty cut-off, abovewhich it is worth executing high-touch. In future work,we plan to use our shortfall surprises framework to addressthese questions.

A P P E N D I X

MODEL MISSPECIFICATION AND THE SHORTFALL SURPRISE

In this Appendix, we formally decompose the shortfall sur-prise into the surprise caused by model misspecification (e.g.,omitted factors), and the component caused by the surprise in theincluded factors. Suppose the true model for actual shortfall is:

SA = α + β Y + γ XA + δ Z + PA + u

where Y = factors known with certainty pre-trade, XA =actual values of factors not known with certainty pre-trade,Z = factors not included in the liquidity impact model, PA =actual execution-horizon price move, and u is a randomvariable.

Suppose the estimated liquidity impact model is:

L = a + b Y + c X

Post-trade, we can generate two estimates from the impactmodel. One using the expected values of the X and P factors:

SE = a + b Y + c XE + PE

The other using the actual values of the X and P factors(perfect-foresight estimates):

SEA = a + b Y + c XA + PA

The shortfall surprise is SS = SA – SE, and by adding andsubtracting SEA, we decompose SS into two components, SS1

and SS2:

SS = ( SA – SEA) – (SEA – SE) = SS1 – SS2

But

SS1 = SA – SEA = (α + β Y + γ XA + δ Z + PA + u )

– (a + b Y + c XA + PA)

Re-arranging:

SS1 = (α – a) + (β - b) Y + (γ – c) XA + δ Z + u

If the impact model is not misspecified a = α, b = β, c = γ, δ =0, and SS1 = u, a random variable.

The shortfall surprise component SS1, therefore, mea-sures the extent the shortfall surprise SS is caused by other fac-tors (model misspecification). Similarly:

SS2 = SEA – SE = c (XA – XE) + (PA – PE)

The shortfall surprise component, SS2, therefore, mea-sures the extent the shortfall surprise SS is caused by the factorsurprises (what we are testing in this article).


E X H I B I T A 1Further Disaggregating the Shortfall Surprise1

1Sample period June 1 through December 31, 2006; 241,610 orders.

Using the Goldman Sachs t-cost model and the data inour sample, we estimated both SE and the perfect foresightshortfall SEA. We then decomposed the shortfall surprise intothe other-factors component (SS1), and the factor-surprise com-ponent (SS2), and estimated the shortfall surprise model only onthe SS2 component.

Exhibit A1 shows our empirical results using this decom-position. Our estimated other-factors component SS1 accountsfor 56% of the shortfall surprise, exactly matching the “unex-plained” component in our one-step procedure in Exhibit 1.In the regression of the factor-surprises component SS2, on thefive factors, we get an R-square of 93%, and the overallexplanatory power of each of the five surprises is similar tothe results from our one-step procedure (Exhibit 1). Forexample, the alpha surprise explains 80% of the variation inSS2, which itself accounts for 44% of the overall shortfall-sur-prise variation; so in the two-step approach, the alpha sur-prise explains 35% of the overall shortfall compared to 38%in the one-step procedure.

ENDNOTES

The authors thank Jeff Bacidore, Tianwu (Michael) Cai,Barbara Dunn, Oliver Hansch, Kilian Mie, and Ingrid Tierens,all at Goldman, Sachs & Co., for their comments.

1See Sofianos [2006] for a discussion of execution shortfall.2Orders that arrive before the market opens (9:30 am),

we strike at the opening price.3Liquidity impact will never be negative for liquidity-

seeking orders. Liquidity-providing orders, for e.g., limit orders,conditional on execution, may have a negative impact (save thespread).

4For more details on the model, see Rodella [2005].5Only if the trader had executed all 60,000 shares at 14:00

would the execution horizon and the execution turnaroundtime be the same (two hours).

6This allocation procedure assumes the alpha is growinglinearly from order arrival to market close.

7We exclude limit orders, held orders, and any orders whereGoldman Sachs provided capital at the request of the client.

8NASDAQ GS, NMS, and Small Cap.9The sample also includes a few AMEX stocks.10We define participation rate as the executed quantity as

a percent of the actual overall market trading volume in thatstock from strike time (order arrival time) to the time of thelast execution.

11We use only firm and valid NBBO quotes in our calculation.

12We tried, for example, several different ways of speci-fying the volume surprise: actual minus expected volume, thelog of actual divided by expected (the specification we use inour analysis), the actual participation minus the expected par-

ticipation, etc. Moving expected shortfall to the right hand sideof the regressions also does not make much difference; it has acoefficient of one.

13These results on the coefficients of EH-market and EH-alpha are robust across all specifications and sub-samples that weestimated.

14We define the RMSE as the square root of 1/N ∑(SAi –

SEi)

2, where SA is the post-trade actual shortfall and SE is the pre-trade expected shortfall for each order.

15At the same time, however, the larger the trade, thehigher the likelihood that the trade was generated by a strongalpha signal (the true EH-alpha is higher).

16For further discussion of this important point, seeRakhlin and Sofianos [2006], pp. 43–45.

17In this article, we focus on agency executions. Volatilitysurprises may be more important for capital commitment(principal) trades. In a capital commitment trade, the dealerprices volatility (execution risk) in the price premium/dis-count. So an increase in volatility will make principal tradesmore expensive.

18For examples of our use of the decomposition of short-fall into liquidity impact and price loss, see Cai and Sofianos[2006] and Rakhlin and Sofianos [2006].

19For further discussion of this point, see Sofianos [2006].20For example, the Goldman Sachs single-stock 4Cast

and portfolio PortX algorithms. See Rakhlin and Sofianos[2006a, 2006b] for descriptions of these two algorithms.

21Bacidore and Sofianos [2002, 2003] through Cai andSofianos [2006].

REFERENCES

Bacidore, Jeffrey and George Sofianos. “Evaluating ExecutionQuality for Large Orders.” Goldman Sachs Trading and MarketStructure Report (2002).

——. “Choosing the Best Order Execution Strategy.” GoldmanSachs Trading and Market Structure Report (2003).

Cai, Tianwu (Michael) and George Sofianos. “Multi-day Exe-cutions.” Journal of Trading, Vol. 1, No. 3 (2006), pp. 25–33.

Rakhlin, Dmitry and George Sofianos. “The Choice of Exe-cution Algorithm: VWAP or Shortfall.” Journal of Trading, Vol. 1, No. 1 (2006a), pp. 26–32.

——. “The Impact of an Increase in Volatility on TradingCosts.” Journal of Trading, Vol. 1, No. 2 (2006b), pp. 43–50.

Rodella, Elena. “Introducing Cost Wizard: Comparing Two-Hour and All-Day Executions.” Goldman Sachs Cost WizardReports, Issue 1 (May 12, 2005).


Sofianos, George. “Execution Benchmarks: VWAP or Pre-trade Prices.” Journal of Trading, Vol. 1, No. 1 (2006), pp. 22–25.

To order reprints of this article, please contact Dewey Palmieri [email protected] or 212-224-3675

This material was prepared by the Goldman Sachs Equity ExecutionStrategies Group and is not the product of Goldman Sachs Global InvestmentResearch. It is not a research report and should not be construed as such.

The information in this article has been taken from trade data andother sources we deem reliable, but we do not represent that such informa-tion is accurate or complete, and it should not be relied upon as such. Thisinformation is indicative, based on among other things, market conditions atthe time of writing, and is subject to change without notice. Goldman Sachs’

algorithmic models derive pricing and trading estimates based on historicalvolume patterns, real-time market data, and parameters selected by the GSATuser. The ability of Goldman Sachs’ algorithmic models to achieve the per-formance described in this article may be impacted by changes in market con-ditions, systems or communications failures, etc. Finally, factors such as orderquantity, liquidity, and the parameters selected by the GSAT user, may impactthe performance results.

The opinions expressed in this article are those of the authors and donot necessarily represent the views of Goldman, Sachs & Co. These opinionsrepresent the authors’ judgment at this date and are subject to change.

Goldman, Sachs & Co. is not soliciting any action based on this article.It is for general information and does not constitute a personal recommenda-tion or take into account the particular investment objectives, financial situa-tions, or needs, of individual users. Before acting on any advice orrecommendation in this article, users should consider whether it is suitable fortheir particular circumstances.

Copyright: Summer 2007, Goldman, Sachs & Co.


shortfall surprise

Documents