Revisiting leverage

Many charities aim to influence how others (other donors, governments, or the private sector) allocate their funds. We call this influence on others “leverage.” Expenditure on a program can also crowd out funding that would otherwise have come from other sources. We call this “funging” (from “fungibility”).

In GiveWell’s early years, we didn’t account for leverage in our cost-effectiveness analysis; we counted all costs of an intervention equally, no matter who paid for them.¹For example, see row 3 of our 2013 cost-effectiveness analysis for Against Malaria Foundation. For example, for the Schistosomiasis Control Initiative (SCI), a charity that treats intestinal parasites (deworming), we counted both drug and delivery costs, even when the drugs were donated. We did this because we felt it was the simplest approach, least prone to significant error or manipulation.

Over the last few years, our approach has evolved, and we made some adjustments for leverage and funging to our cost-effectiveness analyses where we felt they were clearly warranted.

In our top charities update at the end of 2017, we made a major change to how we dealt with the question of leverage by incorporating explicit, formal leverage estimates for every charity we recommend.

This change made our cost-effectiveness estimates of deworming charities (which typically leverage substantial government funding) look more cost-effective than our previous method. For example, our new method makes SCI look 1.2x more cost-effective than in the previous cost-effectiveness update. More details are in the table at the end of this post.

We also think the change makes our reasoning more transparent and more consistent across organizations.

In this post, we:

• Describe how our treatment of leverage and funging has evolved.
• Highlight two major limitations of our current approach.
• Present how much difference leverage and funging make to our cost-effectiveness estimates.

Details follow.

How our thinking has evolved

We last wrote about our approach to leverage and funging in a 2011 blog post. In short, we didn’t explicitly account for leverage in our cost-effectiveness analysis, counting costs to all entities equally. We concluded:

When we do cost-effectiveness estimates (e.g., “cost per life saved”) we consider all expenses from all sources, not just funding provided by GiveWell donors. For SCI, we count both drug and delivery costs, even when drugs are donated. (Generally, we try to count all donated goods and services at market value, i.e., the price the donor could have sold them for instead of donating them.) For [the Against Malaria Foundation (AMF)], we count net costs and distribution costs, even though AMF pays only for the former. In the case of VillageReach, we even count government costs of delivering vaccines, even though VillageReach works exclusively to improve the efficiency of the delivery system.

We consider this approach the simplest approach to dealing with the issues discussed here, and given our limited understanding of how “leverage” works, we believe that this approach minimizes the error in our estimates that might come from misreading the “leverage” situation. As our understanding of “leverage” improves, we may approach our cost-effectiveness estimates differently.

Since 2011, our thinking changed. Over time, we started applying some adjustments to our cost-effectiveness model to account for leverage and funging when it seemed important to our bottom line and fairly clear that some adjustment was warranted:

• We applied discounts to costs incurred by certain entities. For example, we applied a 50% discount to the value of teacher time spent distributing deworming tablets, and excluded the costs to pharmaceutical companies donating these tablets.²See our May 2017 cost-effectiveness analysis. Our rationale was that without our top charities, these resources would likely otherwise been used less productively.
• We applied ‘alternative funders adjustments’ to account for the possibility that we were crowding out other funders. For example, some of the distributions that AMF considered funding, but didn’t ultimately fund, were picked up by other funders (more).

This helped us explicitly think through considerations relevant to our top charities. But by the end of 2016, our model had a handful of ad hoc adjustments that were difficult to identify, understand, and vet. For example, the discounts we applied to costs incurred by certain entities were ‘baked in’ to our estimates of cost per treatment, rather than explicit on the main spreadsheet of our cost-effectiveness analysis.

Changes to how we incorporate leverage and funging into our cost-effectiveness analysis

We revisited the way we thought about leverage and funging in preparation for our 2017 top charities decision. We wanted to make sure our adjustments were transparent and consistent across all charities.

We now explicitly make quantitative judgments about (i) the probability that our charities are causing governments and multilateral aid agencies to spend more or less on a program than they otherwise would have and (ii) the value of what those funds would otherwise have been spent on.³Our current best guess of a reasonable benchmark for the counterfactual value of government funds is ~75% as cost-effective as GiveDirectly (discussed later in the post). We view this is a very rough guess.

Here’s an exercise that some GiveWell staff have found helpful for getting a more intuitive feel for different ways of treating leverage.

Suppose a charity pays $5,000 to purchase magic pills. This would cause (with 100% certainty) the government to spend another $5,000 distributing those pills. The pill distribution saves 1,000 lives in total. If the government didn’t fund the pill distribution, it would have spent $5,000 on something that would have saved 250 lives.

How should a philanthropist think about the cost-effectiveness of this charity?

1. One option is to include all costs to all actors on the cost side of the cost-effectiveness ratio. Total costs are $10,000 to save 1,000 lives and cost-effectiveness is $10 / life saved. This was GiveWell’s approach in 2011.
2. Another option is to discount government costs by 50%, because the government would otherwise have spent the funds on something 50% as effective. So total costs are $5,000 + (50% x $5,000) = $7,500. 1,000 lives are saved and cost-effectiveness is $7.50 / life saved. This was GiveWell’s approach from 2014 through 2016.
3. A third option is to include only the costs to the charity on the ‘cost’ side. The charity causes the magic pill distribution to happen, saving 1,000 lives. But it also causes the government to spend $5,000, which otherwise would have been used to save 250 lives. So the total costs are $5,000, and 1,000 – 250 = 750 lives are saved. Cost-effectiveness is $6.66 / life saved. This is GiveWell’s approach now.⁴In order to isolate the effect that leverage/funging has, we first calculate the impact of the program using the first method (including all costs equally), then apply a “leverage/funging” adjustment to transform the answer to the third method.

We believe the third way of treating leverage best reflects the true counterfactual impact of a charity’s activities. It also makes charities that are leveraging other funders look substantially more cost-effective than we previously thought.

Limitations of our approach

There are two important limitations to the way we account for leverage and funging.

First, these estimates rely on more guesswork than most of our cost-effectiveness analysis, reflecting a fundamental tradeoff we face in deciding which considerations to explicitly quantify. Quantification forces us to think through not just whether a particular consideration matters, but how much it matters relative to other factors, and to be explicit about that. On the other hand, incorporating very uncertain factors into our analysis can reduce its reliability, give a false impression of certainty, and make it difficult for others to engage with our work. In this case, we thought the benefits of explicit quantification outweighed the costs.

Two examples of assumptions going into our leverage and funging adjustments that we’re highly uncertain about:

1. Our best guess is that the average counterfactual use of domestic government spending that could be leveraged by our top charities is ~75% as cost-effective as GiveDirectly. We think using this figure is a useful heuristic, which roughly accords with our intuitions (and ensures we’re being consistent between charities), but we don’t feel confident that we have a good sense of what governments would counterfactually spend their funds on, or how valuable those activities might be.
2. We estimate there is a ~70% chance that, without Malaria Consortium funding, the marginal seasonal malaria chemoprevention (SMC) program would go unfunded, but only a ~40% chance that, without Against Malaria Foundation funding, the marginal bednet distribution would go unfunded. Estimating these probabilities is challenging, but taking our best guess forces us to evaluate how much weight to place on the qualitative consideration that there are more alternative funders for bednet distribution than SMC.

Second, we don’t explicitly model the long-term financial sustainability of a program. One worldview we find plausible for the role of effective philanthropy is in demonstrating the effectiveness of novel projects that, in the long run, are taken up by governments. This is not captured within our current model, which only looks at the effects of leverage and funging in the short term. Due to the difficulty of explicitly modelling this consideration, we take it into account qualitatively.⁵For example, we allocated more discretionary funding than we would have on the basis of cost-effectiveness alone to No Lean Season in 2017 due to our view that it was demonstrating the effectiveness of a novel program, which may have long-run funding implications.

 

How much of a difference do leverage and funging make?

In the table below, we present how our new method of accounting for leverage and funging compares to (i) counting all costs equally and (ii) our previous method of accounting for leverage and funging.

Adjustments range between a modest penalty for AMF (because we expect AMF crowds out some funds from other sources) to a large boost to SCI (because the cost to pharmaceutical companies of manufacturing donated drugs comprises a substantial proportion of cost per treatment in SCI distributions, and we expect that without SCI, these resources would have been put to less valuable uses).

Note: 1.2x implies the adjustment makes the charity look 20% more cost-effective; 0.8x implies the adjustment makes the charity look 20% less cost-effective. All charities listed are GiveWell top charities as of November 2017.

Charity	Versus counting all costs equally6Calculations here. “N/A” refers to charities for which we had not completed a cost-effectiveness analysis before October 2017.	Versus our 2014-16 methodology	Commentary
Against Malaria Foundation	0.8x	1.1x	Government costs represent a small proportion of funding for AMF programs. Our analysis of distributions that AMF considered, but did not fund, suggests that some of these distributions are covered by alternative funders, who would otherwise have supported less valuable programs.
Schistosomiasis Control Initiative	2x	1.2x	We estimate ~60% of the costs of SCI-supported deworming programs are incurred by either governments or pharmaceutical companies. We expect that without SCI, most of these resources would have been used on less valuable programs.
Evidence Action’s Deworm the World Initiative	1.4x	1.1x	We estimate ~40% of the costs of Deworm the World-supported deworming programs are incurred by either governments or pharmaceutical companies. We expect that without Deworm the World, most of these resources would have been used on less valuable programs.
Sightsavers’ deworming program	1.6x	1.3x	We estimate ~50% of the costs of deworming in Sightsavers supported programs are from governments or donated drugs from pharmaceutical companies. We expect that without Sightsavers, most of these resources would have been used on less valuable programs.
END Fund’s deworming program	1.3x	N/A	We estimate ~40% of the costs of END Fund-supported deworming programs are incurred by either governments or pharmaceutical companies. We expect that without the END Fund, most of these resources would have been used on less valuable programs.
Helen Keller International (HKI)’s vitamin A supplementation (VAS) program	1.1x	N/A	We estimate ~25% of the costs of HKI-supported VAS programs are covered by governments. We expect that without HKI, most of these resources would have been used on less valuable programs.
GiveDirectly	1x	1x	Due to the scalability of GiveDirectly’s program, we believe it is unlikely that GiveDirectly crowds out funding from other sources. GiveDirectly does not leverage funds from other sources.
Malaria Consortium’s seasonal malaria chemoprevention program	.98x	1.04x	Government costs represent a small proportion of funding for Malaria Consortium programs. We believe it is possible but unlikely that Malaria Consortium crowds out additional government funding.
Evidence Action’s No Lean Season	1x	N/A	No Lean Season is a novel program, and we think it’s unlikely to be crowding out funding from other sources. No Lean Season does not leverage substantial funding from other sources.

Notes   [ + ]

1.	↑	For example, see row 3 of our 2013 cost-effectiveness analysis for Against Malaria Foundation.
2.	↑	See our May 2017 cost-effectiveness analysis.
3.	↑	Our current best guess of a reasonable benchmark for the counterfactual value of government funds is ~75% as cost-effective as GiveDirectly (discussed later in the post). We view this is a very rough guess.
4.	↑	In order to isolate the effect that leverage/funging has, we first calculate the impact of the program using the first method (including all costs equally), then apply a “leverage/funging” adjustment to transform the answer to the third method.
5.	↑	For example, we allocated more discretionary funding than we would have on the basis of cost-effectiveness alone to No Lean Season in 2017 due to our view that it was demonstrating the effectiveness of a novel program, which may have long-run funding implications.
6.	↑	Calculations here. “N/A” refers to charities for which we had not completed a cost-effectiveness analysis before October 2017.

The post Revisiting leverage appeared first on The GiveWell Blog.

from The GiveWell Blog http://ift.tt/2EFOgiG