Research plans 2018 [draft]

This is the second of four posts that form our annual review and plan for the following year. The first post reviewed our progress in 2017. The following two posts will cover GiveWell’s progress and plans as an organization and metrics on our influence on donations in 2017.

Summary

Our primary research goals for 2018 are to:

• Explore areas that may be more cost-effective than our current recommendations by investigating (i) interventions aimed at influencing policy in developing countries (ii) opportunities to influence major aid agencies (iii) new potential GiveWell Incubation Grantees outside these categories.
• Find new top charities similar to our current recommendations by (i) completing intervention reports for at least two interventions we think are most likely to result in GiveWell top charities by the end of 2019 (ii) renewing GiveWell Incubation Grants to current grantees who may become top charities in the future.
• Improve our internal processes to support the above two goals. Specifically, (i) continue to delegate significant parts of our top charity update process to non-management staff (ii) improve our year-end process for giving recommendations (iii) increase the pace with which we publish completed reports.

Secondary goals are to:

• Improve the quality of our decisions and transparency about our decision-making process.
• Hire more flexible research capacity to increase our output.

We discuss each of these goals in more depth below.

Goal 1. Explore areas that may be more cost-effective than our current recommendations

In the last two years, we’ve added a number of new top charities, and now believe our current top charities have more room for more funding than we are able to fill. This increases the relative value of identifying giving opportunities which are substantially more cost-effective than our current top charities (because identifying similarly cost-effective opportunities will crowd out marginal funding for our current top charities), even if we believe we have a lower chance of success of identifying these opportunities. We’re therefore prioritizing investigating areas where we believe we have the highest chance of finding opportunities which are substantially more cost-effective than our current top charities.

Goal 1.1: Influencing policy in developing countries

Why we’re prioritizing it: Our current top charities all implement direct delivery interventions (although some leverage substantial domestic government funding). We think there’s a reasonable intuitive case that philanthropists may, in some cases, be able to have a greater impact by influencing developing country government policy because (i) governments have access to regulatory interventions which are unavailable to philanthropists (ii) there may be opportunities to help improve the allocation of large pools of funds. We’ve already started work investigating advocacy for tobacco control, lead paint regulation, and road traffic safety interventions.

Metric for success: Publish at least five interim intervention reports on interventions to influence policy in developing countries, and prioritize one to three for deeper assessment.

Goal 1.2: Improve our understanding of aid agencies

Why we’re prioritizing it: Aid agencies (such as DfID or USAID) collectively spend [$x] on health, either bilaterally or through multilateral international organizations. Several people who we believe are well-informed, and aligned with our values, have suggested there may be opportunities for GiveWell (or potential GiveWell grantees) to help improve the allocation of spending by aid agencies. We want to improve our understanding of what aid agencies spend their funds on, whether there are opportunities to improve this allocation, and whether GiveWell (or potential grantees) would be in a good position to assist.

Metric for success: As this project is at an early stage, we don’t yet have specific metrics to assess success.

Goal 1.3: Other incubation grants

In the past, we’ve made GiveWell Incubation Grants to promising opportunities which didn’t fit within our priorities at the time. We currently have a long list of opportunities to investigate for potential grantees, and also want to remain open to investigating opportunities we are not yet aware of.

Metric for success:Prioritize at least two of these opportunities, or opportunities we are not yet aware of, and complete a thorough investigation.

Number of areas we haven’t looked at. Want to know better
Fairly narrow set of interventions — focused on these because easier to evaluate
We want to expand our research project
Funding constraints push towards this
Two of the areas where we guess we may be able to do better are in policy and influencing aid agencies
We also want to stay responsive to other opportunities as they arise as that’s led to grants in the past.

How will we know if we succeeded?

Intervention assessments are key to our research process. We generally only consider recommending funding for programs that are implementing one of our priority programs (an exception is if an organization has done rigorous evaluation of its own program, though in practice we have found this to be very rare). In recent years, we have completed few intervention reports, which has limited our ability to consider new potential top charities. We plan to increase the rate at which we form views on interventions this year by:

• Hiring a Senior Fellow (or possibly more than one). We expect a Senior Fellow to have a Ph.D. in economics, public health, or statistics or equivalent experience and to focus on in-depth evidence reviews and cost-effectiveness assessments of interventions that appear promising after a shallower investigation. In addition, Open Philanthropy Project Senior Advisor David Roodman may spend some more time on intervention related work.
• Doing low-intensity research on a large number of promising interventions. We generally start with a two to four hour “quick intervention assessment,” and then prioritize interventions for a 20-30 hour “interim intervention report” (example). We don’t yet have a good sense of how many of these of these we will complete this year, because we’re unsure both about how much capacity we will have for this work and about how many promising interventions there will be at each step in the process.
• Continuing to improve our systems for ensuring that we become aware of promising interventions and new relevant research as it becomes available. We expect to learn about additional interventions by tracking new research, particularly randomized controlled trials, in global health and development and by talking to select organizations about programs they run that they think we should look into.

Charity applications

In the past few years, we have been surprised by how little interest there has been from charities in applying for a GiveWell recommendation. Our impression is that for global health and development charities there are relatively few funders of our size: in 2015, we tracked $110 million given due to our research; we are in the process of compiling the data for 2016, but expect it to be in the range of $80-90 million to recommended charities. We would like to better understand whether we have failed to get the word out about the potential value we offer or communicate well about our process and charities’ likelihood of success, or, alternatively, whether charities are making well-informed decisions about their fit with our criteria. (More on why we think more charities should consider applying for a GiveWell recommendation in this post.)

This year, we have designated GiveWell Research Analyst Chelsea Tabart as charity liaison. Her role is to increase and improve our pipeline of top charity contenders by answering charities’ questions about our process and which program(s) they should apply with, encouraging promising organizations to apply, and, through these conversations, understanding what the barriers are to more charities applying.

We aim by the end of the year to have a stronger pipeline of charities applying, have confidence that we are not missing strong contenders, or understand how we should adjust our process in the future.

Incubation Grants

We made significant progress on Incubation Grants in 2016 and plan in 2017 to largely continue with ongoing engagements, while being open to new grantmaking opportunities that are brought to our attention.

Among early-to-mid stage grants, we plan to spend the most time on working with IDinsight and New Incentives (where our feedback is needed to move the projects forward), and a smaller amount of time on Results for Development and Charity Science: Health (where we are only following along with ongoing projects).

Another major priority will be following up on two later-stage grantees, No Lean Season and Zusha!, groups that are contenders for a top charity recommendation in 2017. For No Lean Season, a program run by Evidence Action, our main outstanding questions are whether the program will have room for more funding in 2018 and whether monitoring will be high quality as the program scales. We have similar questions about Zusha! and in addition are awaiting randomized controlled trial results that are expected later this year.

Cost-effectiveness model

We plan to continue making improvements to our cost-effectiveness model and the data it draws on (separate from adding new interventions to the model, which is part of the intervention report work discussed above). Projects we are currently prioritizing include:

• Making it more straightforward to see how personal values are incorporated into the model and what the implications of those values are.
• Revisiting the prevalence and intensity adjustment that we use to compare the average per-person impact of deworming in places that our top charities work to the locations where the studies that found long-term impact of deworming were conducted. More in this post.
• Improving the insecticide-treated nets model by revisiting how it incorporates effects on adult mortality and adjustments for regions with different malaria burdens and changes in malaria burden over time.

Current top charities

Our goal this year is to maintain the quality of top charity updates while decreasing the amount of staff time we spend and we ask top charities to spend on this work. Below, we detail our plans for following up with each charity.

Deworm the World Initiative, GiveDirectly, and Schistosomiasis Control Initiative (SCI)

We have now followed these groups for several years and do not have major outstanding questions about them. We plan to ask for updates on financial information, monitoring results, and room for more funding and have regular phone calls with them to learn about operational changes that might lead us to ask additional questions.

Against Malaria Foundation (AMF)

We have two major outstanding questions about AMF that we hope to make progress on this year:

1. Will AMF’s monitoring processes be high quality? We wrote about our concerns about AMF’s past monitoring last year and expect new information to be available this year.
2. Going forward, AMF aims to fund larger distributions and commit funding further ahead of when a distribution is scheduled to occur than it has, for the most part, done in the past. Will this increase the extent to which AMF funds displace funds from other sources, or will there continue to be evidence that AMF’s funds are largely adding to the total number of nets distributed? More on this question in our review of AMF.

To help us make progress on these questions, we and AMF have agreed to have monthly calls to discuss questions we have about the monitoring AMF is producing and what AMF is learning about distributions it is considering funding. We will likely also seek out calls with AMF’s partner organizations to discuss these questions.

In order to estimate AMF’s room for more funding, we will seek out information on the location and size of funding gaps for mass net distribution campaigns from AMF, the African Leaders Malaria Alliance, and possibly other funders of nets. As we have in the past, we will use this information in conjunction with conversations with AMF about non-funding bottlenecks to its ability to fill various gaps.

The END Fund – deworming program

Compared with the charities we have recommended for several years, we have more open questions about the END Fund. The main questions we plan to seek more information on this year are:

1. We have not yet seen monitoring on par with that from our other top charities from the END Fund. We expect results from coverage surveys from END Fund programs this year. Will these surveys be high quality and demonstrate that the END Fund is funding successful programs?
2. We have not yet tried to compare the cost-effectiveness of the END Fund to our other top charities in our cost-effectiveness model. We will be seeking additional information from the END Fund about cost per treatment and baseline infection rates.
3. Questions around room for more funding: the extent to which funding due to GiveWell’s recommendation increases the amount that the END Fund spends on deworming versus other programs, actual and projected revenue from other sources, and what deworming grantmaking opportunities the END Fund expects to have.

We visited the END Fund’s programs in Rwanda and Idjwi island, DRC in January 2017 and will publish notes and photos from our visit shortly.

Malaria Consortium – seasonal malaria chemoprevention program

As with the END Fund, we have more open questions about Malaria Consortium than we do for the charities we have recommended for several years. Our main priorities are:

1. Further research on the evidence of effectiveness, cost-effectiveness, and potential downsides of seasonal malaria chemoprevention (SMC) (due to time constraints we have not yet completed a full intervention report, though we felt sufficiently confident in the intervention to recommend Malaria Consortium).
2. Getting a better understanding of the methodology Malaria Consortium uses for estimating coverage rates.
3. Completing a more in-depth room for more funding analysis for the program for 2018 than we did for 2017.

Malaria Consortium expects to have several new studies of its SMC programs to share in April 2017 (details).

We may visit a Malaria Consortium seasonal malaria chemoprevention program in summer 2017.

Sightsavers – deworming program

As with the END Fund and Malaria Consortium, we have more open questions about Sightsavers than we do for the charities we have recommended for several years. We expect to make limited progress this year because the first deworming mass drug administration funded with GiveWell-influenced funds is not expected to take place until September at the earliest and monitoring results aren’t expected until early 2018. Because Sightsavers has done fairly little deworming in the past year, we don’t expect to be able to learn much from its ongoing programs. Our main priorities for the year are:

1. Getting more information from Sightsavers about baseline prevalence and intensity of worm infections in the areas it is working, to inform our cost-effectiveness analysis.
2. Using Sightsavers’ budget for the projects and planned treatment numbers to improve our estimate of the cost per treatment – another input into our cost-effectiveness analysis. Our current cost per treatment estimate is very rough.
3. Completing a room for more funding analysis for 2018.

Standout charities

Standout charities are groups that we have a large amount of information about and that meet some but not all of our criteria. Because we have not followed them closely over time, it is possible that they may now be a stronger fit (or that they no longer focus on the program we reviewed). We plan to have at least one phone call with each of these groups to discuss whether anything has changed that might lead us to reopen consideration of the organization as a potential top charity. Due to our focus on organizations that are most likely to become top charities, we don’t expect to make this work a priority beyond that.

The post Research plans 2018 [draft] appeared first on The GiveWell Blog.

from The GiveWell Blog http://ift.tt/2HgBS6m