Publishing more frequent updates to our cost-effectiveness model

We’ve recently made a number of adjustments to improve our research process. Not all of them are easily visible outside of the organization.

This post is to highlight one of them: Publishing more frequent updates to our cost-effectiveness model throughout the year.

Summary

This post will explain:

• What changed in how we make updates to our cost-effectiveness model. (More)
• Why we made this change. (More)
• How to engage with updates to our model. (More)

What changed?

Last week, we published the ninth and tenth versions of our cost-effectiveness model in 2018. We made a number of updates to the newest versions of the model. They included accounting for reductions in malaria incidence for individuals who don’t receive seasonal malaria chemoprevention (SMC), the treatment one of our top charities distributes to prevent malaria, but who might benefit from living near other people receiving SMC (version 9) and the cost per deworming treatment delivered by another top charity, Sightsavers (version 10). These changes, and six others that were incorporated in the two latest versions, are described in our changelog.

Up until last year, we generally updated our cost-effectiveness model once or twice per year. However, as our model grew in complexity and we dedicated more research staff capacity to it, we decided that it would be beneficial to publish updates more regularly. We published our first in this series of more-frequent updates to our cost-effectiveness model in May 2017, as well as “release notes” (PDF) detailing the changes we made and the impact each had on our cost-effectiveness estimates.

We published five versions of our cost-effectiveness model in 2017. In 2018, we shifted from publishing PDF release notes to creating a “changelog“—a public page listing the changes we made to each version of the model, to be updated in tandem with the publication of each new version.

Internally, we moved toward having one staff member, Christian Smith, who is responsible for managing all changes to our cost-effectiveness model. He aims to publish a new version whenever there is a large, structurally complicated change to the model, or if there are several small and simple changes. Our internal process prioritizes being able to track how each change to the model moves the bottom line.

Changes we’ve published this year include updated inputs based on new research, such as the impact of insecticide resistance on the effectiveness of insecticide-treated nets; changes to inputs we include or exclude from the model altogether, such as removing short-term health benefits from deworming; and cosmetic changes to make the model easier to engage with, such as removing adjustments to account for the influence of GiveWell’s top charities on other actors from a particular tab.

Why we moved to this approach

Although it involves uncertainty, GiveWell’s cost-effectiveness model is a core piece of our research work and important input into our decisions about which charities to research and recommend. However, we believe it is challenging to engage with our model—to give a sense of the scale, our current model has 16 tabs, some of which use over 100 rows—and to keep up with changes we’ve made to the model over time.

Our hope is that publishing more frequent and transparent updates brings us closer in line to our goal of intense transparency and presenting a clear, vettable case for our recommendations to the public. It makes clearer the magnitude of any given change’s impact on our bottom line, and makes the evolution of the model over time easier to track. We also expect that it reduces the likelihood for errors, as fewer elements are being changed at any given time.

How to engage with updates to our model

We update our changelog, viewable here, when we publish a new version.

Going forward, we also plan to publish an announcement to our “Newly published GiveWell materials” email list when we do this. You can sign up to receive alerts from this email address here.

The post Publishing more frequent updates to our cost-effectiveness model appeared first on The GiveWell Blog.

from The GiveWell Blog https://ift.tt/2NhE6oK