A short snapshot of what we've built recently, and directions we could take with new funding. Made to give our team a sense of what's already underway — and an easy way to say what they'd like to work on next.
Over the past year we've built out a Pivotal Questions program that starts from decision-makers' real uncertainties, an animal welfare evaluation effort now tied to a live funding application, early LLM-evaluation and AI-assisted prioritization work, and a set of tools and interfaces for evaluators — on top of a standing pool of ~230 vetted evaluators. Several of the items below are early-stage or proposed rather than finished. Why several of these look rough.
Our most developed direction, and more than a set of workshops. Instead of starting from existing papers, we start from the empirical uncertainties that could actually change a funder's, policymaker's, or researcher's priorities — then connect those questions to evidence through expert evaluations, structured workshops, forecasts, and public databases. Workshops so far have covered wellbeing measurement (comparing WELLBYs with DALYs and QALYs), cultivated-meat cost trajectories, and plant-based substitution. We're also prototyping a lightweight modeling approach and a synthesis of experts' beliefs.
A project testing how far large language models can assist and benchmark research evaluation — where AI assessments line up with expert human judgment, and where they don't. This is directly relevant to scaling evaluation affordably, and to funders focused on AI, epistemics, and the reliability of automated research assessment.
See the project →Resources: evaluator pool and statistics — around 230 vetted evaluators from 50+ countries.
Some directions we could grow into with funding. These are prompts, not a fixed menu — if you'd do something adjacent, or something we haven't listed, say so.
Structured expert evaluation of high-impact legal research — for example in AI governance and biosecurity — filling a gap left by student-edited law reviews. Legal scholarship shapes courts, regulation, and policy, so rigorous public assessment could matter a great deal.
Read the proposal →Run a focused stream of evaluations in your area — development economics, global health, animal welfare, AI governance, and more — as an evaluation manager or lead.
Identify decision-relevant questions and commission evaluations tied to specific decision-makers who would act on the answers.
Build interfaces, guides, and rubrics that help evaluators write better, faster, and more consistent referee reports.
Run in-person or online sessions on how to judge research and write referee reports, including evaluation games — in India and beyond.
Improve the prioritization tool and prototype AI-assisted evaluation and summary workflows, keeping humans in the loop for judgment and ratings.
Use our accumulating ratings and evaluations to study evaluation quality, calibration, bias, and how well ratings predict later outcomes.
Commission evaluators to rate a random sample of impact-relevant published research, building a benchmark and strengthening credibility.
Bring in new authors, institutions, or co-funders, and build relationships with decision-makers who would use our evaluations.
Possible collaborations that have come up in recent discussions. To be clear, these are proposed or potential — early conversations, not confirmed partnerships — and the organizations named haven't committed to anything. They're the kind of joint, funder-legible work we'd like to build out. Tell us if you'd want to help lead any of these, or know a partner we should talk to.
Commission expert evaluation of research on how advanced AI affects biological capabilities and biosecurity risk — a fast-moving area where the evidence base badly needs scrutiny. A possible collaboration with RRID and other partner organizations; early conversations only, nothing confirmed.
Evaluate research on welfare technologies and interventions — for example, whether electrical stunning reduces pain in shrimp — where better evidence could shift funding and industry practice. A potential fit with Rethink Priorities and our animal welfare stream, not yet a set commitment.
Build "evaluation games" — structured exercises where people judge research and compare their assessments — potentially with the Institute for Replication (i4rep), tying our evaluation work to reproducibility and replication. A proposed collaboration we'd like to pursue.
Grow evaluation capacity and training outside the main US/UK hubs — workshops, evaluation games, and evaluator development in South Asia and the broader Global South. A direction we'd hope to co-lead from India with Anirudh Tagat.