Randomized trials are NOT the hottest thing in the fight against poverty

In his recent op-ed in the New York Times Nicholas Kristof praised randomized trials are the "hottest thing in the fight against poverty“. This statement by one of the popular columnist on aid issues in the US sounds so wrong I don't even know where to begin. Let me begin with the assumption that the fight against poverty is hampered by a lack of knowledge about what works and what doesn't. This knowledge, Kristof and many others argue, can be created by randomized controlled trials.

I agree, that randomized controlled trials are a valuable contribution to the learning tools in development. But they can only answer very specific questions that are primarily valid for the context in which they were executed. Many of the complex questions on democratisation, strengthening of governance, etc. are not easily amenable to randomized controlled trials. In addition, learning on what works and what doesn't is not new to development cooperation. In the past decades issues like sustainability, gender, global warming, governance etc. have all entered the aid discourse and policy thanks to an ongoing learning process among practitioners and experts on what works and what doesn't. Also in more technical areas like health, agriculture, water and sanitation there are numerous case studies and research projects that can guide future projects. 

In fact, I would argue, that in many cases there is abundance of knowledge but a lack of knowledge management. Think about the thousands of consultancies commissioned by the donor agencies, expert papers written by researchers and conferences held every year around the globe. How is this wealth of knowledge aggregated and made accessible to project managers? My guess is that most project managers are so overwhelmed with work that their intake of expert knowledge is limited to the absolute minimum. 

Of course, advocates for the “hottest thing in fighting poverty” may argue that randomized trials unlike other studies do not provide long assessments and nuanced judgement but clear-cut straightforward answers. What is the best way to keep children in school? Deworming!

But will deworming work similarly well in Uganda? Or in Sierra Leone? Or in the Philippines? Maybe, maybe not. Unless randomized trials are repeated in different contexts they don't have much validity. Would an alternative approach that strengthened the health system as a whole instead of giving out deworming drugs in schools be more sustainable in the long run? Maybe, maybe not. Would the results of this approach be the same if it was up-scaled or would the costs and the administrative burden increase significantly? Maybe, maybe not. The certainty you can get with randomized trials is valuable, but should not be overrated.

One last point on the rigour of the method: the answer you get always depends on the question you ask. No rigorous tool can control for biased questions that may or may not take into account the interests of the people concerned. So, basically I strongly disagree with Kristofs the naïve assessment of randomised trials. They are a very suitable tools for certain questions and like all other tools they have strengths and weaknesses. 

Much more critical, however, is Kristofs underlying assumption that lack of knowledge is a major problem in the fight against poverty. Where is the evidence for that? What about all the other usual suspects for global poverty? Conflicts, global terms of trade, lack of infrastructure, corruption, weak governance, tax evasion and weak tax systems, etc. etc.? We cannot seriously claim that knowledge created in randomized trials focusing on very specific questions can do away with these causes for poverty.

 But even if we are much more modest, even if we consider the impact of randomised controlled trials on policy in the respective sector, I do not share the enthusiams for randomized trials. The study on deworming by Kremer and Miguel from 2003 is again and again cited by people to make the case for randomized controlled trials. I wonder to what extent these valuable findings have changed policies of donor agencies and education ministries around the world? Yes, there are some programs targeting worms, but worms are still on the list of the neglected tropical diseases. How can that be, if we have the hottest thing in the fight against poverty targeted at worms? If the evidence is so convincing, the method so simple and so cheap, why are there no deworming programmes in many schools all over the world?

Obviously, political decision making and policy decisions within donor agencies are not primarily guided by scientific research, even if research findings are presented in a straightforward way. Decisions are to a large extent based on interests and on the influence different stakeholders bring to the negotiating table. This is illustrated by one of the bigger problems in the fight against poverty: aid effectiveness. There is not only research to support the need for more effective aid, but also an international agreement supported by donors, by partner countries by civil society and researchers. All agree that donor coordination, the ownership of national governments and other elements of donor governance have to change. Despite this consensus progress in these areas has been slow.

What does it mean for randomized trials? It means, that they are certainly a valuable contribution to the methodological tool box of researchers and evaluators. But even on a small scale decisions are only partially based on evidence. On the big scale, however, interests and power of the stakeholders matter most. So, the unfortunate truth is probably, that there is no "hot" thing in fighting poverty except  perseverance in pursuing structural changes and helping the poor to take an active role in the negotiations.