Explained: What Are The 'Natural Experiments' That Won This Year's Nobel

Many natural experiments have helped researchers and policymakers understand impacts of policies and programmes in India. For instance, canal construction in India increased the pace of urbanisation, found a 2021 study. And the chances of a woman winning on a seat that was reserved for women in the previous election were five times higher than those on a seat where the quota had not applied in the previous election, found a 2008 study in Mumbai. Another natural experiment allowed researchers to study whether schooling improved in marginalised groups after they were added to the list of Scheduled Castes. The 2011 study found that such impacts were based on whether people resided in urban areas and had access to schools.

Such natural experiments have added to our knowledge of how programmes and interventions impact people's well being, information that can be used to make evidence-based policies in the future.

In this #TIL piece, we explain what natural experiments are and why they are important to study the impact of programmes and policies.

What is causality and why do we need to study it?

If two variables (for instance, the number of two-wheelers, and crime, in a neighbourhood) are observed over time, it is possible that the two show some sort of relationship: both are increasing over time, both are decreasing over time, or one is increasing while the other decreases. Such relationships are known as correlations. But this correlation is not enough to say that one variable causes the other. So, if both the number of two-wheelers and crime are increasing over time, we cannot say that more two-wheelers are leading to more crime (or vice-versa)!

To establish that causality, a researcher would have to prove that there is no third factor impacting both the variables. For instance, it is possible that growing incomes are causing both--families buying more two-wheelers, and more thieves targeting the neighbourhood.

Measuring causality

Since economics involves studying complex interactions between people in diverse social and political settings, economists reduce these interactions to models to understand causality. A model is a simplified version of reality and rests on many assumptions, and a model is only as useful as its assumptions are valid. Essentially, teasing out causality involves ruling out other explanations for a particular phenomenon using experimental design.

The simplest technique to establish causality is called least squares regression, a statistical method that "controls" for all possible other factors that can cause the change in the variables (one control in the above example would be family income). Similarly, to determine the effect of gender on wages, economists control for schooling, ethnicity and work experience, to figure whether gender roles alone impact wages (they do.)

While regression is a powerful tool, progress in computation and the availability of vast, high-frequency data have also allowed econometrics--the use of statistical methods to understand economic data--to move much further.

Researchers now also use difference-in-differences, in which an outcome is measured before and after a treatment, such as difference in rates of anaemia before and after an iron supplementation programme. Another approach uses instrumental variables, when the actual "cause" variable cannot be measured, by using a proxy. For instance, let's say the hypothesis is that children born in the October-December quarter are exposed to more pollution than those born in January-March, based on pollution data. So, researchers would, instead of using data on actual exposure of children to pollution, compare children born in January-March with those born in October-December and look at the differences in, say, lung capacity.

The gold standard for establishing causality is a randomised controlled trial, or RCT. An RCT in economics, just like one for a drug or vaccine, involves assigning people to two groups randomly and administering a treatment, such as admission to a remedial tutoring programme, to one of them. To know the efficacy of the programme, the outcome of interest (maths scores, in this example) is measured for both groups that are identical in every way other than the remedial programme.

An early example of an RCT is one conducted by Donald Davidson in 1955 to measure utility--the satisfaction derived from something. In 2019, Michael Kremer, Esther Duflo and Abhijit Bannerjee, won the Nobel prize in economics, for their work with RCTs.

Natural experiments are also one way to test models with data.

What makes an experiment natural?

An experiment that uses a 'natural' happening, and not a programme designed especially for the purpose of that experiment, is called a natural experiment. For instance, David Card, professor of economics at the University of California in Berkeley, USA, and one of the Nobel prize winners, used the Mariel boatlift, in which thousands of Cuban refugees from the Mariel harbour in Cuba landed in Miami in the USA, to study how immigration impacts jobs and wages for the local population. He found that though the refugee influx increased the population of Miami by 7%, and added significantly to its labour supply, it did not create a shortage of jobs, even of the semi-skilled type.

The only disruption in the Miami job market was the entry of immigrants, who did not come there looking for jobs. This allowed Card to impose a crucial condition called 'ceteris paribus' (all other things being equal) on his model. It allowed Card to study the effect of immigration on jobs in a town without other factors that could have caused the immigration, such as a sudden burst of job openings, higher standards of living in Miami or a desire to live in the United States.

It would be unethical to create forced migration to study such phenomena, and a natural experiment allows economists to get around these ethical concerns.

Card also studied the effect of legislation such as a minimum wage law, which applied to people on one side of a state border. Comparing the number of jobs available in New Jersey and Pennsylvania before and after the minimum wage law was passed in New Jersey, Card found that having a minimum wage did not result in fewer jobs. In such an experiment, people are part of either the treatment group (where the minimum wage law applied) and of the control group (where no such law existed), based on a border, rather than because of the researchers' intervention. Such natural experiments help researchers eliminate other reasons for the effect they see, and pinpoint the most probable cause.

Another experiment, conducted by the other Nobel prize winner, Joshua Angrist, an economics professor at the Massachusetts Institute of Technology in Cambridge, USA, showed that soldiers returning from war made less money as civilians than their non-combatant peers. Since the USA relied on a random allotment of men to the "draft", as the compulsory service in the armed forces is known, the difference in earnings of the workers, after considering age, education and professional experience, could be attributed entirely to serving in the military.

In another example of a natural experiment, Guido Imbens, an economics professor at Stanford University, USA, the third Nobel prize winner, studied lottery winners in the USA and found that unearned income reduced labour earnings and increased leisure marginally, and most for those between 55 and 65 years.

Before the use of natural experiments to study cause and effect, economists used correlations (such as that between unemployment rate and inflation) to explain social phenomena. But given the complexity of understanding cause and effect, these studies had varying degrees of success. For instance, William Phillip's famous 1958 study showed an inverse relationship between unemployment and inflation. However, in the 1970s, both unemployment and inflation increased at the same time, calling the theory into question.

No method is foolproof

RCTs cannot be used for all research as they are expensive and have ethical and methodological concerns. For instance, in order to study the effect of low calorie intake during pregnancy on the health of children, it would be wrong to starve one group of mothers. However, a natural experiment, such as a short-duration famine in a developed country, provides the conditions necessary to study this, as did this study by L.H. Lumey and Frans van Poppel of the 1944-45 famine in the Netherlands.

Both natural experiments and RCTs can be used by policy makers if the design is sound and the data are analysed correctly, said Martin Ravallion, a professor of economics at Georgetown University, in Washington D.C., USA, and previously the director of research at the World Bank. "The difference is that a natural experiment uses the actual, 'real world' policy, while for an RCT you are creating a new policy, which (unlike almost any real world policy) uses randomised assignment."

We contacted the Ministry of Social Justice and Empowerment for their comments on the use of these different evaluation techniques by government officials, but did not receive a response.

"There can be something unnatural about experiments designed to randomise assignments to 'treatment' and 'control' groups. That is a rather artificial setup, not common in the real world, Ravallion explained. "The beauty of a natural experiment is that it is real--not something contrived for the purpose of research," he added. "Natural experiments can provide valuable real-world lessons, including for policy making. But learning from natural experiments brings challenges too, most importantly in assuring that the lessons drawn can be believed."

Unlike physics or chemistry, economics is not an exact science, which means that theories are not as easily represented mathematically or always reproducible in different settings. For instance, a match struck will always light up if it is of good quality and dry. But a social programme can have different results in different geographies and cultures, such as the use and impact of free mosquito bed nets versus those that people have to buy.

Another shortcoming of economics is its inability to forecast a result based on experiments.

Because of these reasons, social sciences, like economics, have been accused of "physics envy"--attempting to measure and predict the result of a social or behavioural experiment in the same way as a physics experiment.

There are many reasons for why economics is not an exact science: Bias always creeps in, and it is impossible to rule out confounding factors, such as the role of age in an experiment to determine the effect of exercise on weight gain, even in the most rigorous natural experiments. Even Card's results on the effect of a minimum wage on the unemployment rate do not always hold up, as in this 2012 study comparing New York, Pennsylvania, Ohio and New Hampshire -- all cities in the USA.

However, this does not mean that empirical experiments can be discredited entirely. "Card and Krueger's findings have established that an increase in minimum wage does not always lead to higher unemployment," explained Anand Srivastava, a professor of econometrics at the Azim Premji University in Bengaluru.

We welcome feedback. Please write to respond@indiaspend.org. We reserve the right to edit responses for language and grammar.