Should Agricultural Policy Experiments Use 'Real' Farmers?

A new study examines the validity trade-offs when using farmer samples in economic experiments designed to inform agricultural policy.

Economic experiments are increasingly used to test the design of agri-environmental policies before they are rolled out at scale. But most laboratory experiments use student volunteers rather than the farmers who would actually participate in such schemes. Does this matter? A new project-funded study published in Q Open directly addresses this question, examining the trade-offs between internal validity (experimental control) and external validity (real-world relevance) when choosing who participates in policy experiments.

What are the trade-offs?

Experiments conducted with student participants in controlled laboratory settings offer high internal validity: researchers can tightly manage conditions, minimise confounding variables, and draw clean causal inferences. However, the behaviours observed in such settings may not reflect how real farmers would respond to the same policy instruments. Farmers bring domain-specific knowledge, professional experience, established risk preferences, and social norms that student participants do not share. These differences can substantially alter how people respond to incentive structures such as results-based payments, collective contracts, or monitoring requirements.

Conversely, experiments conducted with farmer samples improve external validity (the findings are more likely to hold in practice) but come at a cost. Farmer recruitment is more difficult and expensive, sample sizes tend to be smaller, and the diversity of farming contexts introduces additional variation that can make it harder to isolate specific treatment effects. Farmers may also interpret experimental instructions through the lens of their own experience, introducing systematic differences in how they engage with the task.

What does this mean for policy design?

The study argues that neither approach alone is sufficient. Student-based experiments are valuable for isolating causal mechanisms and testing theoretical predictions, while farmer-based experiments are essential for understanding whether those mechanisms operate as expected in real agricultural contexts. The authors propose that these two approaches should be treated as complementary: student experiments to establish proof of concept, followed by farmer experiments to validate findings and refine policy design before implementation. This staged approach can improve the evidence base for agri-environmental policy while making efficient use of limited research resources.

The findings are directly relevant to FRAMEwork’s research programme, which has used both experimental and survey-based methods to test incentive designs for farmer clusters and results-based payment schemes across multiple European countries.

Read the paper

The full open-access paper is available from Q Open. For more on FRAMEwork’s experimental research on agri-environmental policy design, see our related publications on results-based payments experimental evidence from Germany and results-based payments for groups. Browse all FRAMEwork publications at framework-biodiversity.eu/publications.