Research design, case selection, and bullshit

Bullshit and you

Most if not all of the methods courses you take as a university student will have some component on “case selection” or “research design”, where you talk about technical and abstract concepts like “bias” and “inference”. While this stuff may sound dry, abstract, academic, and “not relevant in the Real World”, it is at its core really about one major thing: How to avoid bullshitting yourself and others.

Bullshit is most likely a term you are familiar with: Claims or statements that sound kind of plausible but are inherently erroneous and misleading (Bergstrom and West 2021). Bullshit can be intentional, which happens when someone tries to mislead someone, but it can also happen by accident. In either case, bullshit can do serious harm: It can lead people (or companies and institutions) to make decisions that cost them real money, and it can lead them to avoid decisions that would be beneficial. This can range from buying a used car with lots of hidden defects to basing a company’s marketing strategy on biased and misleading customer data.

Bullshit is directly related to evidence. Usually, when people bullshit, they present some evidence – but the evidence they present is flawed: It is incomplete in a distorting way, leaving out a relevant part of the whole picture and thereby misrepresents the true state of the world. By being misleading and biased, such evidence leads others to make irrational and misinformed decisions (“inferences”). Again, this can be intentional, but it can also happen by accident.

Students can produce bullshit, too. This happens for example when students select cases for the comparative case study they are doing in their thesis project in way that it distorts their result (which happens quite a lot). These students most likely did not want to deceive anyone, they just either did not pay enough attention in their methods courses, they might think that case selection and bias do not apply to qualitative research, or they might simply see a careful case selection and research design as an inconvience they want to skip over. In any case, they are still involuntarily and unknowingly producing bullshit.

As a soon-to-be academically trained professional, a significant part of your future work will be to help others to identify and avoid bullshit (e.g., job applications from people who are unqualified but try to hide it, investment decisions that are financially harmful to your company but beneficial to the seller), but also to produce non-bullshit (valid reports, evaluations, white papers, etc.) that others can use to make good decisions.

And this is why you need to learn about research design and case selection strategies as a student in any empirical scientific discipline – a good research design is simply a strategy for how to select your evidence (quantitative or qualitative) in a way that your project does not produce misleading bullshit. Learning about good research design is also helpful because it trains you to be a bullshit detector.

Bullshit and case selection

As mentioned, comparative case studies can a major source of misleading evidence if they are not based on a proper case selection strategy, i.e., research design (King, Keohane, and Verba 1994, Geddes1990). This is often the case with student theses, but one occasionally also sees this in “adult” research.

We also know two main reasons for why a given case study can be faulty and produces misleading evidence:

1. Selection on the dependent variable
2. An insufficient number of cases

Let’s go over each of them to illustrate how they produce misleading results.

Selection on the dependent variable

Let’s say you are a student who is interested in the causes of war – why do some countries go to war with each other? – and you want to do a qualitative case study to figure out what is going on. Your intuition may then lead you to select two or three relevant cases (wars) where countries did fight each other and look in detail at what led up to each war and why countries chose to engage.

As another example, let’s say you want to find out why some countries become economically wealthy. Intuitively, you decide to select two or three countries that are wealthy and study them really, really closely using super-sophisticated qualitative methods and highly detailed evidence to see what is behind their wealth.

In both cases, you select on the dependent variable, meaning you only select positive cases (did go to war) or “high” cases (high wealth) – but you ignore the negative or “low” cases, the countries that did not go to war or did not become wealthy, even though they plausibly could have done either of those things. Both can really lead you to completely misleading results (Geddes 1990).

To see why selection on the dependent variable distorts your results, have a look at the interactive plot below:

• The plot shows the relationship between two generic variables, x (the explanatory variable) and y (the dependent variable)
• As you can clearly see, the relationship is positive: More x means more y

Now play around with the three sliders to see how different case selection strategies (limiting the number of cases, i.e., going from large-n quantitative toward small-n qualitative research, limiting the range of the x variable, and limiting the range of the y variable) affects the relationsip between the two variables.

tplot_dat = transpose(plot_dat)

viewof n_obs = Inputs.range(
  [2, 200], 
  {value: 200, step: 1, label: "Number of observations:"}
)
viewof range_x = Inputs.range(
  [1,100],
  {value: 100, step: 1, label: "Max. range of explanatory variable (X)"}
)
viewof range_y = Inputs.range(
  [14,180],
  {value: 180, step: 1, label: "Max. range of dependent variable (Y)"}
)

sliced = tplot_dat.slice(0,n_obs)
filtered = sliced.filter(function(sliced) {
  return range_x > sliced.xvar &&
         range_y > sliced.yvar;
})

Plot.plot({
  marks: [
    Plot.dot(filtered, {x: "xvar", y: "yvar", fill: "cornflowerblue"}),
    Plot.dot(tplot_dat, {x: "xvar", y: "yvar", fill: "currentColor", fillOpacity: 0.2}),
    Plot.linearRegressionY(filtered, {x: "xvar", y: "yvar", stroke: "cornflowerblue"}),
    Plot.linearRegressionY(tplot_dat, {x: "xvar", y: "yvar", stroke: "currentColor"})
],
    x: {label: "Explanatory variable (X)"},
    y: {label: "Dependent variable (Y)"},
    title: "How the evidence you select affects the answers you get",
    caption: "Shaded areas indicate 95% confidence intervals."
})

You should notice that limiting the number of observations is not inherently problematic. Sure, the line becomes wobbly and the confidence interval becomes larger (your conclusions become less certain), but your results still reflect the true relationship reasonably well unless you go for really low numbers of observations. The same applies if you limit the range of the explanatory x variable – the positive relationship stays more or less the same, unless your selection becomes very restrictive.

But things are very different when you limit the range of the y variable: The relationship becomes flatter and flatter – which completely misrepresents the true relationship in the data. The result is bullshit.

The main lesson:

If you want to find out the causes of war, you need to study cases where war happened – and cases where war could have happened, but did not. Similarly, if you want to find the causes of national wealth, you need to study countries that did get wealthy and those that could have gotten wealthy but did not. You need variation in your dependent variable.

Insufficient number of cases

If the example above gave you the idea that limiting the number of cases – going from quant to qual – is not by itself a problem as long as you don’t select on the dependent variable, then that is correct (see also King, Keohane, and Verba 1994): Both qualitative and quantitative studies can produce valid results as long as they are based on a valid research design – and they can both produce misleading bullshit results if the research design is flawed.

Still, there can be situations where your number of cases can be too small.

To give you a really simple example, let’s say you have a friend called Tim who is really into tennis. As is always the case, he sometimes wins matches and sometimes he loses them. You are interested in finding out why – what causes him to win and lose – and you you think that, theoretically speaking, it could be either the weather or the food he had for lunch that makes the difference.

You observe him playing tennis on two different days and note down your results in a table:

Table 1: First round of observations

	Day 1	Day 2
Weather	Sunny	Cloudy
Lunch	Meat	Fish
Result	Lost	Won

Have a look at the findings: Can you clearly tell why Tim won on day 2 but lost on day 1?

Now imagine that you add a third day of observations and your new results look like this:

Table 2: Adding a third day of observations

	Day 1	Day 2	Day 3
Weather	Sunny	Cloudy	Sunny
Lunch	Meat	Fish	Fish
Result	Lost	Won	Lost

Now you should be able to see a pattern: On the two sunny days, Tim lost; on the one cloudy day, however, he won. On the other hand, he had fish on days 2 and 3, and both won and lost on both of these days. So you see a relationship between his tennis success and the weather but no relationship with his food.

The main point is: The first research design in this example, which relied on two cases (days of observations) was indeterminate (King, Keohane, and Verba 1994): It had two few observations or data points relative to the number of variables: two potential reasons (or explanatory variables) and two observations. No ammount of in-depth “looking at the data” is going to tell the answer here because the “dataset” is simply not large enough to reveal a pattern. Once you add a third observation – you have two variables but three cases – the pattern emerges and you can give at least a preliminary answer to the question.

In other words, the first research design produces bullshit: It leads you and others to believe that there is no pattern, when in fact there really is one. The second one is valid because it at least makes it possible for a pattern to emerge and be visible.

The main lesson:

You always need to have one more observations than you have potential causes or explanatory variables. Anything else is likely misleading and will end with an unclear conclusion.

Research and case selection designs that avoid bullshit

By now you should hopefully understand the meaning (and importance) of the following statement: the evidence you select affects the answers you get (Geddes 1990). Faulty selection techniques produce faulty, misleading evidence – bullshit. This applies to both qualitative and quantitative research.

If you want to learn how you can design your research in a way that it does not produce misleading evidence, you are in luck: There are many different textbooks and articles that show you how to do this, for example:

• Chapter 9 in Ringdal (2018)
• Chapters 2 and 3 in Landman (2003)
• The book by King et al. (1994)
• The article by Geddes (1990)
• The article by Seawright & Gerring (2008)
• The book by Przeworski & Teune (1970)
• The book by Gerring (2007)

References

Bergstrom, Carl T., and Jevin D. West. 2021. Calling Bullshit: The Art of Skepticism in a Data-Driven World. New York: Penguin Random House.

Geddes, Barbara. 1990. “How the Cases You Choose Affect the Answers You Get: Selection Bias in Comparative Politics.” Political Analysis 2 (1): 131–50.

Gerring, John. 2007. Case Study Research. Cambridge: Cambridge University Press.

King, Gary, Robert O. Keohane, and Sidney Verba. 1994. Designing Social Inquiry. Scientific Inference in Qualitative Research. Princeton: Princeton University Press.

Landman, Todd. 2003. Issues and Methods in Comparative Politics: An Introduction. 2nd ed. London & New York: Routledge.

Przeworski, Adam, and Henry Teune. 1970. The Logic of Comparative Social Inquiry. New York: Wiley.

Ringdal, Kristen. 2018. Enhet Og Mangfold: Samfunnsvitenskapelig Forskning Og Kvantitativ Metode. 4. ed. Bergen: Fagbokforlaget.

Seawright, Jason, and John Gerring. 2008. “Case Selection Techniques in Case Study Research.” Political Research Quarterly 61 (2): 294–308.