It should come as no surprise that a blog called the Art of Boring would ask me to contribute on the scintillating topic of statistics: the science of collecting, analyzing, presenting, and organizing data. Cue the cheers and wolf whistles.
We’re bombarded with stats. We see them in the media, on our food labels, on our Fitbits, and certainly in investing. As a portfolio manager, I’m steeped in the stuff down to the nitty gritty. Yet the ubiquity of statistics in our lives doesn’t mean there is broad understanding of them. Not only is awareness of the science of statistics rather abysmal, the way these figures are reported and interpreted is frequently…not great. Too often, we see headlines that are ostensibly supported by evidence, only to discover that the studies they reference do not support the conclusions presented. Unfortunately, we cannot simply take the statistics we encounter (and any interpretations of them) at face value.
In our experience, it is useful to have a checklist to question the validity of statistics. We find checklists to be powerful tools in our research process—especially in our forensic accounting and risk work—and they are no less potent here. And one of the simplest first “checks” when it comes to evaluating a stat is what I will call the New Zealand vegetarian problem.
In my life, I have only met two New Zealanders in person (including the manager of our global bond fund, James Redpath). Both are male and both are vegetarians. Now every time I think of New Zealanders, I think “vegetarian” and “male.” Of course, I understand there are plenty of women and meat eaters in the population of 4 million New Zealanders (especially with all the wonderful lamb they have down there).
The New Zealand vegetarian problem is fundamentally about “what makes a good sample?” In statistics there are populations (all the members of a specified group) and there are samples (a portion of that population). Samples are used because it is almost always impractical to solicit an entire population; so we take a sample, evaluate it, and come to conclusions about what might be true in the overall population. A quality sample is one that is a) likely to be statistically significant (i.e. large enough) and b) a good representation of the overall population’s characteristics.
This may sound obvious but it is surprising how often we encounter the New Zealand vegetarian problem in daily life. Faulty conclusions can come from poor sample criterion: the sample size is not large enough; the inclusion criterion—attributes of subjects that are critical to the study—are not present; or the study falls victim to sampling bias—the samples are collected in a way that excludes certain members of the population.
An extreme example—when getting the sample right meant the difference between life and death—occurred in World War II. The RAF was losing a lot of planes to German anti-aircraft fire so they decided to reinforce them with armour. But where to put the armour? The obvious answer was to look at the planes that returned from missions, count up the bullet holes in various places, and then put extra armour in the areas that attracted the most fire. This was their sample criteria: the holes in the planes that returned.
Obvious but wrong. As Hungarian-born mathematician Abraham Wald explained at the time, if a plane makes it back safely even though it has a bunch of bullet holes in its wings, it means that bullet holes in the wings aren’t the problem. What you really want to do is armour up the areas that don’t have any bullet holes. Why? Because planes with bullet holes in those places never made it back.
In this case, the best sample is to look at the planes that never returned. In the absence of this sample, the next best thing is to look at all the places on the planes that had no holes (as Abraham Wald suggested). Thankfully for the RAF, they took Wald’s advice!
In investing, we must be wary of selecting good samples before we make inferences. It is dangerous to draw conclusions after seeing only a couple of data points. If we only talk to the CEO when assessing a company, we probably do not have enough information to be confident we understand the company’s culture. Similarly, one discussion with a disgruntled former employee may not give us an accurate cultural picture either.
We also need to make sure the sample we take is a good representation of the population we are evaluating. Arguably, the lessons we gain from reading the history of financial markets over the last five years would look very different from the lessons derived over the last 100. We need to be careful not to “overlearn” the lessons from a particular period: bear markets tend to teach us certain things, while bull markets teach us others.
Ultimately, our best advice is just to be aware of samples. When reading the news, question whether the underlying studies being cited utilized good enough samples for the conclusions being drawn (e.g., a study on rats is probably insufficient to conclude that eating meat is worse for you than smoking).
And for the record, a sample size of two is not meaningful. Not all New Zealanders are male and vegetarian.