On a number of occasions I’ve explained how scientific data is collected, analyzed, interpreted and presented. I’ve told how scientists must reveal their methods so others can try reproducing their results. I’ve stated that results are always tentative until new discoveries confirm or contradict them.
In everyday life we conduct ourselves differently. We can’t always wait for more information. Not every impression we form can be examined to assure it is free of contradictions with everything else we think or how we’ve acted in the past.
There’s usually not enough time to test the correctness of what we intend to do. We can’t search data bases to find supporting evidence for our actions. More often we rely on recollections of specific events of a similar nature to guide us in our judgments.
Surely we have innate instincts, but some of what we attribute to instincts is really an unconscious state of mind generated from experiences. Most of those experiences were anecdotal — meaning that though they share similarities with other events, they happened once and their application to a current situation may be a problem. You might get lucky, but anecdotes are generally unreliable. This is the best we can do as the day whizzes by.
Science can and does do better.
Scientists take time to explore the literature for contra-indications of their ideas. In daily tasks we don’t remember all the circumstances impacting an event, but scientists must design experiments to take into account extraneous factors.
Since there are frequently a number of factors dictating the outcome of a phenomenon, scientists must design filters for each effect to assure it is accounted for. Filters shield or isolate the experimental apparatus or methodology from undesired effects.
Depending on the nature of the study, this might mean doing two nearly identical experiments. In one all factors are left in play — the “control.” In the other, all factors, save the one of interest, are removed or held constant. The difference in outcome (barring synergistic effects) is the result of the factor of interest.
Perhaps a psychology experiment has the objective of measuring how people with a particular personality trait react to some event. A sufficiently large number of people with that trait would have to be tested along with another group, identical to the first except that its members haven’t been screened for the trait. The number of people tested is called the sample size.
Every aspect of the testing of the two groups must be identical. The room conditions (temperature, lighting, seating, etc.) should be identical. Perhaps even the time of day is important to control for how rested the subjects are.
The experimenter analyzes the data by looking at the differences in response from the two groups. Even within each group, of course, data will have some variation.
Those variations are themselves indications of factors that can’t easily be controlled. They might include one’s life experiences, memory acuity, general health and the like. However, if the experiment is well designed, and each group is composed of a large number of individuals, the results can be revealing.
Take this in contrast to anecdotes. For anecdotes, there is no preparation to see what “filters” might be used to control for irrelevant factors. They are by definition single events, so the sample size is as small as it gets. Unless the event was of particular import, analysis of the event was probably minimal. Most events are simply unconsciously stored away.
I’m no different than most people. My unconscious mind makes judgments and acts all the time. But, I do wince when I see hucksters attempting to influence the public by telling of some anecdotal story or isolated incident.
An example may help. I think the case for anthropogenic global warming is sound. But, I refrain from pointing to every bizarre, extreme weather event as definitive evidence. I don’t look at every higher than normal temperature reading of my backyard thermometer as proof.
I don’t obsess over a single satellite photo of a retreating glacier in one location in Greenland. An especially cold winter day or a jet stream fluctuation bringing an arctic “vortex” for a period of time isn’t a good sample size.
What I do is rely on experts having access to data from across the globe. They have historical records and understand historical patterns. They have satellite photos with accompanying data that dwarfs what we see out our window or what most pundits are likely to draw upon.
They have powerful computers to digest global weather data. This allows them to search for trends we would not see before they were in full swing. In other words, their sample sizes are huge and they are not so naïve as to not apply filters to account for extraneous factors.
Scientists ask, “What is going on across the globe?” They look at long term patterns as their equivalent of a control group. How far does this deviate from the norm over varying periods of time? From history, can we see how frequently these fluctuations can be expected?
The news media can’t seem to function without drawing on anecdotes. I certainly find them easier to relate to. They’re not as dry as meticulous analysis of mountains of data. But, when resources are available, we must do better.
Anecdotes are statistically unreliable and the chance they are not representative of deeper phenomena is high. We are not helpless in these matters. We can make some assessment of how the data used by a spokesperson was gathered. We can look at how they present their results.
Finally, pay attention to their willingness to be reviewed by larger, authoritative groups of experts. Most importantly, don’t take the easy way out. Be engaged and don’t succumb to appeals made from anecdotes.
Steve Luckstead is a medical physicist living in Walla Walla. He can be reached at firstname.lastname@example.org.