The Problem with Anecdotal Evidence

It's been interesting to watch and to listen to the latest debate in Washington DC over the extension of the so-called "Bush tax cuts" and unemployment benefits. What I have been cogitating on the past few days is how often anecdotal evidence is employed to draw a sweeping conclusion. In fact, the use of anecdotal evidence (utilizing a single or a few incidents) to draw conclusions is quite common, and all of us employ such evidence from time to time. It is not that anecdotal evidence is unimportant, but too often we use it to support or undermine conclusions which, on further and more sophisticated reflection, are obviously strained.

For example, in reference to the recent debate over whether or not extending tax cuts to wealthy Americans is good for the economy-- those who support the extension of such cuts have argued that business owners by and large will put that extra money into their businesses which will result in more jobs and less unemployment. A true story or anecdote will then be told of the small business owner who says he will take that money he will not have to pay in taxes to expand his restaurant, which means he will have to hire more people. This one incident is meant to demonstrate the conclusion that lower taxes for the wealthy is good for the economy. Of course, on the other side, those who oppose extending the tax cuts tell the story of the business owner who says what extra money he does not pay in taxes will either be saved and sit on the sidelines of the economy, or it will be used to take a vacation. Again, the one incident is supposed to demonstrate that such tax cuts are not helpful and therefore should not be extended.

In reference to the extension of unemployment benefits, those who support the extension will appeal to the person who has been out of work for two years and has been hustling the entire time to find work, but cannot. (I know some of them.) The other side will point to those who have decided to wait to find work, living off the government's dime until the last possible minute. (I have spoken to a few local business owners who have told me that they have had more than a few unemployed individuals who come in to apply for a job, but honestly tell them they are not yet looking; they just need to report that they have been. A business owner, who is a friend of mine, has told me that he has offered several jobs to persons who have turned them down because they do not yet want to return to work.) Once again, anecdotal evidence is used both to demonstrate that people need benefits extended and to prove that as long as the government pays unemployment, people will not be forced to seriously look for a job.

Another example of how anecdotal evidence is used to prove a strained point is the recent debate over health care. Those who desire a single-payer system will tell a truly sad and tragic story of someone without health insurance in the U.S. who has gone bankrupt over insurmountable hospital bills. They will also interview someone from Canada or the U.K. who just loves their health care system in order to tout its benefits. Those on the other side of the health care debate who oppose a single-payer system will tell the story of the person in the United States whose life has been prolonged with the latest cancer drug which is not available in other countries. They will also find someone from a "socialized" health care system to tell the story how she came to the United States to get brain surgery that saved her life, while she was getting worse waiting for months in Canada (saw this story just a couple of weeks ago). Of course, both sides have their horror stories they can appeal to in order to make their case.

It is not that any of these stories are false-- in fact, they are true (and perhaps all of them are spun somewhat as well). The problem is that they are utilized to employ conclusions that do not follow without further overall evidence. I have known parishioners who had a bad experience at a certain hospital, and they conclude from their own experience that the entire hospital is incompetent and all the surgeons there are "butchers." But then I know other church folk who had a wonderful experience at the same hospital and cannot say enough good about the place. (By the way, we do this with mundane things as well-- e.g. the one bad meal we had at a restaurant.) We draw conclusions based on a few personal pieces of the puzzle when the picture on the puzzle is not yet clear.

The point here should be obvious-- anecdotal evidence is not irrelevant, but by itself it cannot do what we so often want it to do-- draw general conclusions from individual stories that appear to prove the point. The one problem with this kind of logically fallacious thinking is that the anecdotes that appear to disprove the conclusion are either ignored or, at the very least, minimized.

Without ignoring the stories we also need to remember that often the larger story needs to be supported by larger trends of which those anecdotes may be a significant part. The problem is that we too often assume that our experience is everyone's experience. In addition, many persons tend to spend most of their time with other people who think like they do. Thus it becomes a short step from what you and your six friends think to what the majority believes.

The point here is not to abandon anecdotal evidence-- indeed, it is impossible to do so-- rather it is a plea to keep such evidence in its place and not to ask it do what it possibly cannot. Perhaps, if we do so, our debates will be more substantive and nuanced and we will at least appreciate the views of those who take opposing positions, even if we do not agree-- after all... everyone has their stories to tell.