Between the Numbers
26 March 2004
of taking a course on statistics and poring over data, the best way to get
a sense of which data to trust is through common sense. Does anyone really think that 1 in 5 mothers suffer from post-natal
Last week, sociologist Dr. Ellie Lee cast doubt
upon the increasingly popular theory that postnatal depression affects as
many as one in five new mothers, telling the BBC that the research underlying
that claim is "wrong."
"According to 'experts,' growing numbers of women are traumatized by childbirth
and are not capable of child-rearing without professional help," Lee said,
prompting a much larger question than the mental status of post-natal women:
How can we trust research?
The question is not trivial, since studies and statistics form the basis
for many of the laws under which we live. If they are wrong, then the laws
may be as well.
Short of taking a course on statistics and poring over data, the best way
to get a sense of which data to trust is through common sense. There are
five questions you should demand of any statistic.
Who Says So? The purpose of this question is to discover
possible bias on the part of those offering the data. The bias may be conscious
-- such as research conducted expressly to win a government grant. The bias
may be unconscious -- such as research conducted by those who have deep ideological
convictions that influence the questions asked.
Bias does not invalidate findings. Just because a researcher seeks funds
or has a personal opinion doesn't mean his finding that 2+2=4 is false. But
it does mean you should look at the math more closely.
How Does He Know? Imagine a researcher who rang doorbells
at random to ask, "Are you a criminal" or "Do you suffer stomach gas often?"
That researcher might discover the world to be both crime and gas-free --
not because it actually is, but because many people will not admit to either.
One of the most common methodological mistakes is to rely upon an unrepresentative
sampling, such as polling only Baptists or limiting your sample to 10 people.
"Seventy-five percent of Americans prefer milk to lemonade" is an impressive
finding until you realize that only 12 people were sampled, all of whom were
Wisconsin dairy farmers. At that point, the surprising statistic is that
25 percent preferred lemonade. Demand to know the exact question asked or
studied, the size of the sampling and whether it was random.
What does the competition say? Many studies contradict
past findings or constitute "surprising revelations." It could be that past
or competing studies were flawed; times may have changed. Data, like opinions,
can vary and a finding that "eating cheese increases your chance of cancer
by 12 percent" should be considered in light of past or current studies that
render different results. It is possible for a multitude of small surveys
to be conducted until one of them produces the desired results.
For example, after tossing a coin many times, it will land "heads up" nine
times in a row. From that isolated experience, a researcher could conclude
that a tossed penny will come up heads 90 percent of the time. Do other findings
What is missing? Does the data tell you enough to evaluate its statements?
Consider the statement, "the average salary at this company is $30,000 a
year." Ninety percent of employees may make much less than that amount but,
when total incomes are divided by total employees, $30,000 may be the "mean"
result. A "median" result reflects what the person at the exact middle of
the earning range takes home. The "mode" is nothing more than the most frequently
encountered figure. Does the figure $30,000 indicate a mode, a median or
Did Someone Change the Subject? A newscaster states, "reports
of domestic violence have increased" and concludes that "domestic violence
is on the rise." This conclusion is not justified because the increased reporting
may reflect nothing more than a greater willingness on the part of women
to contact the police or a greater willingness of police to file the reports.
The newscaster has changed the subject from increased reporting to increased
Does It Make Sense? Never allow a statistical finding
to override common sense or your own perceptions: guesstimate. That technique
involves taking a statistic to its logical conclusion and seeing if it reduces
to absurdity. Consider the alarming statement, "over 3,000,000 teenage girls
on welfare became pregnant this year."
Start with the total population of the U.S. -- roughly 300 million. Assume
that roughly half are male, leaving 150 million. Assume a uniform female
age-spread of one to 75 years, with teenagers (13-19) constituting approximately
9.3 percent, or 14 million. Assume every teenage girl can become pregnant.
Divide this figure by the reportedly three million pregnant welfare teens
and the ratio you get is 4.67. One in five teenage girls is not only on welfare
but has also become pregnant in the last year. Does this make sense, does
it accord with your own perceptions?
The research on postnatal depression may or may not be valid. Lee accuses
its advocates of constructing a problem, of "medicalizing motherhood." Lee
states, "There is every possibility that ... parents will come to experience
the normal disruption that parenting brings with it, as highly disabling,
and find themselves less able to manage ... This risks branding an essential
part of life a hazard."
Our society rewards those who construct problems. They receive financing
and media attention, write books and become "experts." Statistics are tools
and those who wield them should be neither glamorized nor ignored. But they
should be required to answer basic questions before being included in that
rare category: purveyor of truth.
Wendy McElroy is the editor of ifeminists.com
and a research fellow for The Independent Institute in Oakland, Calif. Her
new book is Liberty for Women: Freedom and Feminism in the 21st Century.
Reprinted with permission of ifeminists.com.
Email Wendy McElroy
this Article to a Friend