title=”typifying a national phenomenon—not.”>typifying a national phenomenon—not.
I never imagined myself a data wonk. I’m terrible at math, know nothing about statistics. If Sandra Jamieson weren’t my research partner, the Citation Project would have no quantitative analyses whatsoever.
And yet. Yet my work on the CP has made me acutely aware of how claims can—and cannot—be made with quantitative data. I’m not talking about skewed bar charts here, but more about sample size versus claim size. I have this awareness because of all the research we’ve consulted for the CP, most of it from single campuses. It’s not that these research projects aren’t useful; on the contrary, they raise all manner of important questions, and they allow a window into the behaviors of isolated groups. Susan Blum’s research at Notre Dame? Very provocative. But as you read her book carefully, you find her dancing between the admission that hers is a one-campus study, and her desire to claim that her findings illuminate what college students everywhere are thinking and doing.
Thus it was with the pilot data for the Citation Project. Tricia Serviss, Tanya Rodrigue, and I designed and conducted a study of students’ source use. We conducted the research at our own institution. We published our findings in the journal Writing and Pedagogy. People were interested. All was well.
Then, however, Sandra joined the project as co-principal researcher. Two more campuses came on board. We analyzed the data—now using SPSS. Our findings began to change. Thirteen more campuses came on board. More SPSS. Our findings began at last to settle into a pattern. Each school had its own distinct profile, but few were remarkably different from the aggregate. And the findings from the pilot? One of the chief findings was overturned in the 16-campus study. At Syracuse, incidents of patchwriting outnumbered paraphrase two to one. In the 16-campus study, paraphrase outnumbered patchwriting two to one.
That’s an important difference. So I’ve become sensitive to claims like these, from the New York Times article linked here: “90 percent of the 2,800 boys in the survey — who lived in the Minneapolis-St. Paul area, but typify what doctors say is a national phenomenon.”
Oh, please. On what basis do doctors claim that the boys of Minneapolis-St. Paul are typical of the nation? Is there other data we’re not hearing about in this story? The only way we can know the answer is to go to the research that’s being reported.
That’s a second thing that I’ve become very alert to, especially in my teaching: I’ve become acutely uncomfortable with students’ willingness to find a story like this and regard it as a good source on which to build claims. No, it isn’t. What this story does is provide a journalist’s reading of someone else’s research. It is not the research itself. So I find myself insisting, even with my first-year writers, that students find and read the research being reported, and not depend exclusively on what even a good journalist has to say about it.