Media analysis 101: Metrics are everything

The Culture and Media Institute today released a study of news coverage of vice-presidential nominee Sarah Palin showing “character assassination” by the major broadcast news networks ABC, NBC and CBS. The report also concluded “the public cannot trust what they’re hearing from [the networks].” The research is getting some major attention in the blogosphere and on several major news outlets, so I thought I would comment on its methods, findings and conclusions considering I believe CMI seriously cherry-picked their data. I should note that I’m commenting on the full report the organization provides on their web site, which offers the most details about how they arrived at their conclusions.

First, their findings - CMI found three narratives about Sarah Palin:

1. Palin Is a Dunce, Lacking Qualifications and Intellect
2. Conservatives Are Rejecting the Dunce
3. Palin Is an Obama-Attacking Demon

While there are several inconsistencies in the way CMI coded the media coverage for positive, negative and neutral (more on that in a bit), the biggest issue I have with the report is the number of stories actually included in the study. CMI says there were 69 news segments that covered Sarah Palin over two weeks (Sept. 29 – Oct. 12) on ABC, NBC and CBS. If there really were only 69 stories on the networks, the average major network would have aired less than 2 stories per day that mentioned Palin. Perhaps this would be possible during a slow news week (or two), but certainly not during this campaign cycle or this specific time period, which included the run-up to the vice-presidential debate. The reality is that Palin was mentioned in more than 200 stories on major networks during those two weeks. CMI chose to include only the stories that fit their predetermined conclusion.

A solid media analysis is based in solid, objective coding of news coverage. CMI coded stories almost entirely on subjective points. They say, “Negative stories contained more unfavorable elements than favorable, or gave preferential treatment to Biden.” CMI doesn’t actually define what unfavorable is. Is a piece coded unfavorable because of certain buzzwords, the amount of speaking time allotted to campaign spokespeople or number of minutes devoted to one candidate over the other? CMI simply doesn’t give us these objective metrics. Furthermore, many of the items CMI coded negative, or what they claim proves the networks perpetuated an attack narrative, is actually spurred by Palin’s newsworthy comments that Barack Obama was “palling around with terrorists.” This soundbite was designed to get media attention, so the fact that it was picked up by the networks doesn’t mean the stories were “negative.” Again, it depends on what metrics we’re using, which CMI doesn’t clearly define.

In addition, CMI didn’t even bother to examine coverage of the Troopergate scandal, which should have been included in their analysis because the decision came down during the last week over coverage. I have no doubt that coverage of Palin was critical, but how much of it was “media bias” versus the factual reporting that she violated state ethics laws? We don’t know because CMI didn’t look at the stories and didn’t objectively define their coding method.

Lastly, if this were a serious media analysis, CMI could have gotten media attention by using words less alarmist than “dunce”, “demon” and “ridicule” considering those words didn’t even show up in the coverage they looked at. Their study was clearly designed to support a preset conclusion and their metrics were broad enough to fit a small selection of news coverage. There may actually be some unfair reporting of Sarah Palin (as is the case with each one of the candidates), but this particular study doesn’t make the cut and certainly doesn’t support the extreme conclusion that the networks cannot be trusted.