Measuring the impact of media accurately is challenging. Findings of minimal effects abound when intuition tells us that an activity that an average American engages in over forty hours a week is likely to have a larger impact. These insignificant findings have been typically attributed to the frailty of survey self-reports of media exposure, though debilitating error in dependent variables has also been noted as a culprit. Others have noted weaknesses in research design, inadequate awareness of analytic techniques that allow one to compensate for the error in measures, etc. as stumbling blocks.
Here are a few of the methods that have been used to overcome some of the problems in media research, along with some modest new proposals of my own:
- Measurement
Since measures are error-prone, one strategy has been to combine multiple measures. Multiple measures of a single latent concept can be combined using latent variable models, factor analysis, or even simple averaging. Precaution must be taken to check that errors across measures aren’t heavily correlated, for under such conditions improvements from combining multiple measures are likely to be weak or non-existent. In fact, deleterious effects are possible.Another point of worry is that measurement error can be correlated with irrelevant respondent characteristics. For instance, women guess less than men on knowledge questions. Hence responses to knowledge questions are a function of ability and propensity to guess when one doesn’t know (tallied here by gender). By conditioning on gender, we can recover better estimates of ability. Another application would be in handling satisficing.
- Measurement of exposure
Rather than use self-assessments of exposure, which have been shown to be correlated to confounding variables, one may want to track incidental consequences of exposure as a measure of exposure. For example, knowledge of words of a campaign jingle, attributes of a character in a campaign commercial, source (~channel) on which the campaign was shown, program, etc. These measures factor in attention, in addition to exposure, which is useful. Unobtrusive monitoring of consumption is, of course, likely to be even more effective. - Measurement of Impact
- Increased exposure to positive images ought to change procedural memory and implicit associations. One can use IAT or AMP to assess the effect.
- Tracking Twitter and Facebook feeds for relevant information. These measures can be calibrated to opinion poll data to get a sense of what they mean.
- Data Collection
- Data collection efforts need to reflect half-life of the effect. Recent research indicates that some of the impacts of the media may be short-lived. Short-term effects may be increasingly consequential as people increasingly have the ability to act on their impulses – be it buying something, or donating to a campaign, or finding more information about the product. Behavioral measures (e.g. website hits) corresponding to ads may thus be one way to track impact.
- Future ‘panels’ may contain solely passive monitoring of media use (both input and output) and consumption behavior.
- Estimating recipient characteristics via secondary data
- Geocoded IP addresses can be used to harvest secondary demographic data (race, income, etc.) from census
- Para-data like what browser and operating system the customer uses etc. are reasonable indicators of tech. savvy. And these data are readily harvested.
- Datasets can be merged via matching or by exploiting correlation across items and by calibrating.