This Week In Media Measurement

Transcript 27 lines

Cold Open Stats Overview Paper Walkthrough free_promo

Cold Open

Jenny If an app seems to know what you like, does that actually make you stick around?

Davis My lazy answer is yes, because a decent row of recommendations saves me from scrolling, but I've also quit apps that knew my taste and still felt overpriced.

Jenny That's the part I don't buy in the magic-recommendation story: knowing me is not the same as being worth another monthly charge.

Davis And this week we've got research saying measurement has to stop treating clicks, mentions, and personalization as stand-ins for value, because in one subscription video study the algorithm didn't directly predict renewal intention, perceived value did ...welcome to This Week In Media Measurement on paperboy.fm.

Stats Overview

Jenny This week is smaller and tighter: six hundred fifty-three hits, seventy-seven qualified papers, eighty-one authors, and twenty-four countries. That already tells me the measurement story is narrower than last week, not absent.

Davis Right, qualified papers fell from one hundred nine to seventy-seven, down thirty-two papers, or about twenty-nine percent. The practical read is that we're hearing from fewer studies, so each theme has to work harder before we call it a trend.

Jenny The bigger drop is the search pool: two thousand eight hundred eleven hits last time, six hundred fifty-three now, down about seventy-seven percent. What's driving that — a quieter publication week, tighter indexing, or fewer papers that actually name media measurement instead of circling it?

Davis Method-wise, surveys led with twenty papers, then qualitative work at eighteen, and quantitative studies at fifteen. That fits the through-line: the field is still counting behavior, but it's also asking people what attention, trust, and user experience feel like.

Jenny The author mix is pretty balanced: twenty-eight first-time authors, meaning their first-ever paper in this metadata, twenty emerging authors, and thirty-three experienced authors. So about a third brand new, about a quarter early-career, and about forty-one percent established.

Davis Theme sweep: social media is out front with twelve papers, student engagement has five, and consumer behavior plus adolescents plus digital media sit at four each. So the center of gravity is moving away from easy proxies like clicks, toward messier questions about value, attention, and who the measurement actually serves.

Paper Walkthrough

Paper 1 Replacing Undefined Earned Media Value Metrics with a Rigorous Mention Quality and Impact (MQI) and Mention Earned Media Value (mEMV) Framework

Jenny Alright, let's get into the papers with Replacing Undefined Earned Media Value Metrics with MQI and mEMV, a twenty twenty-six framework paper that basically says brands need to stop treating unpaid mentions like magic money.

Jenny The plain version is this: instead of saying a mention was worth some vague ad-equivalent amount, the authors propose a one to ten Mention Quality and Impact score, where MQI means a structured grade for how valuable a mention actually is.

Jenny That score looks at sentiment, engagement, credibility, and AI visibility, then feeds into mEMV, or Mention Earned Media Value, which converts the mention into money using CPM, meaning cost per thousand impressions, with platform-specific benchmarks.

Davis What would make this more than a tidier version of the same old earned media value problem?

Jenny The authors are building the measurement recipe here, not testing it across hundreds of campaigns, so the useful move is the transparent scoring system and bounded conversion-intent adjustment, but the big missing piece is empirical validation in real service-marketing campaigns.

Davis That feels like the opening theme in miniature: measurement beats proxies, because if someone hands you an earned media value number, the first question should be which quality signals are inside it and how tightly the money claim is bounded.

Paper 2 Persona-prompted LLM agents achieve modest but genuine prediction of human social media reactions

Davis That bounded-money-claim idea carries over neatly here, because this paper is basically asking how bounded the AI claim should be: Persona-prompted LLM agents achieve modest but genuine prediction of human social media reactions, in Scientific Reports in twenty twenty-six, tested one thousand five hundred eleven Serbian participants and twenty-seven large language models.

Davis Plainly, the agents could guess human reactions better than chance, but not well enough to call them little replicas of people. Across one hundred twenty thousand-plus agent-persona combinations, they hit seventy point seven percent overall accuracy on reactions like like, dislike, comment, share, or no reaction, and in the stricter like-versus-dislike test they reached an MCC of zero point two nine, where MCC is a score that checks whether a prediction is really better than lucky guessing.

Jenny But if a simpler text model beats the persona agent, are we measuring simulated people at all, or are we mostly measuring what the post itself says?

Davis That’s the key check, and the authors pretty much land on your side of it. A conventional supervised classifier using TF-IDF, which is just a way to count which words matter in a text, reached an MCC of zero point three six, higher than the agents’ zero point two nine, while the choice of LLM moved performance by a thirteen-point spread; so the signal looks more like content semantics than individualized behavioral simulation, and the big caveat is that all of this comes from Serbia, not a global population.

Jenny So the practical takeaway is cautious but useful: you might use LLM personas for rough engagement forecasting, especially when you don’t have training data, but not for precise individual targeting. This fits the algorithms-are-not-magic thread too, because the sample is substantial enough to take the signal seriously, and narrow enough that I wouldn’t want a campaign strategist treating it as a universal map of human reaction.

Paper 3 Algorithmic deception: a scoping review of mass communication research on algorithmic dissemination of mis- and disinformation, propaganda, fake news, and conspiracy theories

Jenny That TF-IDF result is a nice bridge, because Algorithmic deception asks almost the same measurement question from the platform side: are we seeing what people want, or what ranking systems make easier to see?

Jenny Klevanskaya, Siegel, and Tsegmid review thirty-five mass communication articles, and the plain finding is that search engines and digital algorithms often give deceptive material more visibility. Their umbrella term is deceptive communication, meaning misinformation, disinformation, propaganda, fake news, and conspiracy theories all grouped as content that misleads people in public life.

Davis But how do these studies know the algorithm amplified deception, instead of just reflecting what people already wanted to click?

Jenny That's the hard part, and the review is more map than verdict. They did a scoping review, which means they surveyed the shape of a research area rather than pooling one clean effect size, using semi-structured searches in EBSCO's Communication and Mass Media Complete, screening titles and abstracts, and having three researchers code each article for visibility, contributing factors, and mitigation ideas. The strongest through-line is high visibility, with possible causes like appetite for polarizing content and platform profit incentives, but the review also says studies handle search personalization inconsistently, meaning the way results change by user, location, or history isn't controlled the same way across papers.

Davis So the practical takeaway is very measurement-y: don't stop at detecting bad content; measure whether the system is making it more visible, and ask how personalization was handled. Thirty-five papers is enough to take the pattern seriously, but not enough to pretend there's one universal algorithmic effect, which fits the visibility-needs-context thread almost too neatly.

This Week In Media Measurement

Media Metrics Get More Demanding

Episode

Cold Open

Stats Overview

Paper Walkthrough

Paper 1 Replacing Undefined Earned Media Value Metrics with a Rigorous Mention Quality and Impact (MQI) and Mention Earned Media Value (mEMV) Framework

Paper 2 Persona-prompted LLM agents achieve modest but genuine prediction of human social media reactions

Paper 3 Algorithmic deception: a scoping review of mass communication research on algorithmic dissemination of mis- and disinformation, propaganda, fake news, and conspiracy theories

free_promo

Other Episodes

This Week In Media Measurement

Media Metrics Get More Demanding

Episode

Cold Open

Stats Overview

Paper Walkthrough

Paper 1 Replacing Undefined Earned Media Value Metrics with a Rigorous Mention Quality and Impact (MQI) and Mention Earned Media Value (mEMV) Framework

Paper 2 Persona-prompted LLM agents achieve modest but genuine prediction of human social media reactions

Paper 3 Algorithmic deception: a scoping review of mass communication research on algorithmic dissemination of mis- and disinformation, propaganda, fake news, and conspiracy theories

free_promo

Other Episodes

Get This Week In Media Measurement by email