This Week In Media Measurement

Transcript 29 lines

Cold Open Stats Overview Paper Walkthrough free_promo

Cold Open

Jenny If a website says someone clicked, how sure are we that a real person actually meant to?

Davis I'm less sure than the dashboard is, because a click can be a thumb, a bot, or now an AI assistant doing errands in the background.

Jenny Exactly, and I don't trust any chart that treats every click like a tiny human confession when so much online activity is automated traffic in a trench coat.

Davis So the next question isn't just how many visits happened, it's who or what acted, and whether that action changed anything a person cared about.

Jenny That's the measurement problem this week, from vanishing users to agent-aware analytics, which just means counting the actor behind the signal...welcome to This Week In Media Measurement on paperboy.fm.

Stats Overview

Davis This week is bigger across the board: about sixteen hundred query hits, one hundred seventeen qualified papers, three hundred forty-nine unique authors, and thirty-one countries in the mix.

Jenny Qualified papers rose from ninety-five to one hundred seventeen, so that's twenty-two more papers, up 23.2 percent, and the first thing I want to know is whether that growth is signal or just a wider net.

Davis The net did widen: query hits went from one thousand three hundred seventy-five to one thousand six hundred forty-two, up two hundred sixty-seven, or 19.4 percent, while the semantic shortlist stayed fixed at two hundred, meaning the system still only kept two hundred papers for closer reading.

Jenny So the filter got more selective, and the mix underneath is very survey-heavy: thirty-three surveys, twenty-three qualitative studies, and twenty quantitative papers, with social media at twenty-three papers, consumer behavior at ten, and digital media at eight.

Davis That fits the through-line: measurement here is less about counting who saw media and more about asking what the signal represents, whether it's a survey response, a consumer action, or a platform trace.

Jenny The author pool jumped from two hundred fifty-five to three hundred forty-nine, up ninety-four authors, or 36.9 percent, and countries rose from twenty-seven to thirty-one, led by Indonesia with ten papers, the U.S. with nine, and China with eight.

Davis And the author tiers make this feel like an open week: eighty-four first-time authors, meaning their first-ever paper in the metadata, one hundred seventy-six emerging authors, and eighty-nine experienced authors, so half the field here is early-career rather than the same senior names repeating.

Paper Walkthrough

Paper 1 The Vanishing User: Web Analytics in an Agent-Dominated Internet

Jenny Alright, let's get into the papers with a big measurement problem hiding in plain sight. Babu George and Divya Choudhary have a twenty-twenty-six position paper in Information called The Vanishing User: Web Analytics in an Agent-Dominated Internet, and the basic claim is that the old unit of web analytics, the human user, is getting shaky.

Jenny Plain version: a click may no longer mean a person wanted something. The authors argue that crawlers, bots, AI agents, LLM-powered agents, and more autonomous agents can all create web traces, and they flag three things that make LLM agents especially messy: identity discontinuity, meaning they don't keep one stable identity; task-based instantiation, meaning they appear for one job and vanish; and agent-to-agent loops, meaning one system may be talking to another with no human in the moment.

Davis So if a click might come from a person, a bot, or an AI agent acting for someone, what can a publisher still safely infer from it?

Jenny Not as much as dashboards usually imply. This isn't an empirical test with a measured share of traffic; it's a conceptual roadmap that synthesizes work on bot detection, agent architecture, web measurement validity, automated-system governance, and digital trace data, then proposes five measurement primitives: task chain, actor class, interaction provenance, objective alignment, and signal authenticity.

Jenny In normal language, they want analytics to ask what task produced the trace, what kind of actor made it, where the interaction came from, whether it matched a human goal, and whether the signal is trustworthy. The limitation is real, though: the paper names the measurement problem more than it proves how big the problem is.

Davis That still feels like the right opening frame for this week. If AI scrambles signals, then engagement and conversion can't just be treated as little pieces of human intention anymore; they need actor labels before anyone builds a newsroom plan, an ad report, or a product decision on top of them.

Paper 2 Contextual gatekeeping in a platformized news ecology

Davis That actor-label point is a nice bridge, because this next paper asks what happens after a signal gets believed by real newsrooms. S. Dvir-Gvirsman and Lidor Ivan call it Contextual gatekeeping in a platformized news ecology, meaning editorial choices made inside platform rules, dashboards, and audience feedback loops.

Davis The plain finding is that audience engagement did predict more coverage later, but not evenly. Across two and a half years of data from thirty-nine English-language outlets in the United States, the United Kingdom, Canada, and Ireland, the strongest bump showed up on Facebook pages of digitally born outlets, weaker on Facebook pages of legacy outlets, smallest on legacy websites, with digitally born websites in the middle.

Jenny How did they know engagement came before the extra coverage, rather than just showing that a topic was already getting hotter and everyone was chasing it?

Davis They modeled whether engagement in a topical beat in the prior month predicted later story counts, split by platform and by outlet lineage, so the timing is built into the test. That gives this more weight than a vibes-based dashboard story, especially with thirty-nine outlets across four countries from twenty-seventeen to twenty-nineteen, but it's still a historical baseline from a social-media-dependent era, not automatically a map of subscriber-first newsrooms today.

Jenny So the takeaway isn't, engagement controls journalism. It's sharper than that: this is the When Metrics Push Back thread, because the metric seems to push hardest where the newsroom is already organized around the platform, and weakest where the website and legacy routines still give editors more room to say no.

Paper 3 Made With AI: Consumer Engagement with Social Media Containing AI Disclosures

Jenny That last paper treated engagement as a signal that can push a newsroom around, and this one asks what happens when the platform changes the meaning of the signal itself: Stephan Carney, Ignacio Riveros, and Stephanie M. Tully have a Journal of Consumer Research paper from twenty-twenty-six called Made With AI: Consumer Engagement with Social Media Containing AI Disclosures.

Jenny The plain version is pretty sharp: when social posts carry an AI-generated-content disclosure, people engage less, not mainly because they think the content is worse or because they hate AI, but because the creator feels less personally present. The authors call that reduced parasocial connection, meaning the one-sided emotional bond a viewer feels with a creator, and they tie part of it to perceived effort.

Davis So is this really about AI, or is it about people feeling like the creator did less work?

Jenny They try to separate those with two kinds of evidence: real TikTok engagement after TikTok introduced its AI-generated content disclosure policy, and eight preregistered experiments, which means the tests were planned before the data were collected. Across those experiments, the drop didn't seem to come from quality worries, artificial-content wariness, or broad AI aversion, and disclosures that signaled more human effort softened the engagement hit, though the field evidence is centered on TikTok, so Instagram, YouTube, or niche creator communities could behave differently.

Davis That makes the policy problem more concrete: if disclosure is required, don't just slap on a label that says made with AI and walk away. This is the AI Scrambles Signals thread again, because the same like or comment now partly measures whether the audience still believes a human did meaningful creative work.

This Week In Media Measurement

Episode

Cold Open

Stats Overview

Paper Walkthrough

Paper 1 The Vanishing User: Web Analytics in an Agent-Dominated Internet

Paper 2 Contextual gatekeeping in a platformized news ecology

Paper 3 Made With AI: Consumer Engagement with Social Media Containing AI Disclosures

free_promo

Other Episodes

This Week In Media Measurement

Episode

Cold Open

Stats Overview

Paper Walkthrough

Paper 1 The Vanishing User: Web Analytics in an Agent-Dominated Internet

Paper 2 Contextual gatekeeping in a platformized news ecology

Paper 3 Made With AI: Consumer Engagement with Social Media Containing AI Disclosures

free_promo

Other Episodes

Get This Week In Media Measurement by email