This Week In Media Measurement

Transcript 28 lines

Cold Open Stats Overview Paper Walkthrough free_promo

Cold Open

Jenny When you buy something after seeing a bunch of ads, which ad should get the credit?

Davis My lazy answer is the last one I clicked, but that mostly rewards the ad standing closest to the cash register.

Jenny Right, and I'm suspicious whenever a model says it knows what caused a sale, because without an experiment, cause can become a very confident guess.

Davis That's why this one hooked me: it learned from 2,226 Meta ad experiments and made campaign calls that disagreed with randomized tests, the split-the-audience way to see what an ad actually changed, less often than seven-day last-click attribution, the rule that gives all credit to the final click ... welcome to This Week In Media Measurement on paperboy.fm.

Stats Overview

Jenny This week we had 121 qualified papers out of 2,830 query hits, with about 400 unique authors across 27 countries. So the feed got narrower, but the people inside it got broader.

Davis Right, and that's the weird movement: qualified papers rose from 117 to 121, about a 3.4 percent bump, while raw hits fell from 4,848 to 2,830. That sounds less like the field got bigger and more like the search pulled in less noise, or the shortlist got closer to the measurement question.

Jenny But I'd be careful with that claim, because we don't have the venue mix here. If the drop is mostly duplicate platform studies or off-theme social media work disappearing, that's useful; if it's database coverage changing, that's just plumbing.

Davis The author picture is more interesting: unique authors jumped from 318 to 384, up about 21 percent, while countries fell from 33 to 27. So fewer national contexts, but more hands on the work, which can mean bigger teams, more multi-author empirical papers, or a cluster around a few active regions like Indonesia, China, India, and the U.S.

Jenny And the career mix is pretty balanced: 90 first-time authors, meaning first-ever paper in the metadata, 137 emerging authors, and 157 experienced authors. That's roughly 23 percent first-time, 36 percent emerging, and 41 percent experienced, so this isn't just a senior scholar week.

Davis Theme-wise, social media still dominates with 29 papers, then digital marketing at 9 and consumer behavior at 8, and the methods are survey-heavy with 42 survey papers, 21 quantitative studies, and 20 qualitative ones. That fits the episode's through-line: measurement is trying to move toward causal claims and auditable infrastructure, but a lot of the evidence still starts with people answering questions.

Paper Walkthrough

Paper 1 Predicted Incrementality by Experimentation for Ad Measurement

Jenny Alright, let's get into the papers, and this first one is very much the spine of the week: Predicted Incrementality by Experimentation for Ad Measurement, by Brett Gordon, Robert Moakler, and Florian Zettelmeyer, in Management Sciences in twenty twenty-six.

Jenny The plain idea is this: if you can't run a clean experiment on every ad campaign, maybe you can train a model on campaigns where you did run clean experiments, then use that model on the rest.

Jenny They build this from two thousand two hundred twenty-six Meta ad experiments, and the target is incrementality, meaning the extra conversions caused by the ad, not just the conversions that got clicked near the end.

Davis So what exactly is the model allowed to learn from after the campaign is already over, because that sounds like the place where measurement can quietly cheat?

Jenny That's the clever part: because the training data comes from randomized controlled trials, meaning some people were randomly held out so the causal effect is known, PIE can use post-campaign summaries like test-group outcomes, exposure rates, and seven-day last-click conversions as predictors rather than causal controls.

Jenny In decisions, PIE disagreed with the experiment-based answer in only eight to twelve percent of campaigns, while seven-day last-click attribution disagreed in twelve to twenty percent, though the big caveat is that this is strongest inside a huge Meta experiment archive and still depends on having a relevant bank of past RCTs.

Davis That feels like the right opening move for this episode: don't treat last-click as fallback truth when the experiment is missing; if you need a proxy, make it a proxy trained on causal evidence.

Paper 2 Engagement as evaluative infrastructure, and the quiet transfer of value from firms to platforms

Davis That word proxy is the bridge here, because Alina Koz, Chris Soulz, and Galvin Kuan Sian Lee are much less forgiving about it in Engagement as evaluative infrastructure, and the quiet transfer of value from firms to platforms.

Davis Their claim is that engagement is not just a messy signal of customer value. It is evaluative infrastructure, meaning a platform-built scoring system that tells firms what counts as success, and the paper builds eleven propositions around that shift.

Jenny So how would a firm know whether the engagement metric is measuring customer value, or just platform value dressed up as customer value?

Davis The authors say the hard part is evaluative asymmetry, which means the party defining the measure is not the party acting on it, and their money comes from different places. A brand may chase comments, shares, or watch time, while the platform monetizes the extra attention and can reprice access to that audience later. The catch is that this is a theory paper in Digital Enterprise Studies from twenty twenty-six, so the claims are sharp, but they are not tested on a dataset here.

Jenny That makes the practical takeaway pretty stark: treat platform engagement as a managed dependency, not a neutral dashboard, because this is Platform Metrics, Platform Power in its cleanest form.

Paper 3 Poster: Legal Imprints as an Identity Anchor for Web Measurement

Jenny Okay, if the last paper was about platforms defining the metric, this one asks whether we can at least identify who is actually behind the web page: Poster: Legal Imprints as an Identity Anchor for Web Measurement, by Gerrit Seifert and Ralph Holz at the twenty twenty-six ACM Internet Measurement Conference.

Jenny Plain version: don't count domains as if each domain were its own actor, because one operating company can sit behind a whole pile of sites. They crawl eight point two million dot-de domains, use German legal imprints, the Impressum, which is the legally required page naming the responsible company, and end up with seven hundred seven thousand four hundred twenty-six verified domain-to-operator mappings.

Davis Why does measuring the company behind a domain change the story compared with measuring the domain itself?

Jenny Because domain counts can make the web look more fragmented than it is, while company-level counts can show centralization through ownership, hosting choices, and corporate structures. Their pipeline combines deterministic patterns, meaning fixed rules for fields like company names and addresses, with constrained local LLM extraction, meaning a language model run in a bounded way to pull legal fields, then checks records against the German commercial register, with reported quality of ninety-nine point four percent precision and ninety-six point nine percent recall. The big limitation is geographic: the strongest evidence is for German dot-de domains, while France and Spain are still described as ongoing work.

Davis That feels like Measurement Infrastructure Rebuilt in a very practical way, because for web concentration, attribution, or ad supply-path analysis, a domain is often just the label on the door, not the business inside the building.

This Week In Media Measurement

Media Measurement Gets More Practical

Episode

Cold Open

Stats Overview

Paper Walkthrough

Paper 1 Predicted Incrementality by Experimentation for Ad Measurement

Paper 2 Engagement as evaluative infrastructure, and the quiet transfer of value from firms to platforms

Paper 3 Poster: Legal Imprints as an Identity Anchor for Web Measurement

free_promo

Other Episodes

This Week In Media Measurement

Media Measurement Gets More Practical

Episode

Cold Open

Stats Overview

Paper Walkthrough

Paper 1 Predicted Incrementality by Experimentation for Ad Measurement

Paper 2 Engagement as evaluative infrastructure, and the quiet transfer of value from firms to platforms

Paper 3 Poster: Legal Imprints as an Identity Anchor for Web Measurement

free_promo

Other Episodes

Get This Week In Media Measurement by email