This Week In Media Measurement

Transcript 28 lines

Cold Open Stats Overview Paper Walkthrough free_promo

Cold Open

Jenny When a platform says something worked, how do you decide whether to believe it?

Davis I want to know what counted as worked, because a click, a watch, and an actual purchase are three very different promises.

Jenny Exactly, and the more polished the dashboard looks, the more I want to ask who was measured, when they were measured, and what else was happening that day.

Davis Because sometimes the boring answer beats the shiny one: the algorithm matters, but so do the video, the audience, and whether people just got paid.

Jenny So if TikTok Shop in Bali looks better with Gen Z when personalization, creative content, and payday sales line up, the question is whether measurement can separate magic from timing...welcome to This Week In Media Measurement on paperboy.fm.

Stats Overview

Davis This week is smaller but wider: 86 qualified papers out of 1,969 search hits, with 368 unique authors across 26 countries.

Jenny The qualified pile fell by 25 papers, down 22.5%, and I don't want to invent a cause; with social media leading at 18 papers, is the filter catching fewer outcome-testing studies, or was this just a quieter publication week?

Davis The upstream drop is even sharper: query hits fell to 1,969, down 1,263, or 39.1%, so the haystack shrank faster than the final stack, which makes me ask whether the venues, the keywords, or the timing changed.

Jenny But the pool broadened while volume fell: unique authors rose to 368, up 17.2%, and countries rose to 26, up 30%, with Indonesia at 9 papers, the U.S. at 5, and Spain at 2.

Davis The author mix also looks less top-heavy: 72 first-time authors, meaning first-ever paper, 147 emerging authors, and 149 experienced authors, so roughly 60% are not in the established bucket.

Jenny Method-wise, this is still more measurement than proof: 23 surveys, 15 qualitative studies, 12 quantitative studies, 6 experiments, and only 3 randomized trials, where people are assigned by chance, which fits the through-line but keeps the causality claims modest.

Paper Walkthrough

Paper 1 The effect of TikTok Shop algorithm and Payday Sale period on conversion rate among Generation Z in Bali

Jenny Alright, let's get into the papers with a very practical TikTok Shop question: when the feed feels perfectly timed, does that actually connect to buying? The paper is The effect of TikTok Shop algorithm and Payday Sale period on conversion rate among Generation Z in Bali, by Ni Putu Evie Sintya Wati and Nyoman Sri Subawa, published in Priviet Social Sciences Journal in twenty twenty-six.

Jenny They surveyed two hundred fifty Generation Z TikTok users in Bali who actively use TikTok and had experienced a Payday Sale promotion. The plain finding is that better algorithm personalization and more creative content were both linked to higher conversion rates, and conversion rate here just means moving from interest or browsing into an actual purchase.

Davis How much of this is the algorithm doing real work, and how much is just people being primed to buy because Payday Sale already tells them, hey, this is the shopping window?

Jenny That's exactly the tension, and the authors try to separate it by using SEM-PLS in SmartPLS four point oh, which is Structural Equation Modeling Partial Least Squares, basically a way to test several survey-based relationships at the same time. In their model, algorithm personalization and creative content each had positive significant effects, meaning the links were unlikely to be random in that model, and the Payday Sale period strengthened those links rather than replacing them. But it's still a survey of Gen Z TikTok users in Bali, so I wouldn't stretch it to every market, every age group, or claim it proves TikTok caused the purchase.

Davis That makes the takeaway pretty concrete for a seller: don't treat the algorithm as a magic conversion engine by itself. This fits the conversion-in-context thread, because the stronger story is algorithm plus creative plus sale timing, all landing at once when a young shopper is already ready to move.

Paper 2 The Influence of Live Streaming, Brand Image, and Perceived Usefulness of Online Review on Purchase Intention on TikTok Shop

Davis That algorithm plus creative plus sale timing point has a close cousin here, because this paper moves from the sale window to the live room itself: The Influence of Live Streaming, Brand Image, and Perceived Usefulness of Online Review on Purchase Intention on TikTok Shop, by Hapipah Hapipah and Yasri Yasri.

Davis They studied three hundred sixty consumers who had watched Unilever live streaming sessions and made purchases on TikTok Shop, and the cleanest finding is that live streaming had a significant direct effect on purchase intention. Brand image did not have a statistically significant direct effect, perceived usefulness of online reviews did not either, and consumer trust did not mediate the pathways, meaning trust was not the bridge carrying those factors into buying intention.

Jenny If trust did not mediate the effect, what exactly are people responding to in the livestream: the host, the demo, the chat, the discount pressure, or just the fact that everyone in the sample had already bought from Unilever on TikTok Shop?

Davis The authors used survey data and Partial Least Squares Structural Equation Modeling in SmartPLS four point oh, which is a way to test several linked survey relationships at once, including direct paths and indirect paths through trust. Their interpretation is that immediacy and interactivity are doing the work, but the big caution is that this is one brand context, Unilever on TikTok Shop, so it may not travel cleanly to smaller brands, luxury goods, or categories where trust is the whole purchase barrier.

Jenny That makes this a useful counterweight to brand-first thinking: if live commerce is in the plan, measure the live format as its own conversion mechanism instead of assuming brand equity will carry the sale. And it fits the conversion-in-context thread again, because three hundred sixty respondents and a real survey model give you a solid signal, but not a universal law of TikTok shopping.

Paper 3 Comparing the algorithmic fidelity of large language models in predicting human decision making: a case study of vaccination choice

Jenny That Unilever TikTok paper made me wonder what happens when the “sample” isn’t three hundred sixty shoppers, but a simulated audience. So this next one is Comparing the algorithmic fidelity of large language models in predicting human decision making: a case study of vaccination choice.

Jenny Plainly, the authors ask whether large language models can stand in for people when the choice is vaccination. Algorithmic fidelity means how closely the model copies real human decisions, and they tested five established model architectures using different inputs: basic demographics, revealed survey attitudes, and personalized media diets.

Jenny The striking part is that the models didn’t behave like neutral mirrors. Three of the five showed a clear pro-science alignment, meaning they leaned toward predicted pro-vaccination behavior even before you treat the output as a population forecast.

Davis So when a model predicts what people would do, are we measuring people, or are we measuring the model’s priors? And did they actually vary the media environment, or just ask the models to role-play survey respondents?

Jenny They used individual survey responses plus real online news content, then ran a counterfactual analysis, which means they changed the assumed media exposure and watched the prediction move. Specifically, they varied the ratio of authoritative content to low-credibility content, but the limitation is real: this is five specific LLM designs in a vaccination-choice setting, so it’s a warning about model fidelity, not a universal verdict on simulation.

Davis The practical takeaway is pretty sharp for anyone building synthetic audiences. Before you use an LLM to forecast health behavior, campaign response, or media effects, you need to test whether it reacts to exposure like your actual humans do, because in this algorithmic exposure and risk thread, the model’s bias can become part of the measurement system.

This Week In Media Measurement

Media metrics grow more human

Episode

Cold Open

Stats Overview

Paper Walkthrough

Paper 1 The effect of TikTok Shop algorithm and Payday Sale period on conversion rate among Generation Z in Bali

Paper 2 The Influence of Live Streaming, Brand Image, and Perceived Usefulness of Online Review on Purchase Intention on TikTok Shop

Paper 3 Comparing the algorithmic fidelity of large language models in predicting human decision making: a case study of vaccination choice

free_promo

Other Episodes

This Week In Media Measurement

Media metrics grow more human

Episode

Cold Open

Stats Overview

Paper Walkthrough

Paper 1 The effect of TikTok Shop algorithm and Payday Sale period on conversion rate among Generation Z in Bali

Paper 2 The Influence of Live Streaming, Brand Image, and Perceived Usefulness of Online Review on Purchase Intention on TikTok Shop

Paper 3 Comparing the algorithmic fidelity of large language models in predicting human decision making: a case study of vaccination choice

free_promo

Other Episodes

Get This Week In Media Measurement by email