MMMs influence $30bn in global media spend - here’s why marketing must watch out for media v market mix models, agency principal deals and Google’s Meridian, Meta’s Robyn for bias v rivals
Last week MMM provider Mutinex announced an open source framework to compare and test models from any vendor - and landed qualified support from a broad sweep of rivals and users. Mutinex co-founder and CEO Henry Innis fears market mix modelling risks becoming a “wild west playground of mess with differing standards that degrades trust in the overall industry”. So, a transparent comparison and governance framework for buyers was needed for all MMMs, like AI firms already do. Some Mutinex rivals went further – Analytic Partners and Prophet said the open source comparison model was particularly apt for the open source MMMs developed by Google and Meta, suggesting bigger agendas – or underlying model assumptions – that protect their own ad-funded assets like search and social video. Meanwhile, marketers like Uber APAC lead Andy Morley have tried but don’t trust the recommendations from any MMM trials they’ve conducted due to volatility and fast-moving variables that shape demand in car rides and food and consumer goods delivery. But Morley, like most canvassed by Mi3, say a more informed, educated market, is critical for the sector’s trust and longer-term viability. Morley might yet be swayed although “it’s a way off”.
Uber’s MMM alternative
Uber’s APAC CMO Andy Morley doesn’t trust any of the results from MMM models his team have tested for the business. There were just too many quickly changing variables and not enough robust historical data for relatively new and fast changing categories that Uber operates in to make accurate predictions, Morley says. For consumer goods brands, Morley says MMMs are more appropriate.
Instead, Morley says Uber has defaulted to triangulating current marketing theory and measurement tools around “how brands grow” – brand tracking, sales, brand-up lift scores and incrementality tests where markets on a geographical basis are exposed to marketing activity and compared to those that are not. For Morley, who backs the MMM open source initiative, he remains open but cautious and concerned that progress around marketing effectiveness principles will “go backwards” if his marketing peers don’t apply enough rigour when using MMM models.
Others like Omnicom Media Group’s Chief Operating Officer Kristiaan Kroon raised principal media deals – particularly a variant known as “bundling” in which media agency groups sell a pre-packaged mix of channels they arbitrage typically from a blend of premium and second-tier publisher advertising inventory – as a key data input that “reduces models effectiveness”.
Bias in models
Perhaps the most challenging for parts of the market are the warnings from MMM vendors like Analytic Partners and Prophet, who say despite the open source moniker, MMM products from Google and Meta pose risks because of underlying assumptions built into their models that protect their own advertising-supported channels like search and social video. And because they were being white-labelled by various agency groups, the ability to customise and skew recommendations for specific agendas was a “live” problem.
Prophet co-founder and CEO Jordan Taylor-Bartels and Analytic Partners APAC chief Paul Sinkinson were blunt with their market warnings. “You can build a model that would potentially recommend putting money where there’s a bigger [media] rebate or puts money where somebody wants money to go rather than where it helps the client," Sinkinson told Mi3 last week. Taylor-Bartels concurred and raised Analytic Partners on another – unknown assumptions in the tech giants MMM models that favour their own advertising channels. In-sample metrics, he says, are major watchouts for buyers.
It essentially means when an MMM model is assessed on the same data it was trained on, it can appear more accurate that it is. By applying MMM models to real-world data for a brand or category it’s not seen before gives better visibility on its prediction accuracy.
Black box fight
“We’ve got a fight between black box solutions, or more productised solutions [Prophet, Analytic Partners, Mutinex] and the likes of Robin and Meridian,” says Taylor-Bartels. “Right now Google is using Meridian and open source as a marketing tactic rather than an academic practice which infers trust. Meridian has a marketing budget, and it has a partnership budget and typically you associate open source with trust and an academic approach to make sure that we're producing research that is open source to make it better.”
Taylor-Bartels argues these open source models can be used by a “variety of parties” but “so many components of those aren't necessarily controllable as they should be. Not all organisations want everything to be open source. It seems to be a creative narrative coming from those who benefit most from it.” In Google's case, Taylor-Bartels says the objective is to use open source to “get agencies involved”. Agencies, he says, then sell the product to their clients and “Google then controls measurement or controls part of that conversation. Google then can protect, obviously, the biggest thing for them, which is search. It's quite an easy kind of Sherlock Holmes logic that can be applied and this is one of the few times where I believe it's true.”
When asked if he thought these assertions had any validity, Innis at Mutinex said: “I think it would be very interesting for everybody to run the open source framework that we've developed - I think there'd be some extremely interesting results that would come out of it.”
Principal, bundled media problems
On another contentious practice raised by OMG’s Kristian Kroon – arbitraged agency principal media deals - which many claim to be widespread, Innis acknowledged Kroon’s concern that “bundled” media products, at least dilute MMM model effectiveness. He said similar concerns were raised in the US recently at an Association of National Advertisers [ANA] conference by then head of media technology and analytics at The Hershey Company, Kyle Shank.
“So a premium video asset, for example, is very different from something random on some deep, dark corner of the web,” says Innis. “And when those two things are bundled together and just reported up as video spend, it can certainly make it a lot more difficult to establish the true effectiveness of a channel. One of the things that we're increasingly doing is trying to get to the bottom of whether there's a line item like video or a line item like social - I don't think you can just take social as a line item anymore. You need to break it up into its various factors to align to best practice - you've really got to isolate it down because with a lot of these grouped categories, there can be a lot of different things in them that can confuse the signal in a model.”
He said correct data inputs and definitions were critical for predictive accuracy – broader principal media deals, for instance, were often entered at the start of a month “when a deal was cut” between an agency group and a media owner when “really, they’re spending it across the next three months. That’s a common thing you have to untangle and Kroon’s right – it degrades the quality of a model. I think it’s one of the reasons that various groups charge high fees [to clients] for the first data ingestion for an MMM. They’ve got to clean up a lot of that stuff…you still see some weird stuff come through from time to time.”
It won’t work
And what about Andy Morley’s reservations around MMMs for Uber? Morley explains it this way: “So we've got things like weather patterns that play a really big shift in the week to week of our different products. We are constantly changing the product mix of actually how the experience is like in the app. We're launching lots of new products and each of those will get surfaced to customers in different ways. We'll have different spikes in demand versus supply, which cause reliability challenges at times or actually over supply at different times. There's an algorithm that's constantly trying to balance all of those factors and then there’s the relationship between us and competitive factors that changes pretty extensively - in tech it’s much more dynamic in terms of the competitive landscape then I've seen play out in the slower moving FMCG categories.”
Market mix v media mix models
Innis says Morley’s reservations point to another confusing element of MMM models in market – are they “media mix” or “market mix” models.
“Most market mix models that are credible would be modelling things like weather and all these kind of other variables that impact the business that Andy is talking about,” Innis says. “That’s typically why it's called market mix but I do think there's been a lot of confusion where a lot of vendors have sold media mix into the market and that's become a bit of the norm. There's a huge volume of people being sold market mix but what they're really being given is media mix. What marketers have to realise is that market mix models your business and directs you to what the commercial drivers of your business are.”
Media mix was a way of providing an ROI number for every channel but not a “reliable source of truth” for budget forecasting, Innis says. “The flow on effect is complex businesses, where demand is affected primarily by things other than media, typically aren't modelled well. I'd say a lot of the MMMs that I see when we win them off somebody would massively overstate the role of media and probably not include anything else like, for example, product changes or pricing.”
Still, Innis admits some companies may also struggle with the analysis of market mix modeling.
Too many variables
“There are certain businesses where you just have far too many variables that exist and and those variables can make it hard to disentangle signal from the noise,” he says. “That is true. I will say cheekily that Mutinex’ head of data science, Peter Photinos used to work at Uber. He seems pretty confident that it's a very solvable challenge. Maybe Andy and and Peter need to have a chat.” Morley, however, has previously quipped that when at Uber, Photinos was not an supporter of MMMs.
“I think he [Photinos] saw what we were doing under the hood and was convinced of the robustness of what we've been building,” Innis says. “ Look, MMM is a casual term for growth modelling and growth modelling can be how MMMs are done. This is the point of the open evaluation framework – it can be very different vendor to vendor so part of what we're trying to achieve with this framework is the ability to actually assess how viable these MMMs are given a certain business state. My fear is that market mix modeling becomes a wild west playground of mess with differing standards across the industry. We believe as the market leader, we should be investing resources in this because we want to see a better industry. We don't want to see a degradation of MMM standards to where it becomes a tool that isn't trusted and isn't used, particularly where you may have players provisioning these services that have conflicts of interest. We decided we had no choice but, but to release this as an open source standard.”