MMM unmuddled: Mutinex lands industry-wide backing for open-sourced model comparison framework; Google, Analytic Partners, Prophet, agency groups, pitch advisors in – with caveats
The resurgence in take-up of market mix modelling by companies seeking CFO-ready analysis and recommendations on where they should direct their marketing and media budgets for optimal business results will implode if the selection process continues on its current trend – i.e. relying on “pitch decks and promises rather than a rigorous evaluation process”. That’s the underlying argument in a white paper from Mutinex which today announced a vendor-neutral, open-source comparison and testing framework for MMM buyers which has garnered surprising support from rivals, including self-proclaimed sceptics such as Uber’s APAC marketing boss Andy Morley, who is yet to find an MMM model that works for his business but backs the open-sourced transparency effort.
Booming AI start-ups which are already subjecting themselves to open-source models for testing accuracy – or hallucination rates – and more to ensure user trust in their outputs are the precedent for resurgent corporate interest and a proliferation of models and players in market mix modelling (MMM), reckons Mutinex co-founder and CEO Henry Innis.
Mutinex claimed today in an announcement that it has developed and launched a vendor-neutral MMM evaluation framework for buyers and existing users to test any vendor or model using the same data inputs and compare the analysis and recommendations – i.e. outputs.
A surprising number of rivals and MMM buyers in a sweep by Mi3 have backed the initiative, along with marketing academics like Melbourne Business School’s associate professor Nico Neumann who is also founder of Marketingscience.today.
Analytic Partners (AP), Google’s Meridian, newcomer Prophet and agency groups including WPP Media, Publicis-owned Atomic 212, Omnicom and independent econometrics firm Accurace have also backed the initiative, with caveats and some competitive jabs from AP and Prophet. They said Mutinex’s “Open MMM Validation Framework” was important for Google’s open-sourced Meridian, Meta’s Robyn and the agency groups that sell-in those models to clients – and are at risk of being used to skew results to outcomes that meet pre-determined outcomes.
Hate the player?
“You can’t be the parent of a player, and referee the game – agency, Google or otherwise. This ends the era of modellers hiding behind in-sample metrics,” said Prophet co-founder and CEO Jordan Taylor-Bartels, referring to MMM models that showcase their capabilities using historical business data but don’t account for future, real-world “unknowns” and variables in their forecasts.
“Some practitioners have built businesses on pulling the wool over even the savviest buyers. Open-source MAC [model acceptance criteria, or in lay parlance, accuracy for specific companies and categories] helps shift the focus to what matters. We know metrics are just 15 per cent of the battle. Educating the market is the real battle ground for everyone.”
AP’s ANZ and Asia managing director Paul Sinkinson had similar warnings on the open-source MMM products from the likes of Google and agency groups essentially white labelling them.
“Google and Meta to their credit have built tools that are very easy to use and unfortunately that makes them very easy to use,” Sinkinson said. “It allows people who aren’t trained to run them. You can build a model that would potentially recommend putting money where there’s a bigger [media] rebate or puts money where somebody wants money to go rather than where it helps the client," he suggested.
"It might be that people don’t know you need to transform a certain bit of data, otherwise it can screw your model. If people are being held to tests then it makes it all a whole lot harder to be able to do that stuff. These solutions that have come out, that are DIY, the likes of [Meta's] Robyn and [Google's] Meridian, you probably need a little more scrutiny on those because it’s not trained people running it.”
Cage fight
Sinkinson told Mi3 he backed the Mutinex initiative – with some conditions. Forrester and Gartner were already reviewing and rating competing MMMs for a level of impartial analysis – although not tested for specific companies in particular categories – and that the open-sourced framework proposed by Mutinex wouldn’t account for “confounding variables”. “Depending on how you do the models, we’ve proven out the correlation between Nicholas Cage movies and suicide rates,” said Sinkinson. But that is a correlation, not a causation effect he stressed – and some MMM models will not pick up the difference. “It’s why they say 50 per cent of the value of a model comes from the modeller, 50 per cent of the value comes from the model," per Sinkinson.
Taylor-Bartels at Prophet, which has a proprietary model like AP, Kantar and Mutinex, was also explicit on the risks of bias in the open-source offers from Google and Meta, which agency groups from Publicis to WPP and Omnicom are incentivised upon when they sell in to their client base. His concern is underlying bias. “It wouldn’t be very Googley if Meridian recommends all spend goes to Meta,” he said. “There’s a statistical likelihood that there's a bunch of one percenters that do not go against the Google set, or how Google channels are used."
Still, Google’s MMM lead in ANZ, Amir Jangodaz, told Mi3 he had “heard about” but not seen the detail of Mutinex’s open source evaluation framework – and supported the intent.
“What I like about this direction – whether it's coming from Mutinex or Google or anybody – is the sense that open source is not just for developers,” he said. “It breaks down walls between black-box vendors. Our Meridian platform is all about making our own approach clearer and easier to compare," he suggested.
"It may not be perfect, but at least the industry has something to fact-check with. I'm not sure about every last detail of what Mutinex is dropping, but if it's based on openness, I'm supportive. We want data scientists, advertisers, even agencies, to have something objective to judge us all by. That's the only way anyone gets to a fair, transparent playing field.” Mi3 spoke to Jangodaz before the comments from AP and Prophet – a response on those points was not possible before publishing deadline.
Shift happens
Omnicom Media Group chief operating officer Kristiaan Kroon also backed the Mutinex initiative with a few pops of his own.
“The shift toward open-source evaluation standards is what the market’s been asking for,” he said. “While OMG hasn’t seen the detail of today’s announcement we welcome Mutinex’s decision to make their code available. This initiative can be seen in the wider context of increased transparency across not only measurement but also AI models, as best practice that builds trust and confidence and also highlights the model itself is no longer the strategic differentiator on its own for businesses.”
Kroon said while it's “clearly very important” to use best practice models and algorithms for advanced measurement solutions, “the value lies in the quality of the data inputs, that avoid bundled media products that reduce models' effectiveness, and then the outputs that drive high value implementation insights and media optimisations for advertisers. We look forward to working with Mutinex and the wider industry on best practice for measurement models and critically the inputs for them.”
Tom Braybrook, managing director ANZ for WPP Media’s data and tech unit, Choreograph, also backed the open source MMM framework and the upside that comes from MMMs more broadly for advertisers. “Especially given that more of the modelling engines are becoming open source – it gives clients more options in terms of how to deploy and design the models. So having the ability to understand what you know best practice looks like, and compare the accuracy of different models, becomes ever more important,” he told Mi3.
Marginalisation risk?
Publicis-owned Atomic 212’s chief digital officer, James Dixon, said Mutinex was the dominant MMM provider amongst his client portfolio – but there are a number of vendors in the mix, including Meridian and AP.
“What Mutinex has done represents a very brave and bold move to further the discussion and development of MMM as a non-black box solution for marketers,” per Dixon. He said many of Atomic’s clients are “not too concerned about that technical discussion about which models are better than the other at this present time, but it will come to them … so let's all open up our evaluation and make the world a better place.”
TrinityP3’s Stephen Wright said the initiative is timely. “Every MMM provider has glowing case studies – you are never going to see a bad one at the tender, pitch, sales stage,” he said. “If the [Mutinex open source] system gains widespread acceptance and use, those that don’t engage will find themselves marginalised'.
'Exactly what industry needs'
Melbourne Business School associate professor and founder of the Marketingscience.today platform, Nico Neumann, said there has been an “explosion” in MMMs over the past two years, forcing competitive pressure on vendors “as we can see on LinkedIn” where “everyone tries to play the content marketing game.
“I think the initiative is visionary and progressive. It’s also ambitious, courageous, and exactly what the industry needs. We urgently need more transparency, shared understanding, and mechanisms for accountability. And open-source approaches and discussion on governance frameworks play a critical role in laying that foundation. Of course, for something like this to be truly impactful, it will require broad support from multiple vendors and alignment on the principles behind any shared framework. That won’t be easy, but that doesn’t make it any less necessary. It’s high time we start such a conversation.”
The founder of independent econometrics firm Accurace, Clare Fernando, who has worked as an econometrician for the Reserve Bank modelling cash rates along with tenures at IPG Mediabrands and Dentsu, said some MMMs are narrow in scope – but the open-source initiative is critical for trust and credibility in the sector.
"When you engage a marketing mix modelling provider, most people don't have that fluency in mathematics or regression modelling, so it is a leap of faith for many of them,” she said. “So I am all for that transparency. But the question is what do they mean by transparency? There's a lot of providers that just focus on the media side, which is great if the KPI is performance. But if the KPI, and most of the time it is a hardcore business metric – like sales, like acquisition – you can't just look at marketing for that, as much as a marketer will say they're incredibly important, and they are... but the influence of marketing is much more increasing or enhancing that universe where sales comes from.”
Danger warning
Finally, Uber’s APAC marketing boss Andy Morley: he has tried numerous MMM models and none have worked to the degree Uber is comfortable with in forecasting.
“I'm sceptical on MMMs' abilities to work for us,” he told Mi3. "For categories that have much more stability in terms of the predictability of their growth, where you've got years and years of data and the variables are minimal, MMMs might have a lot more value. It's just for a category like ours that has so many variables that are constantly changing, any attempts that we've had of looking at MMMs or similar models, we haven't been able to trust the results – and I think the risk is that when you can't trust the results on MMMs, it can absolutely send you in the wrong direction.
"You need to have high confidence."
Morley said MMMs are still a “potentially useful tool for us” and “over time” the will hit a point of stability and have enough data and confidence to “use them in a really active way”.
“But I just think we're far away from that. I do agree that the more transparency that we can build into MMM models and to push the analytics forward with the data science community will be valuable," he added.
And then the warning.
"I can already see the marketing community almost opportunistically using them – and I think that's a really dangerous point for us to be in, because I think it has the risk of really undermining the measurement tools that we have for marketing, and therefore risks taking us backwards.”