“Custom models failed us,” was the opening line from a CMO at a high-profile, disruptive consumer brand during Mi3’s MMM roundtable.

“Baseline variability from external factors like weather or competitors makes attribution unreliable.

“We've tried multiple times with MMMs and I think we'll continue to try, but we haven't landed anything that's given us any confidence so far.” 

It was one of at least a dozen core MMM themes discussed with high interest from an eclectic mix of marketing, customer and analytics leads across retail, consumer goods, QSR, auto and marketplaces.

For those that were using MMMs – sometimes central to budget setting and successfully establishing marketing’s contribution to business results and sometimes occasional projects with frustratingly long lags to get insights and recommendations – the biggest hurdle was operational deployment. 

“It’s the hardest part of the whole thing,” said one MMM project lead who after a pre-Covid deployment had seen the entire business from sales to product and finance align their planning and growth forecasts around its MMM." 

“Bottom-up advocacy must meet top-down mandates otherwise you get siloed scepticism,” said one advanced practitioner. “We bring in all the data – pricing data, distribution data, promotional data – and use it to validate everybody's assumptions.” Monthly CFO reviews showing MMM-driven cost savings also  “shared the success”. The entire business, said the MMM lead, was on track this year for double-digit improvements in ROI. 

That disclosure drew plenty of breath from others – few had experienced that sort of MMM impact. 

But outside organisational resistance, leadership buy-in and operational challenges, which were prominent issues for all, the biggest questions and pain points for MMM deployments were common in this group: 

• Isolating creative’s influence
• Vendor and model confusion, conflicts, transparency and governance
• Capturing short-term and long-term impacts
• Bias in various models 
• Time lags in findings from custom MMM models which often render the findings too outdated – up to six months
• Data infrastructure, a key barrier to efficient, accurate results
• The need for common standards to test and compare various models on their outputs or the category risks sliding into distrust and new levels of market scepticism. 

For some convinced about the merits of an MMM, new budget constraints have already iced plans for pilots.

“We had to pull back on some of the marketing expenditure ... so the things that went were things like measurement, unfortunately. We've now at least got an in-house analyst with experience in econometrics. They’re starting to work on cleaning and transforming the data so that you can start to work with an external partner. I'm really bullish on what an MMM can do for our business ... We've been on a journey where we were predominantly going through performance marketing alone, and we are investing more and more in building our brand." 

The big short

Brand building via MMMs was a rub for others who had done extensive testing of various MMM models and remain unconvinced: “My biggest concern is undervaluation of long-term effects... it might take you to a place where you're not realising full value. The models lack long-term lenses. How do you get confidence around longer-term effects?" 

Another though had got a fix on that common concern: “We track weekly sales and annual brand metrics ... it avoids harming three-year outcomes."

There was also broad frustration with the lag times for MMM findings.

“Strategic insights from bespoke models take six months to arrive. Fast, AI-driven platforms are the future. We need answers during meetings, not old reports,” said one.

Using incrementality tests to calibrate models met with broad resistance – it essentially “hacked” an MMM, artificially influencing the modelling on the tension between long and short effects and channel mix.  

“This is a really big problem – a bunch of bad MMM are being made to look good by hacking them with experimental results,” said Mutinex co-founder Henry Innis. "The rise of incrementality testing is great but we’re seeing models manipulated instead of validated. We need clear governance protocols around how incrementality tests are being applied, otherwise we risk undermining trust in MMM outputs.”

And it’s here where the frustrations were acute. Which models – generalised versus custom (generalised i.e. from a shared data pool, tended to win because of real problems with accuracy from limited datasets feeding the models in custom-built projects); conflicting and confusing vendor claims and the ability to engineer or influence MMM results were key themes as attention turned to governance and transparency in MMMs. 

Black box warning

Henry Innis and Mutinex Head of Data Science Peter Photinos acknowledged the market concerns, advocating an industry-wide open-source set of standards in which businesses could independently scrutinise model recommendations, accuracy and overall performance      

“Transparency isn’t optional anymore, it’s essential for building trust in an increasingly competitive market,” said Innis. “If we don’t get good governance protocols in place, we risk perpetuating bad practices.”

Photinos, who has worked in data science programs with the likes of Atlassian and Uber, was blunt: “It’s ridiculous for any provider in this day and age to say their model is objectively good at everything – transparency about limitations is key to building trust,” he said. “Open-source governance protocols will be critical for ensuring transparency across all models—not just proprietary ones.”

Ultimately, though Mutinex execs argue MMM commoditisation is coming, perhaps within three years. “Traditional MMM’s $30 billion [global expenditure] reflects inefficiency... commoditisation is inevitable,” Innis said.

That means competing models and vendors will have their baseline models broadly similar and competitive differentiation shifts upstream. 

"The novelty of MMMs is waning because there hasn’t been enough innovation in how models work or deliver insights,” Photinos said. “The focus will shift from building better models to integrating multiple models seamlessly into systems that provide real-time answers.

"What matters is giving CMOs answers in real-time during boardroom discussions. That’s where we change the conversation about marketing. Our goal is not just to build models but to empower marketers by providing immediate responses to complex questions like budget allocation or ROI predictions. The iPhone moment for us is when a marketer can ask a question and get an answer instantly while sitting with their CFO or CEO."

The CMOs at the roundtable are still waiting for that moment. 

* Editor's note: This roundtable discussion was part of a partner initiative with Mutinex. Mi3 editorial has poached some of the content given the intense interest on the subject from marketers and surging market interest – and confusion - in MMM adoption. More of the debate will follow in a Mutinex Market Voice series.