'Marketers are buying this’: Pitfalls and ‘lies’ to avoid on junk user data, clean room matching, MMM, incrementality tests and B2B tech – Melbourne Business School Associate Professor Nico Neumann

Model professional: Keep questioning the data, the models and whether the fundamentals are sound, per Melbourne Business School Associate Professor, Nico Neumann.
Melbourne Business School Associate Professor Nico Neumann's original paper showed that targeting men or women via data brokers was less accurate than spray and pray, yet cost more. Last year he ran tests with IT giant HP that sharply contests most B2B marketing plans and particularly so for tech sector practitioners. “No matter what we used, it was either equal to random targeting, or even worse,” per Neumann, at least for probabilistic data. While first party data is better, there are caveats – and Neumann urges marketers to test out clean room matching systems by deliberately tainting some the hashed emails they feed into them. Next he's planning to run the rule over the main market mix modelling platforms in market. In the meantime, he suggests marketers keep questioning what those models are telling them – particularly if everything seems to be working. If that's the case, "you're probably being lied to."
What you need to know:
- Nico Neumann is deep in the weeds on digital marketing attribution, market mix modelling (MMM) and incrementality testing, likewise the dangers of narrow audience targeting and junk user data – the latter a $20bn market in the US alone.
- The Melbourne Business School Associate Professor in 2019 published research proving that closing your eyes and randomly selecting male or female audience targets was more accurate than the data brokers and DSPs many advertisers buy from.
- Neumann claims a senior data broker admitted to him privately that they knew their data was crap. Their response: “Who cares? Marketers are buying this”.
- (Like Arielle Garcia, UM’s former US privacy lead who last year told Mi3 she had accessed her data profile from multiple third party brokers with laughable results, Neumann has downloaded his own, “and it’s hilarious”. Mi3 readers can create their own ‘personalised’ laughs by testing their profile segments here.)
- Neumann batted away claims his B2C audience studies were too broad and challenged widely held assumptions that niche segments and B2B were where precision targeting of online users actually works.
- He then ran tests with IT giant HP that fed into a subsequent paper that sharply contests most B2B marketing plans and particularly so for tech sector practitioners. “No matter what we used, it was either equal to random targeting, or even worse,” says Neumann.
- First party data is better, but there are caveats, particularly around clean rooms and matches that can be bogus. Neumann advises marketers to upload made up email addresses and see what they get back – hashed user 'match rates’ may not be what they seem.
- His advice: stick with the first and second party data you can trust, but even then, don’t assume targeting will deliver better bang for buck.
- “I would even take a step back and ask, do you need to target that narrowly? There are very few cases where it makes sense … Why do you even need to exclude people and increase the cost, instead of just letting the content or message do that?”
- Neumann sees the explosion of market mix modelling and measurement approaches as “a good thing”. But there are market rumblings that the big platforms pushing MMMs risk skewing towards inherent model biases.
- Either way, Neumann’s working on a project to compare how all the main MMMs hitting the market actually perform.
- He urges marketers to continually question all models – and his advice for those emerging from business schools is the same as for seasoned CMOs: Hone fundamentals that will last a lifetime; don’t overspecialise in trends and fads. “Ask hard questions – and just test stuff yourself.”
- Get the full download via the podcast.
When you see results from a [marketing analytics] provider where every campaign or everything you have in your model or test is positive, then you're probably being lied to.
Random question
Neumann started to seriously question the claims being made for the accuracy of precision-targeted media back in 2016. Back then, “it seemed like a marketer’s dream,” says Neumann. “Everyone was using it a lot without asking many questions.”
So Neumann started asking the questions: “It was quite eye opening”. First he started with gender, testing “leading brokers” for the accuracy of their data to target to a male or a female. The accuracy? 42 per cent on average. I.e. worse worse than random – and more expensive.
Neumann says people have offered good money to spill the beans on the brokers involved in the trial. He won’t, only that they were “industry wide” and “not just outliers”.
“That's basically the surprising part – that it was even worse than the random part. If you pay nothing you were, on average, better off.”
He and colleagues then did similar tests for things like age and interests, and while some areas showed better results, “that was overall quite disappointing,” per Neumann.
The upshot was that arguments for sharper results, reduced wastage – and therefore greater ROI – from targeted advertising via third party data sources became less clear-cut.
“That's one of the interesting dilemmas with the digital targeting. You pay more to show your ad to fewer people, and then particularly if it's not accurate, this can really backfire,” says Neumann.
He acknowledges ROI has lately become “almost a bad word” as marketers seek more meaningful metrics, “but this calculation, this idea to think about what is my ROI if I do or don’t [pay more for data and targeting] … that’s something I wish more marketers would do.”
No matter what we used [in the B2B test] it was either equal to random targeting, or even worse for the probabilistic ones. If you actually have name lists – deterministic data – that was better than using random data. But again, the question came up – the costs do not really justify that.
B2B conundrum
When the paper was published six years ago, there was inevitable pushback – though some surprising acknowledgement. Neumann said he shared the paper with one broker “behind the scenes” at a conference. “They told me, ‘yeah, I know – but who cares? Marketers are buying this’.
But there were other reactions.
“Of course there were questions about the methodology and the sample – and that is fair. But this is why I tell people to do this themselves,” says Neumann. “Everyone can check this. Download the segments that you and your browser are actually allocated to and then compare it. It's hilarious, I've done this many times.”
There were also criticisms that testing broad B2C segments was very different to the performance of targeted media within deeper niches – like B2B, where finding the right people and leads within complex buying structures has long been seen as critical. (Though of late, that thinking has shifted, with Bain and the LinkedIn-backed B2B Institute among a heavyweight vanguard suggesting lead gen is actually a false proxy and that billions of dollars are being left on the table by overlooking hidden buyers within much broader buying groups.)
Neumann got “very lucky” when an exec from HP approached him to say its data could help to answer the B2B question. Neumann says the resulting study, which also compared first- with third party data, allowed him to test both probabilistic (i.e. inferred stuff from IP addresses, browsers and device type) and deterministic data (name, address, email address, phone number etc., which are available to buy as lists within B2B markets) within that context.
“No matter what we used, it was either equal to random targeting, or even worse for the probabilistic ones. If you actually have name lists – deterministic data – that was better than using random data. But again, the question came up – the costs do not really justify that.”
“We looked only at demographic data for third party providers – and what we found was that actually it’s not the data providers’ secret technology or amazing machine learning that differentiated them for our data. It was just people’s characteristics that made them easier or more difficult to actually profile. So some people are more online, have more digital devices … Other people barely have profiles, they're just not online, you can't get data about them … So … it is more about the person [and their online habits] than the providers’ capabilities.”
Neumann thinks the upshot is some urgency around a requirement for greater data transparency, i.e. “how it was collected, how old it is”. The IAB’s data label was intended to address the current lack of transparency. “But very few companies have adopted this,” says Neumann. “I almost think you have to force them.”
I would take a step back and ask, do you need to target that narrowly … Is it ever bad if someone knows about your brand? Do you want certain people to not know about your product? I think there are very few cases where, really, it makes sense.
First party caveats
Is first party data producing any better results? “Yes,” says Neumann. But there are caveats, particularly around clean rooms and matches that can be bogus.
He advises marketers to upload made up email addresses and see what they get back. His advice: stick with the first and second party data you can trust – “top publishers, or an airline where they can explain to you where the data is coming from” – but even then, don’t assume targeting will deliver better bang for buck.
“I would take a step back and ask, do you need to target that narrowly … Is it ever bad if someone knows about your brand?” says Neumann. “Do you want certain people to not know about your product? I think there are very few cases where, really, it makes sense.”
Does Neumann agree with former UM privacy lead Arielle Garcia that the digital ad industry is constructed upon what she refers to as “the precision illusion”?
“From what I’ve seen, largely, yes.” But he says marketers must make up their own minds based on basic economics. “It is really a starts with the idea of do you need this? And then, if yes, do the math,” says Neumann.
“I wonder often, why do you even need to exclude people and increase the cost, instead of just letting the content or message do that? Just have a message that you think appeals to a certain group – then we don't need to define the people,” he suggests.
“Use the concept, the content, the creative to target. That’s maybe a bit old school, but I do think we should go back to that more often.”
This is open source. It is not like there are hidden commands in the code … that is not what is happening. But there are different ways you can build MMMs ... So you have to ask yourself, if you were a big publisher selling digital [media], which choice would you make? You have to obviously ask, indirectly, what choices were made that could push it in one direction [or the other].
MMM, incrementality watchouts
Neumann welcomes the market’s gradual shift away from last click attribution and the resurgence of market mix modelling (MMM) and incrementality testing. He says the increase in choice with big open source MMM models now in market from the likes of Google and Meta alongside the independents is “a good thing”. But there are always caveats.
“Some people have concerns that ‘they are selling the media which they now have a tool to evaluate, is this okay?’ And that’s fair concern,” says Neumann.
Is he saying that there is a risk of bias or skew in some of those new models from the big platforms?
“This is open source, it is not like there are hidden commands in the code … that is not what is happening. But there are different ways you can build MMMs. Some may, for example, make digital media look worse – and with other choices, how you build it could make these look a little different,” suggests Neumann.
“So you have to ask yourself, if you were a big publisher selling digital, which choice would you make? You have to obviously ask, indirectly, what choices were made that could push it in one direction [or the other]?”
Likewise’s Neumann concerned that the recent trend toward incrementality testing risks being bent out of shape by people calling certain things incrementality tests – when in fact they are not.
“In marketing, we have one big problem in general – if something becomes a hot topic, people jump on the bandwagon and kind of drag the term in the wrong direction,” says Neumann.
“People do funny things in market mix modelling, because you want to make advertising campaigns or marketing campaigns obviously look as good as possible. There's a lot of use just in the political message – you want to go to your CMO [or CFO] and say, ‘this is what my model has said, if we reduce this budget, this is how many sales you will lose’.
“People don’t care if the model is right if it is a standard that many use and it is a well-known vendor. You want to obviously convince the CFO or your stakeholders, and it is tough as a marketer.
“But when it comes to pricing, if you actually change your price and you get that wrong, that has huge implications. So I always feel pricing is the champion discipline of market analytics.”
Either way, Neumann’s now working on a project to compare how all the main MMMs on the market actually perform – something that will no doubt engender robust debate when published.
In the meantime, he has one tip for marketers regardless of MMM platform or marketing effectiveness measurement system they subscribe to.
“When you see results from a provider where every campaign or everything you have in your model or test is positive, then you're probably being lied to,” he says.
“You should see several campaigns that are not working. Whenever you see that [everything seems to be working] it will be very unrealistic or unlikely to have a proper analysis.”
Neumann’s advice to marketers mulling MMM, new attribution models, or even just whether to spend extra money on apparently sharper targeting?
“Ask hard questions, often – and just test stuff yourself.