It’s a bit tough to explain in just a few words, but here’s some pertinent points:

- Correlation is not the right word. I’m not trying to be nit picky, but correlation only holds for a linear relationship. What he was trying to do is fit a model.
- It’s pretty easy to show that advertising works, the real question is how it works, which is why you try to fit a model to varied data points.
- Usually the model doesn’t explain much more than 50% of the data, so it’s hard to pin down exactly what the impact is. This type of modeling is much more suitable for optimization.

In general, it seems like a pretty crappy study. This type of modeling only makes sense if it’s ongoing. Doing it short term almost guarantees to confuse as much as explain.

- Greg

I’d be interested to also get your thoughts on the following example. A client of our hired a statistician to work on finding if there was correlation between media spending and retail sell-through.

- The modeling was quite tough. We’d used a relatively flat, consistent media schedule to build momentum – planning media by response tends to lead away from front-loaded schedules in my experience. But the noise at retail from appearance in circulars and other market forces made it difficult to detect any on-going repeatable media impact.
- This required some transformations (don’t recall the specifics). And, after the transformation work to damp out the noise, there was a clear correlation between media and retail sell-through.
- It showed a roughly 2 week delay between spending and the start of impact.
- And, it showed a 3 week sweet spot where the majority (60%?) of the sales impact was seen.

Fine to this point. Then, the company asked him to “quantify” the impact.

- Being a good statistician, he ran the equations and came up with estimates of the impact size being quite small.
- Market research interviews of purchaser influence showed that a media impact of 15 to 20 times the impact.
- My sense is that the statistics were conservative – especially after the transformation machinations required to damp out other market forces. They also had no ability to detect the very long term impact of a media flight because the numbers were small in any given week (but significant when looked at over a 6 month period).
- On the other hand, literally accepting the research interviews was likely a very aggressive estimate due to an entire variety of challenges from consumer recall, vagaries of responses, etc.

My working hypotheses have been:

- It is easier for statistics to determine whether or not media has an impact. It is much harder for statistics (or any other method) to accurately calculate the specific size of that impact.
- Statistics and consumer interviews probably bracket the true impact of a campaign.

Would love any thoughts…

…Doug

Thanks for your comment.

I really don’t see much difference between online and offline. The main point is that you need to know how your data are collected and what the assumptions of the model you’re using are. Otherwise, you’ll run into problems.

One point about Digital media is that they don’t know enough about how traditional media metrics are used, so there are a lot of misconceptions about what needs to be done. I’m publishing an article about that tomorrow.

- Greg

I personally would be very interested if you can share in a future post interesting examples how numbers lie in online marketing.

Stan

Thanks. It’s a very good point you make in your article, that not only are numbers often wrong, but wrong for the worst of reason – cowardice.

I encourage everybody to read it http://media-proinc.com/ribblog/analytics-are-for-cowards/

- Greg

Hi Greg,

Yesterday, I was getting ready to write an article titled “Analytics are for cowards” when I read your article “Less Numbers – More Math.” It augmented my thoughts so I quoted you and linked to you. I hope your readers and you think it adds value to the discussion.

Thanks for the excellent information and the unwitting help.

John Ribbler

http://media-proinc.com/ribblog/blog/

- Гриша

a great post! You make it sound so obvious and so practical and I agree, math is about understanding rather than complicated science especially in business environment. The numbers need to be simple but understanding should be very deep.

Kind regards,

Katya

Thanks. Good point about benchmarking.

- Greg

A great article! I often work with entrepreneurs and startups, and I must remind people about GIGO (Garbage In – Garbage Out). You can have wonderful pro forma financial models that show your compound growth rate and rising net income, but if the numbers that you base your model on are not valid (and often they are not) it won’t make a difference.

I recommend two ways to avoid such problems: first, do proper research into the financials of your competition as a comparison, and then be a bit modest about your own selling ability!

Kevin