Spend enough time with analysts or people in Finance, and you'll have heard these terms. To my mind, there is a special place in hades for people who deliberately misuse and misrepresent these terms. They have meaning.
And what they mean matters.
Statistical significance
When something is statistically significant, it means that we can quantify that the result is real, that it is not due to chance, and that we didn't just get 'lucky' when we picked the sample. There are plenty of far more complicated ways of saying the same thing - sometimes analysts will go on about the null hypothesis, probabilities and acceptable levels of uncertainty etc. Bottom line - they're saying that the results are real; thing A is definitely different (and better/worse) than thing B; OR, it could be that thing A and B aren't different in any meaningful way at all.
Materiality
Materiality is a Finance term. Something is material when it impacts our understanding; when we might make one decision if we have the information in hand and perhaps a different one if we do not. From an analytical point of view, this usually boils down to whether we make or save more money from option A or option B.
To be fair, the accountants usually have much bigger conversations, but when it comes to analytics, and the types of decisions analysts recommend, the size, often referred to as the materiality, of the impact is crucial.
Why does this matter?
I've worked in large organisations, small ones and middle-sized ones. In really large organisations, people are primarily in their own space; you tend not to get too much cross-pollination of terms. People in Risk (analytics) tend to talk about whether something is significant or not (usually meaning whether it is statistically significant), and no doubt, people in Finance talk about materiality. I'm not sure exactly - I didn't get to chat to too many finance people when I worked in truly large organisations.
However, in mid-sized and small organisations, where I've always found more cross-functional communication (and miscommunication) happening, people in risk and analytics most certainly do speak to the people in Finance. And at this point, without a decent understanding of what people mean, things can become interesting...
And by interesting, I mean misunderstood, and potentially expensive.
When someone says that the results of a test are [statistically] significant, most people hear that the results are real, that there is definitely a difference in the outcomes, and that we can be confident in our choice. What might be less apparent is that a statistically significant difference doesn't mean that the difference is large or material. The difference could be real AND small, economically so tiny that it doesn't make sense to implement changes to remedy the situation.
So, how do we go about figuring out what's what??
I find it helpful to ask questions; my go-to is to be passionately curious. Analysts are not usually known for being the centre of the conversation. However, if you ask them questions and show interest in what they're saying, they will (usually) happily chat about what they're doing and why.
And analysts, my #TopTip: take a feather from Einstein's cap...
The definition of genius is taking the complex and making it simple.
Albert Einstein
Don't try to bamboozle your audience with fancy-schmancy terms; you may think you're coming across as impressive. Unfortunately, there is a much better chance that you're coming across as a bit of a pratt or perhaps an arrogant and insufferable know-it-all.
As analysts, our role is to help the business reach decisions. This means that we need to be able to explain what we've done, why it is important, what it means, and what will happen if we "do nothing", compared with what will happen if we implement the recommended action(s). We also need to estimate and quantify the costs and benefits associated with the change.
Remember, just because something is statistically significant does not mean that it is material.
I'd love to hear your thoughts and stories on the subject. Have you experienced an analyst using really big words and making something sound so difficult to understand? Analysts, have you ever heard someone use terms entirely out of context? Or perhaps you've got stories where analysts took something that really was complicated and made it simple and easy to understand? Please do share them in the comments.
Photo by Rajas Chitnis on Unsplash
Comments