Useful? Or Just Interesting?

Thoughtful business leaders are hungry for new insights and – for better or worse – often find that there is no shortage of data. Dashboards, KPIs, quarterly program reviews and various marketing platforms all provide ample metrics, segments and discoveries. However, not all of this information is necessarily meaningful, or useful.

How do we know when data is useful…or merely interesting?

“Useful” insights can inform decision making. For example, a clear skew in your customer-base toward men over the age of 40 could guide important (and expensive) marketing decisions from creative development to media placement. This is useful, actionable information.

By contrast, “merely interesting” information – such as a coincidental, statistically insignificant spike in women buyers on one random Wednesday in March – has only entertainment value. Though we might try to understand the anomaly, going too far down the path in terms of analytics will only leave us wasting scarce analytical resources and executive bandwidth.

An insight might be interesting but not useful, if:

  • It’s too early to interpret the data because the results are incomplete and the statistical significance – or signal to noise ratio – is too low.
    • A marketing campaign shows five sales and an ROI above target on day one. You do not yet have enough information to draw conclusions or to scale the campaign.
  • It’s too early to interpret the results because we haven’t yet developed the proper context in which to frame them.
    • You launch a new product and notice an early trend toward female buyers in their 30s, but you don’t yet have a proper grasp of the product’s exposure to other audiences.
  • The scale of the data does not match the scale of the business issue being examined – too macro, too micro.
    • You build a model to predict tomorrow’s stock price. You wouldn’t use that same model to predict the stock price one year from now.
  • The analytical method was over-specified by the client, but a different analysis is actually required to address the question.
    • The client wants you to develop a predictive model for the likelihood of a homeowner to purchase new windows based only on age and income, but the purchase history and age of the home would provide better insights.
  • The juice isn’t worth the squeeze – meaning, the insight is not material to the business, while other analyses show more promise.
    • You could try to pinpoint the average highest level of education of your customer base…or you could analyze what offer they responded to most favorably.
  • The question attempts to find insights outside the range of available data. Extrapolation outside that range is risky.
    • Your model was trained on customers who spent between $100 and $400 on their first purchase. You can run the model on a segment of customers who spent $600, but not with guaranteed reliability.

Interesting without action is distraction.

With so much data available, particularly so much useful data, anything that consumes mindshare without implying actionable results is harmful. In fact, sliding down the rabbit hole of incomplete data not only distracts a team from progress but can actually obscure truly useful insights – sometimes unproductive insights can be so interesting that you cling to them unnecessarily, forming unhelpful biases.

Occasionally, however, following up on what seems (to the analyst) to be merely interesting is the right thing to do. This is often early in the exploration phase of a project. Some examples of situations in which “interesting” might actually be worth pursuing include:

  • When the insight hasn’t yet found its way to the right set of eyes, capable of seeing the utility because of broader knowledge, experience or context.
  • When the analyst hasn’t yet developed an intuitive understanding of the business.

In the early phases of data analysis, the team often walks different prototype insights around the business to see what clicks. This also allows us to gather data on how specific leaders process information, for later use in dashboard design.

A mature analyst is, above all, flexible. You need to know when to draw a hard line in terms of not allowing inappropriate interpretation. But you also need to know when to bend the (statistical) rules when the business needs timely directional insight and exploration.

Analysis is only good if it creates value.

As analysts, we create value through the insights we generate. To capture that value, a decision maker must receive, understand and use the insights. Thus, an analyst’s most important responsibility is to maintain the clearest communication pathway. One way to get this right is to keep “just interesting” clutter to a minimum, avoiding dilution, distraction and miscommunication.