A company with a strong Objective & Analytical Culture dedicates data assessment resources to true analysis as opposed to reporting activities that do not drive strategic or operational impact.
We live in an age where there are a lot of information capturing mechanisms and the world’s largest companies trade in information. There is also a proliferation of tools that can create charts and tables for everyone with very little effort. So, it is tempting for everyone to jump on the data bandwagon. Although data related concepts aren’t rocket science and can be learned, it does require discipline to learn and practice to get good at it. Often organizations under-invest in using the precious information they have optimally.
Broadly speaking two things can be done with data – 1) analyze the data, or 2) report the data. They are very different concepts and have different purposes. Not internalizing the difference between analysis and reporting is a huge drain on senior executives and other managers at small- and mid-sized companies, not to mention lost opportunities to use Frontline Resources more efficiently and direct company strategy and operations more optimally.
Analysis vs. Reporting
Analysis and Reporting are casually used in many contexts. But they have specific purposes and expectations as summarized in the Internal Data Ecosystem below. Process-Data Symbiosis creates internal data that can be leveraged for either analysis or reporting.
What is analysis?
Analysis starts with problems with unknown root causes, works with unbiased hypotheses, assesses the hypotheses after understanding available data, identifies root causes, and recommends tangible actions to improve a company’s strategy and operations.
What is reporting?
Reporting is a frequent, cadence-based manual or automated publishing of performance comparisons to predefined thresholds (developed during operational planning and incorporated into process design) with the goal of escalating performance gaps to improve process adherence or highlight need for analysis.
The visual below compares Analysis to Reporting. They both have an important role to play; but misusing their applications will lead to significant organizational issues.
An “Insightful” company only looks at reports when thresholds are breached in processes and red flags are raised. A practical example of a well-functioning report is a car’s oil change light. The car’s electronics is constantly “reporting” in the background comparing the oil levels to a predefined threshold set by the manufacturer. A driver never spends a moment thinking about it until the light comes on – this is the flag that asks for intervention. The manufacturer has designed the car to report on oil performance.
Staying with the car analogy, what if the car starts making an odd noise that the driver knows isn’t a good sign; but can’t see any obvious signs of problems. Most owners would take the car to an expert mechanic who does a hypothesis-based assessment of what could be causing the noise based on their experience. A good mechanic eliminates possibilities and provides a precise recommendation about what is wrong and how to can fix it. This is analysis. Without recommendations, there is no analysis.
In addition to the definitions and comparisons above, three points are worth reiterating about practical usage of analysis and reporting:
1: Reports should always have thresholds
The most important part of reporting is the threshold that clearly draws a line between good enough and not good enough – these are leading indicator management limits that every process should have. Reporting without reference to these thresholds is a waste of organizational time. Setting leading indicator targets and demonstrating the discipline to execute accordingly is where rubber meets the road for a small- to mid-sized company.
For example, even quarterly or annual financial reporting that shows revenue and expenses just compares actual figures to expected figures and states whether goals are met. If the goal is not met, we can say that the underlying process (in this case, the entire company’s operations) is not delivering.
2: Reporting does not require attention, barring escalations
The most valuable part of reporting is to raise awareness broadly that performance is not meeting expectations when thresholds are breached. There is absolutely no reason to stare at a report for any length of time beyond a quick comparison between performance and threshold. In the example above about financial results, spending more time looking at revenue or expense numbers tells the audience no new information. A good process design preemptively states follow-on actions for escalations caused by performance gaps.
3: Analyses require decision-maker mindshare
Analyses directly provide answers to previously unanswered questions and what actions should be taken next. Decision-makers have to dedicate time and mindshare to consume analytical content and ensure that related decisions are made to improve company’s strategic and operational direction.
Analysis & Reporting Pitfalls
Analysis and reporting are powerful tools, if leveraged well. However, misuse creates company-wide risks.
Hawthorne effect: A major reporting risk
Hawthorne effect is a simple idea that living things behave differently when observed. In a corporate setting, observation is measurement of behaviors. When a specific portion of a process is actively observed with the performers’ knowledge, they are likely to change their behavior to fit the observation; but it doesn’t mean the overall outcome will be better.
The greatest risk of Hawthorne Effect comes from highly intense reporting on very specific metrics for extended periods of time. A real-life example of this is Wells Fargo’s account scandal in 2016. The company designed a great metric that it believed would drive growth – number of accounts per customer. There is nothing wrong with the metric or to cross-sell to customers. However, this same metric was used for years to report company’s health in the market and assess employees’ performance internally. Eventually, Wells Fargo employees felt compelled to continue to improve this metric upwards; they figured out short cuts to do so because they were being measured so intently on it.
Such undesired change in behavior is likely to happen when reporting becomes the only method of organizational management. Impact of Hawthorne Effect is often hidden and it will not be obvious without carefully observing the behaviors around the process, including the measurement approach.
Hypotheses developed without strong business acumen
Hypothesis development is the nucleus of analysis. Hoping to get lucky often leads to misunderstanding data flaws or biases (as discussed in Understanding Data) as meaningful messages. Simplistically, hypotheses are unproven theories about underlying reasons for problems or foundational drivers for preferred outcomes. The purpose of analysis is to prove or disprove these theories, which is not easy given the many problems that inherently exist in data.
If an analyst develops poor hypotheses that does not reflect true business behaviors and somehow stumbles into data flaws or data biases, then the company is provided a strong, but incorrect recommendation. It is absolutely critical that a company ensures that senior executives and analytical staff has strong foundational business acumen and a deep understanding of customer tendencies, business strategy, and operational challenges.
Confusing reporting for analysis
As discussed above, analysis requires review and decision-making by senior executives. However, what if an organization mistakes the difference between reporting and analysis? Both analysis and reports usually involve graphs, numbers, and words. Reporting is a more foundational step of parsing available data and visualizing it, while analysis is a more complex exercise of looking for answers to previously unanswered questions.
If decision-makers mistake the difference, an organization can become significantly paralyzed in terms of making decisions based on data. Imagine expensive organizational resources looking at basic parsing of data without any strong key messages to take away; then, speculating how to act based on their anecdotal knowledge. This often leads to poor decisions and has major organizational cost.
In conclusion, an “Insightful” organization should spend very little time on reporting as key thresholds for ongoing operations should be pre-defined as part of process design. On the other hand, the company should heavily rely on internal analyses to learn about strategic problems around market and customers, and iron out kinks in its operations.