You may have seen various people use the terms “reporting” and “analysis” as though they were interchangeable terms or almost synonyms. While both of these areas of web analytics draw upon the same collected web data, reporting and analysis are very different in terms of their purpose, tasks, outputs, delivery, and value. Without a clear distinction of the differences, an organization may sell itself short in one area (typically analysis) and not achieve the full benefits of its web analytics investment. Although I’m primarily focusing on web analytics, companies can run into the same challenge with other analytics tools (e.g., ad serving, email, search, social, etc.).

Most companies have analytics solutions in place to derive greater value for their organizations. In other words, the ultimate goal for reporting and analysis is to increase sales and reduce costs (i.e., add value). Both reporting and analysis play roles in influencing and driving the actions which lead to greater value in organizations.

For the purposes of this blog post, I’m not going delve deeply into what happens before or after the reporting and analysis stages, but I do recognize that both areas are critical and challenging steps in the overall data-driven decision-making process. It’s important to remember that reporting and analysis only have the opportunity of being valuable if they are acted upon.

Purpose

Before covering the differing roles of reporting and analysis, let’s start with some high-level definitions of these two key areas of analytics.

Reporting: The process of organizing data into informational summaries in order to monitor how different areas of a business are performing.

Analysis: The process of exploring data and reports in order to extract meaningful insights, which can be used to better understand and improve business performance.

Reporting translates raw data into information. Analysis transforms data and information into insights.  Reporting helps companies to monitor their online business and be alerted to when data falls outside of expected ranges. Good reporting should raise questions about the business from its end users. The goal of analysis is to answer questions by interpreting the data at a deeper level and providing actionable recommendations. Through the process of performing analysis you may raise additional questions, but the goal is to identify answers, or at least potential answers that can be tested. In summary, reporting shows you what is happening while analysis focuses on explaining why it is happening and what you can do about it.

Tasks

Companies can sometimes confuse “analytics” with “analysis”. For example, a firm may be focused on the general area of analytics (strategy, implementation, reporting, etc.) but not necessarily on the specific aspect of analysis. It’s almost like some organizations run out of gas after the initial set-up-related activities and don’t make it to the analysis stage. In addition, sometimes the lines between reporting and analysis can blur – what feels like analysis is really just another flavor of reporting.

One way to distinguish whether your organization is emphasizing reporting or analysis is by identifying the primary tasks that are being performed by your analytics team. If most of the team’s time is spent on activities such as building, configuring, consolidating, organizing, formatting, and summarizing – that’s reporting. Analysis focuses on different tasks such as questioning, examining, interpreting, comparing, and confirming (I’ve left out testing as I view optimization efforts as part of the action stage). Reporting and analysis tasks can be intertwined, but your analytics team should still evaluate where it is spending the majority of its time. In most cases, I’ve seen analytics teams spending most of their time on reporting tasks.

Outputs

When you look at reporting and analysis deliverables, on the surface they may look similar – lots of charts, graphs, trend lines, tables, stats, etc. However, there are some subtle differences. One of the main differences between reporting and analysis is the overall approach. Reporting follows a push approach, where reports are pushed to users who are then expected to extract meaningful insights and take appropriate actions for themselves (i.e., self-serve). I’ve identified three main types of reporting: canned reports, dashboards, and alerts.

  1. Canned reports: These are the out-of-the-box and custom reports that you can access within the analytics tool or which can also be delivered on a recurring basis to a group of end users. Canned reports are fairly static with fixed metrics and dimensions. In general, some canned reports are more valuable than others, and a report’s value may depend on how relevant it is to an individual’s role (e.g., SEO specialist vs. web producer).
  2. Dashboards: These custom-made reports combine different KPIs and reports to provide a comprehensive, high-level view of business performance for specific audiences. Dashboards may include data from various data sources and are also usually fairly static.
  3. Alerts: These conditional reports are triggered when data falls outside of expected ranges or some other pre-defined criteria is met. Once people are notified of what happened, they can take appropriate action as necessary.

In contrast, analysis follows a pull approach, where particular data is pulled by an analyst in order to answer specific business questions. A basic, informal analysis can occur whenever someone simply performs some kind of mental assessment of a report and makes a decision to act or not act based on the data. In the case of analysis with actual deliverables, there are two main types: ad hoc responses and analysis presentations.

  1. Ad hoc responses: Analysts receive requests to answer a variety of business questions, which may be spurred by questions raised by the reporting. Typically, these urgent requests are time sensitive and demand a quick turnaround. The analytics team may have to juggle multiple requests at the same time. As a result, the analyses cannot go as deep or wide as the analysts may like, and the deliverable is a short and concise report, which may or may not include any specific recommendations.
  2. Analysis presentations: Some business questions are more complex in nature and require more time to perform a comprehensive, deep-dive analysis. These analysis projects result in a more formal deliverable, which includes two key sections: key findings and recommendations. The key findings section highlights the most meaningful and actionable insights gleaned from the analyses performed. The recommendations section provides guidance on what actions to take based on the analysis findings.

When you compare the two sets of reporting and analysis deliverables, the different purposes (information vs. insights) reveal the true colors of the outputs. Reporting pushes information to the organization, and analysis pulls insights from the reports and data. There may be other hybrid outputs such as annotated dashboards (analysis sprinkles on a reporting donut), which may appear to span the two areas. You should be able to determine whether a deliverable is primarily focused on reporting or analysis by its purpose (information/insights) and approach (push/pull).

Another key difference between reporting and analysis is context. Reporting provides no or limited context about what’s happening in the data. In some cases, the end users already possess the necessary context to understand and interpret the data correctly. However, in other situations, the audience may not have the required background knowledge. Context is critical to good analysis. In order to tell a meaningful story with the data to drive specific actions, context becomes an essential component of the storyline.

Although they both leverage various forms of data visualization in their deliverables, analysis is different from reporting because it emphasizes data points that are significant, unique, or special – and explain why they are important to the business. Reporting may sometimes automatically highlight key changes in the data, but it’s not going explain why these changes are (or aren’t) important. Reporting isn’t going to answer the “so what?” question on its own.

If you’ve ever had the pleasure of being a new parent, I would compare canned reporting, dashboards, and alerts to a six-month-old infant. It cries – often loudly – when something is wrong, but it can’t tell you what is exactly wrong. The parent has to scramble to figure out what’s going on (hungry, dirty diaper, no pacifier, teething, tired, ear infection, new Baby Einstein DVD, etc.). Continuing the parenting metaphor, reporting is also not going to tell you how to stop the crying.

The recommendations component is a key differentiator between analysis and reporting as it provides specific guidance on what actions to take based on the key insights found in the data. Even analysis outputs such as ad hoc responses may not drive action if they fail to include recommendations. Once a recommendation has been made, follow-up is another potent outcome of analysis because recommendations demand decisions to be made (go/no go/explore further). Decisions precede action. Action precedes value.

Delivery

As mentioned, reporting is more of a push model, where people can access reports through an analytics tool, Excel spreadsheet, widget, or have them scheduled for delivery into their mailbox, mobile device, FTP site, etc. Because of the demands of having to provide periodic reports (daily, weekly, monthly, etc.) to multiple individuals and groups, automation becomes a key focus in building and delivering reports. In other words, once the report is built, how can it be automated for regular delivery? Most of the analysts who I’ve talked to don’t like manually building and refreshing reports on a regular basis. It’s a job for robots or computers, not human beings who are still paying off their student loans for 4-6 years of higher education.

On the other hand, analysis is all about human beings using their superior reasoning and analytical skills to extract key insights from the data and form actionable recommendations for their organizations. Although analysis can be “submitted” to decision makers, it is more effectively presented person-to-person. In their book “Competing on Analytics”, Thomas Davenport and Jeanne Harris emphasize the importance of trust and credibility between the analyst and decision maker. Decision makers typically don’t have the time or ability to perform analyses themselves. With a “close, trusting relationship” in place, the executives will frame their needs correctly, the analysts will ask the right questions, and the executives will be more likely to take action on analysis they trust.

Value

When it comes to comparing the different roles of reporting and analysis, it’s important to understand the relationship between reporting and analysis in driving value. I like to think of the data-driven stages (data > reporting > analysis > decision > action > value) as a series of dominoes. If you remove a domino, it can be more difficult or impossible to achieve the desired value.

In the “Path to Value” diagram above, it all starts with having the right data that is complete and accurate. It doesn’t matter how advanced your reporting or analysis is if you don’t have good, reliable data. If we skip the “reporting” domino, some seasoned analysts might argue that they don’t need reports to do analysis (i.e., just give me the raw files and a database). On an individual basis that might be true for some people, but it doesn’t work at the organizational level if you’re striving to democratize your data.

Most companies have abundant reporting but may be missing the “analysis” domino. Reporting will rarely initiate action on its own as analysis is required to help bridge the gap between data and action. Having analysis doesn’t guarantee that good decisions will be made, that people will actually act on the recommendations, that the business will take the right actions, or that teams will be able to execute effectively on those right actions. However, it is a necessary step closer to action and the potential value that can be realized through successful web analytics.

Final Words

Reporting and analysis go hand-in-hand, but how much effort and resources are being spent on each area at your company? When I hear a client is struggling to find value from their web analytics investment, it usually means one of the dominoes in the “Path to Value” is missing and often analysis is that misplaced domino.

I recently met with a major media client that found it was missing its analysis domino. The web analytics team was struggling to meet the strategy, implementation, and reporting demands of this large, complex organization – let alone providing analysis beyond just ad hoc responses. Senior management was becoming increasingly frustrated with its analytics staff and system. Fortunately, the web analytics team received additional headcount budget and hired an analyst to perform deep-dive analyses for all of its main product groups and drive actionable recommendations. Not surprisingly the attitude of the senior executives did a 180-degree turn shortly after the company found its missing analysis domino.

You may be wondering how much time your analysts should spend on analysis. As a rule of thumb, I would say at least 25% of their time should be spent on analysis, and generally the more, the better. Surprisingly, 100% is not desirable either because there are many important responsibilities that are needed to keep an analytics program afloat such as reporting, gathering business requirements, training, documenting and communicating successes, etc. I hope after reading this article you at least recognize that 0% of their time is unacceptable. If your company isn’t doing much analysis today, experiment with a 10% focus on analysis and see what success you have from there. In addition, our consulting team is always willing to help with your analysis needs. Good luck!

16 comments
Cool Playbook
Cool Playbook

Great Post Ben! Really informative and I loved the comparison table specially. Thanks

Ahsan
Ahsan

Excellent post. An essential read for anyone involved with BI/DW implementation regardless of vendor or subject area.

Steve Fernandez
Steve Fernandez

@Eric - I wouldn't fully discredit using a true analyst to do reporting; at least the building of the report. There's a real art in the transformation of raw data into something readable and meaningful. I personally take a lot of satisfaction is stretching the Tufte side of my mind to help the organization understand the sea of digital data that's captured. But, I agree whole heartedly in automating the process as much as possible. Mind numbing drudgery needs to be eliminated at any opportunity.

ankur batla
ankur batla

Hi Brent, Thanks for writing this article.., It has clarify my queries .. :) Thanks again.. :)

Abhijith
Abhijith

Well put. One might be employee/employer who read the post could discover two things. where they are? and what they should do? Abhijith

Eli Mueller
Eli Mueller

@Eric T. Peterson - I agree that if possible, a company should strive for increased allocation of time in the analysis stages of the process outlined so well. It's true that automated reporting, where applicable, should be actively pursed to help relieve less constructive tasks from the limited resources that most companies have. Unfortunately in most businesses, resources (both technological and human-based) are limited to the point where the 25% analysis may be a significant improvement.

Brent Dykes
Brent Dykes

Andrew, Thanks for your feedback. Brent.

Andrew
Andrew

Fantastic article!! Very well written.

Brent Dykes
Brent Dykes

Matt, I agree wholeheartedly. As I wrote this post, I didn’t want to marginalize the role of reporting. It has its own unique role (it is a domino), but I hope that as companies discover the importance of analysis that they will be able to realize even more value from their web analytics investments. Brent.

Matt Coen
Matt Coen

Brent, Well put. These concepts are fundamental to realizing the real value of tools like SiteCatalyst. Reporting is necessary but the money is in analytics. Matt

Brent Dykes
Brent Dykes

Adam, If this post can help a few companies in some small way, I’ll be thrilled. Thanks, Brent.

Adam Greco
Adam Greco

Another great post! It is often hard to get companies to leave their analysts alone for enough time to do real analysis...Hopefully this post will help!

Brent Dykes
Brent Dykes

Web Analytics Europa, I’m glad you found the comparisons helpful. I felt they were necessary to help define the different roles of reporting and analysis. Hopefully, this article will help companies to identify where their analysts are currently focusing most of their time. Thanks, Brent.

Web Analytics Europa
Web Analytics Europa

We all knew it but with this article and the included comparisons it gives a very powerful approach to change things within the organisation to improve on your website and strengthen the competence of analytical people. Good read.

Brent Dykes
Brent Dykes

Eric, Thanks for your comments. I think we’re on the same page, but we might be differing slightly on the approach or emphasis. We both agree that analysis is important (“the more, the better” as I stated above). When I mentioned that analysts should be spending “at least 25%” of their time on analysis, I’m trying to encourage companies to get started with analysis. I think we’d both agree that the task at hand is not to get companies to close the gap from 60% to 80%. We’re trying to get organizations to close a bigger gap and go from the 0-10% mark in some cases to a higher level that will start to build tangible momentum or inertia for analysis in their companies – hence, my “at least 25%” goal. My wife started running four years ago, and she loves it. Her first race was a 5K in our neighborhood, and she eventually completed a full marathon. When you say that companies should have the goal of reaching 80% analysis mark (i.e., running a marathon), all I’m advocating is that companies start with a 5K first (>25%). Eventually, they’ll be both ready and excited to run a marathon just like my wife was. I believe when a firm has some success with analysis that it will fuel more analysis, but the key is getting started. That’s why I’m focusing on a smaller, more attainable goal in my post. Thanks again for your comments. We’re a united front for more analysis in our industry. Brent.

Eric T. Peterson
Eric T. Peterson

Brent, Great post echoing a lot of what I've been saying for over a decade. One issue: when you say analysts should spend "at least 25%" of their time on analysis (implying 75% on reporting and similar tasks) you really haven't moved the bar very much. At Web Analytics Demystified what we have long seen is that most experienced web analytics practitioners are spending 80% of their time on reporting and make-work functions and 20% (or less) of their time doing any type of real analysis. This is, of course, messed up for a variety of reasons: 1) Reporting, while valuable, is something that needs to be automated wherever and whenever possible, and reports need to be delivered through intuitive and easily learned systems. See my post on this subject from February of this year for more details. 2) Reporting very rarely translates into the type of insights that drive businesses forward, and so having your most highly trained and qualified people (analysts) spend 75% of their time (your number) producing low-value output doesn't contribute to web analytics return on investment. You captured this point well. 3) Most importantly, I don't know very many analysts worth their weight who enjoy reporting, regardless of what tool set they use. When people aren't able to "stretch their minds" and really consider what the treasure trove of data we work with can tell them about Change the Business initiatives, well, they get antsy. There is nothing worse than antsy analysts --- unless you're Corry Prohens over at IQ Workforce. J In our strategic practice we typically recommend that client build out governance and staffing models that lead to tiered teams (easier to hire) and challenge their most senior resources with spending 80% of their time doing analysis, not reporting. Yes, 80% is the target, and yes, 80% is difficult to hit in the resource constrained environment we work in, but in my experience we've already set our sights too low ... it's time to challenge ourselves, our leadership, and our community to do better. Again, great post. Eric T. Peterson Web Analytics Demystified, Inc. http://www.webanalyticsdemystified.com