What do you mean by “Big Data”? How is it relevant to Digital Marketing?

Over the last year, there has been a lot of buzz regarding “Big Data” analytical tools.  There is an excellent article by McKinsey – Big data: The next frontier for innovation, competition, and productivity that summarizes the opportunity and challenge of big data. There is no clear technical definition of what big data is, but the best framework to think about big data is in terms of four V’s.

  1. Volume: The volume of data collected by digital marketers is upwards of 10’s of TB.  Any and every solution needs to scale economically to query and analyze the data in a reasonable amount of time
  2.   Velocity:  Most of digital marketing data that comes in is real-time i.e click stream and transactional data.  Ideally, the solution should make the data available for analysis in real-time.
  3. Variety: Digital Marketers are interested in getting data from various channels i.e web, call center, social and transactional data (more information in my previous blog).  Supporting various types of data becomes critical for digital marketing analysis.
  4.  Visualization: Querying large multi-dimensional data sets is always a challenge, and SQL does not make it easy for digital marketers.  The summarized results from queries can be large too.  Simple visualizations can be an elegant method to query and analyze the data.

How does Adobe’s Insight address “Big Data”

Insight is a horizontally scalable solution which processes large volumes of data in real-time.  Theoretically, at a computer science model level, it scales similarly to Hadoop.  As the volume of data increases, one can add more machines to the Insight cluster to optimize the query performance.  The solution is cost-effective, as there is no proprietary hardware involved.

Insight primarily uses visualizations to query and analyze data.  The visualizations make Insight a powerful tool to do sequential event-based analysis. Insight provides an incredibly simple mechanism to segment customers based on behavior, attributes or attributes of customer data.  The segmentation capability drives marketers to quickly indentify potential growth/profitable customer segments.

I would like to add few sentences to differentiate between a reporting and an analytical tool.  A reporting tool presents the metrics (based on the data) with limited ability to slice the data by its attributes.  An analytical engine gives the ability to determine why a particular metric is higher or lower.  An analytical engine gives digital marketers the ability to test or verify your marketing hypothesis. Using an analytical engine is about spending hours/days exploring the data to find the next nugget of knowledge that can propel your campaign or your company.

Over the years, it has been great fun to see many companies build large customer analytics solutions using Insight to address difficult marketing problems. The beauty of the solution lies in its simple concepts from computer science, and yet it is a powerful solution to solve difficult analytical problems.

In my next blog, I am going to give a specific example of how a large financial services company has used Insight to increase cross-sell.