24 Mar 14 Data, data, everywhere… only a drop of insight

These days it appears next to impossible to ignore big data and analytics. Big data is the next big thing, and analytics will provide answers to all our problems. When we’re stumped, big data will tell us what to do! And if big data doesn’t have the answer, then don’t worry – we can look at “small” data! And if we have no data, we can always mine it from social media or the Internet. We will make better marketing decisions, acquire more customers, improve profitability and beat the pants off competition. So what are we waiting for?

Is this real, or hype promoted by people who want to sell us expensive hardware, software or services?

The answer is in between. Yes, big data and analytics can have an amazing impact on customer acquisition and retention, growth and profitability. But it can’t answer every question – and there is a real danger of companies spending tons of money, only to end up with white elephants.

Organizations like banks or insurance companies or e-tailers or retailers have access to enormous amounts of customer information, or can derive brilliant insights from the mass of consumer data available across social and non-social media. But does this apply to every company?

Generally, big data solutions make sense when volumes are huge (millions of customer records and transaction data); velocity of data addition is high (thousands or millions of transactions/data changes happening every day); and when all this data comes from multiple, non-integrated sources.

For companies in more mundane B2B businesses that deal with a small number of customers, and generate fewer transactions, big data may not even exist. Also, corporate customers usually don’t talk about their likes and dislikes on social media or the www. Data, if available, is more finite and analysis may not require sophisticated databases or analytical tools.

However, the biggest factor that leads to failure is that companies (and managers) don’t make good use of information in the first place. Even information they already have! This is usually due to the lack of a culture where data is constantly or regularly being analyzed to deliver insights, which in turn guide decision making. This culture and set of associated competencies don’t develop simply because a company spends millions on a big data or analytics solution. Unfortunately, over-emphasis on technology is drowning out the insight piece.

Two years ago, a large consumer goods manufacturer approached us to help them analyze web (social media) data for customer perceptions of their products and brands. The brief was wonderfully vague – no thought about questions the research should answer, or the nature of insights sought. It was clear that there was a directive from the top – let’s do social media analytics, but nobody had thought through what or why. During our efforts to devise a solution, we asked to see typical customer feedback data already collected by their call center. To our surprise, the data was in a huge mess – full of errors, wrong categorization, important fields were blank or incorrectly filled in. Nobody was even looking at this data, forget about analysis! Valuable (existing) data, provided by actual customers (who had taken the trouble to call in) was simply being ignored. At the same time, they wanted to spend a lot of money on collecting fresh data, from people chattering on the web.

The same thing has happened with business intelligence software. Touted as a revolutionary tool for decision making, many companies bought software that could pull data from ERPs or CRMs and provide magical answers. Unfortunately, most companies don’t know how to use the software, or even what data is available. We were asked to do an analysis of likely target segments for an IT products company – and this was to be based on an extensive (and fairly expensive) customer survey. While designing the study, we asked for existing CRM data – and discovered a treasure trove of information. There was data for more than a thousand past interactions with prospects and clients, which hadn’t been cleaned or even looked at. Eventually, we were able to provide actionable insights without spending a lot of money on acquiring new data.

In the final analysis, it’s always good to go back to first principles. Start with business needs or objectives, and what insights or information will help decision makers achieve these objectives. Then drill down to what answers are needed, and therefore what questions need answering (key intelligence questions or KIQs). It’s only after this that one can decide on tools, methodology and data availability.

Data analytics can surely provide profound insights and answers, but only if you know what questions to ask – and how to ask them!


This article was first published in Outsource magazine

Arun Jethmalani

Arun is one of the founders of ValueNotes. Apart from trying to build a high-quality research business, he has spent the last 27 years researching, analyzing, and dissecting companies and industries. He has worked with clients of all shapes and sizes, from all parts of the world – in providing them insights that make a difference to their business.
Prior to ValueNotes, he was an equity analyst/advisor, and wrote extensively on investing – including a column titled “Value for Money” which ran for 10 years in the Sunday edition of the Economic Times. To this day, he remains an avid “value” investor.
He has also been published in several other publications, and is a regular speaker at events related to technology, investing, competitive intelligence, business process management, Internet, etc. See: Valuenotes Events
He has been instrumental in developing a community of research and intelligence professionals in India, and is the founder and current chairman of the SCIP (India) Chapter. Arun holds a B Tech from IIT, Bombay and an MS from Duke University, NC, USA. LinkedIn Profile

  • ramadas patil
    Posted at 17:29h, 21 June Reply

    well thought article on big data

  • Arush Dwivedi
    Posted at 08:46h, 13 May Reply

    If we talk about the usability of big data then I guess there are multiple domains where it is getting implemented. The preposition has started giving out the desired results. If we look at the industries like Utilities, O&G and Aviation, systems are already giving out good insights.

    Social media was just a initial thought. Big data is making a deep dive in the operational management. Telecoms are improving their networks, Utilities are deploying smart meters.

    This technology was initially considered for the “Revenue/sales” side of the business not it is being utilized in the “COST” side of business too.

  • haril
    Posted at 20:58h, 12 May Reply

    Hi Arun,
    Nice to see your blog in the Outsource magazine. You should publish few more. After so much of information overload on analytics, your blog has been refreshing and clear. Globally there is a huge shortage of skilled data scientists, due to which even the best of big data and analytics software and solutions are not able to determine the best results. As you have rightly highlighted, analytical thought-process is a culture and cannot be bought in so easily. According to McKinsey, The United States alone faces a shortage of 140,000 to 190,000 people with analytical expertise and 1.5 million managers and analysts with the skills to understand and make decisions based on the analysis of big data. However, good news is that many big data and analytics vendors have realized this and are working towards learning and development. Example is MuSigma, an analytical firm which started offering its own data scientist course and certificate to promote analtytics as a career and make sure its ever increasing demand of data scientists is adequately met

Post A Comment


Show Buttons
Hide Buttons