This article talks about the need for human interaction with machine data to produce the best insights. In an era where more data is available than ever before, and where the human mind cannot process the volumes of data the way machines can, it is becoming increasingly important for organizations to unite data science with social science in order to get true value from their data.
One of the predictions made by IDC in the field of analytics and information management was about the need for human analysis and insight in addition to big data, machine learning, and decision automation. I have a personal example that underscores the need for a human to interact with machine produced analytics to be truly meaningful. My optometrist recently updated his equipment to state-of-the art software that allows his machines to take measurements that it then compares to other measurements and provides a diagnosis based on the computation. My optometrist said that the machine produced diagnosis is generally accurate and reliable. My question to him was then - Why did I need his time and expertise if the machine could just read out my prescription there by circumventing the need for the optometrist and shortening the time I needed to spend at an appointment? When he ran the data through his new software, I was quite unpleasantly surprised by the diagnosis that indicated that my vision in my right eye was significantly worse than was measured before. Now, I have had the same optometrist for a couple of decades and he really knows my case well. My doctor told me that the reason I couldn't just replace him with the machine was that he knew that my brain has learned to compensate for the weakness in my right eye with the result that my actual ability to see is much better than what the machine data indicated. My optometrist did say that the new software was able to reduce the amount of time it took to manually establish a baseline for new patients. The best results were achieved with machine and human interaction.
This is a simple example of why data and analytics alone can lead us down the wrong path. There is no doubt that big data is here to stay. In fact, the human mind is incapable of analyzing and processing the volumes of data that exist today and data volumes will continue to explode. But it is easy to get carried away with the idea that machine learning and/or artificial intelligence is the future of all decision making and human insight will become irrelevant.
In enterprises, big data and analytics is often considered the domain of data scientists that use sophisticated tools and techniques to analyze data and gain insights. Advances in machine learning and artificial intelligence are promising to automate this decision making and deliver actionable insights without human involvement. However, as this article and this TED video in the Harvard Business Review states, there is another kind of data called "thick" data that is generated by ethnographers, anthropologists and by observing human behavior, which is as important as machine generated data. Enterprises must become better at bringing the world of data science together with social science if they truly want to harness the power of data.
One of the best examples of the need for data ethnographers is related to my favorite toy maker – Lego. A write-up I have particularly liked on this story is by Martin Lindstrom, author of the book Small Data. In the early 2000s, Lego was bleeding money and was alarmingly close to bankruptcy. Big data was predicting that digital native kids would make the cheery little bricks obsolete and that it was nearly impossible for the company to recover from its declining sales unless it diversified and appealed to the generational need for instant gratification. As a result, in the mid-1990s and early 2000s, Lego spent enormous amounts of money in creating a conglomeration of assets – theme parks, books, retail stores and clothing lines and enlarged the brick form factor to appeal to kids growing up with short attention spans. However, the company continued to decline faster than ever before. Fast forward to 2015, Lego surpassed Mattel as the leading toymaker in the world. This remarkable turnaround was attributed to data ethnographers visiting a 11-year-old kid at home to observe the way he played and what he considered important. The company realized that social currency and bragging rights were extremely important to its core demographic and instead of making the blocks easier to play with, Lego re-engineered its bricks to make them smaller and began creating more complicated (and expensive) kits. Sales increased manifold as the challenge of creating these intricate Lego structures and following complicated instructions kept making kids come back for more. Based on the ethnographer's recommendations, Lego refocused its efforts on developing its core products and divested other assets like theme parks and clothing stores. In 2017, the Lego company again saw a decline in its sales, and chairman Jorgen Vig Knudstorp has again redoubled efforts on finding more opportunities to engage with kids and parents to stem the decline, having learned the lesson that machine data doesn't always have all the answers.
There are other examples that show that big data alone cannot deliver insight, perhaps the most notorious being the 2016 US presidential election results, where every poll and nearly every model predicted that Hillary Clinton would win by a landslide margin. Professor Kentaro Toyama, from the University of Michigan's School of Information points out that one of the reasons the polls could have been faulty was because "The pollsters collecting data didn't sample the right populations in the right way, and the respondents might not have been entirely truthful" proving that big data and predictive analytics does not always have all the answers and could not have accounted for cultural biases and human behavior in a large complex society.
A growing number of enterprises are recognizing this need, which has led IDC to predict in our report IDC FutureScape: Worldwide Analytics and Information Management 2018 Predictions by, that "By 2021, 25% of Large Enterprises Will Have Supplemented Internal and External Data Scientists with Data Ethnographers to Provide Contextual Interpretations of Data by Using Qualitative Research Methods That Uncover People's Emotions, Stories, and Perceptions of Their World". To get more information, or to access the entire IDC FutureScape, please visit idc.com.