We are now solidly in the era of big data, where computers are capturing and processing the details of everything we do with all our interconnected devices in real time. Businesses see this as the Holy Grail for finally being able to predict who, where, and when customers will buy their existing solutions, and what their future solutions must look like to be attractive.
According to published estimates, ninety percent of the data in the world today was captured in the last two years, at roughly 2.5 quintillion bytes a day. That’s a lot of data, but the jury is still out on whether technology can make any sense of the data or derive new meaning from it in our changing world. So far, we haven’t been very good at predicting the future in life or in business.
For me, the first step in understanding the potential is to better understand what human data really looks like as it comes in from all these sources. I found some help in this regard from a classic book, “Humanizing Big Data,” by leading consumer researcher Colin Strong. I will paraphrase here the keys ways he outlines that our lives are becoming increasingly datafied:
- Datafication of emotions and sentiment. The explosion of self-reporting on social media has led us to provide very intimate details of ourselves. Many market research companies now use this data by ‘scraping’ the web to obtain detailed examples of the sentiment relating to particular issues, brands, products, and services.
- Datafication of relationships and interactions. We are now not only able to see and track the ways in which people relate, but with whom they relate, how they do it, and when. Social media has the potential to transform our understanding of relationships by datafying professional and personal connections on a global scale.
- Datafication of speech. Speech analytics is becoming more common, particularly as conversations are increasingly recorded and stored as part of interactions with call centers, as well as with each other. As speech recognition improves, the range of voice-based data and meaning that can be captured in an intelligible format grows.
- Datafication of offline and back-office activities. Within many data-intensive domains such as finance, healthcare, and e-commerce, there is a huge amount of data stored on individual behaviors and outcomes. Add to that the emergence of image analysis and facial recognition systems processing in-store footage, traffic systems, and surveillance.
- Datafication of culture. There is a whole new discipline of ‘cultural analytics,’ which uses digital image processing and visualization for the analysis of image and video collections to explore cultural trends. For example, Google’s Ngram service has already datafied over 5.2 million books from 1800 to 2000 to let anyone analyze cultural trends.
Of course, there is a big jump needed from data to real insights, intelligent decisions, and future predictions. This book author also explores some of the major challenges associated with humans making sense of big data, and using it effectively, including the following:
- The human psychology of cognitive inertia. Humans seem to be wired to resist change, with a set of cognitive ‘rules of thumb’ which focus us on short-term loss-averse behaviors. Human are inclined to rely on familiar assumptions and exhibit a reluctance to revise those assumptions, even when new evidence challenges their accuracy.
- Cognitive ability to make sense of data. Even though computers can process and store large volumes of data, assessing the implications still falls primarily in the realm of humans. Sense-making is the process of deriving meaning from experience and situational awareness, which seems to be a struggle for both people and computers.
- Information overload and data quality. In reality, more data does not necessarily lead to better decisions. More information usually means more time is required to make a decision, perhaps leading to inertia, or volumes of one type of data bias the decision in the wrong direction, since more data is not always better data.
