Even though ‘big data’ has now been around for several years, the opportunities for startups seem to keep growing, just as the amount of data keeps growing. According to IBM, companies have captured more data in the last two years than in the previous 2000 years. This data comes from sensors, social media posts, digital pictures and videos, purchase transactions, everywhere.
Every day, we create 2.5 quintillion bytes of data — much of it unstructured and far beyond the capability of conventional databases. Hence one segment of the opportunity is the need for new database technologies, like Hadoop, a distributed file system originally designed for indexing the Web. Data capacity is measured in petabytes (1000 terabytes), or soon even yottobytes (1024).
A while back, Gartner formalized their ‘big data’ definition as a “3V” framework - high Volume, high Velocity, and high Variety information asset, requiring new forms of processing to enable enhanced decision making, insight discovery and process optimization. IBM adds a fourth “V” of Veracity to add trust and noise filtering to the challenge of Big Data analysis.
By any definition, the opportunities from ‘big data’ have the potential to create a next wave of successful technology companies that could change the way we all live and work. I will summarize here some of the key business domains with large opportunities, based on reports by Data Flair, Gartner, and other sources:
-
Targeted marketing. ‘Big data’ can mean big profits. By understanding what you want to buy today, companies large and small can figure out what you'll want to buy tomorrow -- maybe even before you do. By transforming a single shopper's path into data points, companies can see how you move through a store, and how that tracks with sales.
-
Protecting the environment. Analyzing the massive sets of data available on toxic emissions and weather patterns can help us understand environmental threats on a systemic level. We now have the sensors to track and model future environmental shifts -- and how to stop them. We just need ‘big data’ tools to do the analysis.
-
Health care in the U.S. Health care is a large and important segment with huge data challenges, mostly not structured or linked. It has multiple and varied stakeholders, including the pharmaceutical and medical products industries, providers, payers, and patients. Each of these has different interests and incentives, with real money to spend.
-
Social media and web data. Social media postings and e-Commerce transactions are just a couple of the sources of external data that are of great interest to many companies. Facebook now exceeds two billion users posting, the Internet has a billion websites, and there are 600 e-Commerce items ordered every second. That’s a lot of data to analyze.
-
Automated device generated data. Another ‘big data’ opportunity is the vast amount of sensor data—machine generated data—that exists and is growing at an exponential pace as more machines become internet-enabled. Examples include data generated from traffic cameras, parking meters, toll collection, and black boxes in airplanes.
-
Scientific research in overdrive. Data has long been the cornerstone of scientific discovery, and with ‘big data’ -- and the big computing power necessary to process it -- research can move at an exponentially faster clip. The Human Genome Project, which took 13 years, could now be completed in hours. There are many more waiting.
-
Global personal location tracking. Processing personal location data is a domain that includes child safety, law enforcement, tracking terrorists, and travel planning. The data generated are growing quickly, reflecting the burgeoning adoption of smart phones and other applications. This domain is a hotbed of innovation for startup opportunities.
-
Global manufacturing. Manufacturing is a global industry with complex and widely distributed value chains and a large amount of data available. This domain therefore offers opportunities at multiple points in the value chain, from bringing products to market and research and development (R&D), RFID tracking, to after-sales services.
-
Data is the new weapon of defense. The traditional battlefield has dissolved into thin air. In the ‘big data’ era, information is the deadliest weapon and leveraging massive amounts of it is this era's arms race. But current military tech is buckling under the sheer weight of data collected from satellites, unmanned aircraft, and intercepted messages.
-
Public sector administration. The public sector is another large part of the global economy facing tremendous pressure to improve its productivity. Governments have access to large pools of digital data but have hardly begun to take advantage of the powerful ways in which they could use this information. Much change is needed here.
These large opportunities are why IDC says that worldwide revenues for ‘big data’ and business analytics will grow from $130.1 billion in 2016 to more than $203 billion in 2020, at a compound annual growth rate (CAGR) of 11.7%.
