According to Wikipedia, Big Data "consists of datasets that grow so large that they become awkward to work with using on-hand database management tools. Difficulties include capture, storage, search, sharing, analytics, and visualizing." Does Marketing data really fall into this category?
Marketing data adds up quickly if you store:
- Every click to the website
- Every page view on the website
- Every ad impression purchased/served
- Every email ever sent
- Every direct mail piece ever sent
- Every order ever taken
For one large retailer, the marketing touchpoints alone – not including the orders taken or website page views – contained billions of records in just one year. How big is a billion? If you were to stack a billion pennies in a single pile, the stack would reach nearly one thousand miles high. It’s pretty safe to say that this is Big Data.
At UpStream we’ve confronted three major Big Data challenges:
- Statisticians adapted techniques to model event data of this size
- Engineers and statisticians had to collaborate to shape data into modeling use
- Engineers had to design a scoring system that could handle dozens of models per client
We’ve repeatedly asked ourselves: Is there a business payoff for all this complexity? You can guess the answer if we’re writing about it. :)
