The Promise of Big Data: The Evil Twin

This post is a follow-up from my previous article, The Promise of Big Data: For Good.

As a database marketer, I relished in the richness of information that data gave me. Unlike traditional advertising which, to me, had no real basis for tracking or performance optimization, I took comfort in knowing that I had data to validate decisions.

The world today has emerged into data ubiquity and it is generating immense excitement among data scientists about the ever-growing strength of predictability. This is the essence of Big Data. According to Wikipedia,

“Big data is a term applied to data sets whose size is beyond the ability of commonly used software tools to capture, manage, and process the data within a tolerable elapsed time. Big data sizes are a constantly moving target currently ranging from a few dozen terabytes to many petabytes of data in a single data set.”

This wealth of information now provides an avenue that allows ever-complex data sets to be analyzed like never before. This can include where people are more likely to live in the next 20 years based on population, migration patterns, geographical economics etc. Or, it can include the likelihood of people drinking less coffee in the southern states of the U.S. in the next 10 years.

Amazing stuff! The probabilities to affect change in a positive way are absolutely boundless. Now organizations are relying more on data to drive critical organizational decisions. On the same note, there is an equal and opposite reaction to Big Data.

I have introduced the endless possibilities of a world that benefits from Big Data. However, there are personal and cultural consequences to the use of Big Data to serve the greater good. Data can lead to a loss of privacy. Face it, you are going to be asked to opt-in to sharing your data more often. Privacy as a hot topic is bringing issues of disclosure and data usage front and centre. Customers are becoming more and more aware and alarmed by how their data is being used. Consider two recent events in the news:

1. Job Seekers are being asked for Facebook Passwords

Laurie Dillon Schalk brought this article to my attention through Twitter. This was shocking to me. Imagine: you are going in for your dream job and you are asked, as part of the interview process, to divulge your Facebook information so the potential employers could peek into your profile. How would you react? This goes beyond privacy. Laurie tweeted to me,

Completely unacceptable to ask candidates 4 FB passwords. Violation of the worse kind. I don’t know how & if widespread.

This has gone beyond a standard security check. The wealth of data on every individual now gives employers more information at their disposal. Think of it: the more information you create, the more fodder you are feeding to this data well. The long term implications: you as a user will be less transparent in your posts–more guarded about what you share. Will big data encourage the antithesis of social media?

2. How Companies Learn Your Secrets

This is the story of how “Target was able to use their predictive analytics to determine a customer was pregnant. They had mailed her some promotional material, much to the surprise of her father, who made the discovery. According to this article: “Most of the people I spoke with here agreed that Target made a mistake in that case, but they believed the error wasn’t in collecting the data and then using it for marketing so much as doing so without permission.”

It’s one thing to use data to analyze and predict competitive pricing like Walmart’s Rollback Price model, however, organizations are just starting to see the “massive amounts of data to predict everything from what their customers are going to start buying to which of their employees will complete a certain project on time.” Data is also increasingly easy to collect and store, ripe for analysts to sink their teeth into.

The Government has a lot to gain from the information we feed them.

And maybe the Enemy of the State is humanity’s eventuality. We are already seeing instances of it today right here in Canada:

Canada’s “tabled” web surveillance legislation seeks to “monitor and preserve the Internet surfing activities of internet customers” in an attempt to prevent electronic criminal communications. This will further suppress or elicit more guarded online and mobile usage.

Michael Geist, Law Professor at University of Ottawa, and esteemed critic in Internet law states, “One thing (the government) has never provided is the evidence to show how the current set of laws has stymied investigations or created a significant barrier to ensure that we’re safe in Canada.”

It’s clear that we may not be able to stop he growth of big data and its evolution. That would mean we would have to change our practices: limit our usage of internet, mobile, and social networks. By virtue of using these technologies we are putting bits of our information into servers about what we’re doing, where and when. The power of this information presents amazing possibilities. But along the way, as consumers, we must become cognizant of its impact to our personal information, more importantly information that we prefer to keep private.

I’d be interested to hear what you think. What’s your impression of Big Data? The Big Brother of tomorrow or the purveyor of a better society?