I remember the first time I said the word “big data” with pride when describing my work. It, like every good buzzword, meant nothing to me, but conveyed a lot to my imagined prospective audience. It said something about my intelligence that I was working in “big data”, plying away at Excel sheets with way too many lines—a sure sign of a “big data” expert!
I know better now. After doing some research, I’m proud to say that I knew absolutely nothing about the topic at the time. In many ways, I still don’t—but I know enough to talk about the basics of “big data” and what it really represents, so you can explore with me.
The first step is to realize that big data represents data that is so large and complex that conventional data tools such as the table-based SQL cannot handle the load. Big data is not simply a big dataset that can be handled with Excel. Think of, for example, someone tracking every time someone commented on Ahnold’s accent on social media, their location, and other user attributes, in a mad quest to find who had the best “get to the choppa!” or “there is no bathroom!” quote variations: you’d quickly go mad trying to pass through every single one of those data points in a relational table or in an Excel file, even if you worked for a large Arnold-watching company, and had a set data process.
An easy rule of thumb to describe this is to say that big data refers to data sets that become difficult for an organization with a conventional data process to handle. This can be on several orders of magnitude. A smaller business may struggle with a lower threshold than a larger one. Nevertheless, it is the beginning of the struggle, and the search for alternatives to bread-and-butter SQL/Excel that is at the core of big data.
Traditional data tends to group data into tables, and operates with a smaller number of servers. Big data tends to ungroup data, and organize and analyze data through parallel processing across a larger number of servers.
When people in the field comment about the possibilities offered by big data, they are espousing the collection of unfathomable amounts of details we are now leaving on the web which was impossible five or ten years ago—because there were not so many details on the web, and there were no tools to collect them. Now with smartphones, sensors, and social media, data points are multiplying on an exponential level. Those who would take a dragnet over all of this data, pry them through tools not traditionally used in data collection that spread the volume and velocity of data over several servers instead of one or two, and then emerge with finely combed and actionable insights despite the overbearingly massive amount of data, are dealing with big data. This includes the NSA, but also data scientists who won the 2012 election, and health analysts working to ensure better care for all.
Please contribute to big data by commenting or forwarding me your terabytes of favorite Ahnold quotes.
It’s probably big data: new tools and terms
It’s probably not big data
Your Excel spreadsheets of political enemies, no matter how many you have
Your Excel spreadsheets of dateable people, no matter how many you have
Your SQL tables of your favorite Arnold movies, and quotes contained within
Your handwritten list of things you would do for a Klondike bar
Look at me in traditional SQL table form: