Is Big(ger) Data Really What You Need for Better Results?
The past couple of weeks, I have been thinking about “big data,” and its growing popularity*. Prevailing opinion seems to be if your company does not have and use it, you are falling behind and will soon be out of business. The problem is, what is big data and how is it going to help your business?
Big data is a term used to describe many different things, which makes it an awkward term to work with. I suggest that a working definition of true big data is “data sets so large, and likely changing so quickly, that they are not feasible to conceptualize.” In other words, if a data set is too big and/or rapidly changing for you to wrap your mind around, it has the characteristics of big data.
Colloquially, the term “big data” has come to be a stand in for the term “analytics.” By this definition, big data refers to performing analysis, likely including visualizations, on any data set, regardless of size.
These two definitions are miles apart; this month, I am addressing the first definition. There are companies where opportunities exist to greatly benefit from a very effective big data solution. I suggest below a few questions to ask before you make the decision to commit the hardware, software, and people resources to implement this type of big data solution. These are not meant to discourage or dissuade you from implementing a solution—simply to help you think through the process so you get the most out of your investment.
- What questions do you hope to answer using big data? Projects I have been a part of have often started with only a nebulous idea of the problem to be solved or question to be answered. This is an ineffective approach to implementing a big data solution. While it is true that a big data approach means that there can be greater uncertainty or ambiguity in the questions, it is most effective to start with something. Without a question to be answered, a data scientist does not have a clue where to start, and your big data is of little more use than a phone book with no names attached to the numbers.
- How have you tried to answer your question already? A great analyst can do amazing things with even limited data sets. If your in-house team is stumped, it is often significantly more cost efficient to contract a team of outside experts to either produce the solutions or train your staff to produce them. Too often companies add complications needlessly.
- What future value will you gain from this investment? Every project I have consulted or directly worked on has started with this question. A great salesman is going to try hard to convince you that you need big data capabilities and you are missing opportunities that with it you would capture. Test this. Ask yourself, your team, anyone who will listen and give a thoughtful answer—what would we gain in the future from this?
Answering these three questions is not easy. An excellent consultant can help you through this process and help you decide which type of “big data” you need. Your answer may well lead you to a Hadoop server with big data analytics tools, and it may lead you to finding more effective analytics tools to work with what you have. Whatever your answers, I am confident the right solution will yield you significant returns.
Archetype SC has analytics experts who can help no matter what stage of the process you are in. We can provide support to your analytics team, add value to your current tools, or even be your analytics team. If you already have a big data solution, our experts will guide and assist you in getting the most out of it. Archetype SC…We Do Complicated.
*This idea was sparked by the podcast “Digital Analytics Power Hour,” specifically “Big Data—What an Executive Needs to Know.”