In this year’s Gartner hype cycle analysis big data is approaching the trough of disillusionment. So is it unlikely for organizations to succeed with big data adoptions and should they focus elsewhere? No, of course not. Marketing has done its job and everybody knows of big data though few know what it means to their organization and how to approach the topic. Yet, the answer to how to succeed with big data is surprisingly straightforward once we ignore the hype and focus back on the basics.
Technology is not a strategy
A common big data challenge for new adopters is a confusion of technology and strategy. Organisations educate themselves on big data or have organically grown proof of concepts in-house and in both cases, the conversation quickly narrows down to the ubiquitous Hadoop and subsequently equivalating it with big data. Similarly, deploying Hadoop to hopefully achieve an aspect of a business goal becomes the replacement for a big data strategy. That surprisingly common story is, of course, the reverse of what the process should be and usually the first step to reduce risk and focus on value is to turn this upside down.
A top-down approach is common sense but the complexity and novelty of big data and the broad, fast-evolving technology involved lead even seasoned managers astray. Hadoop alone has an overwhelmingly rich ecosystem of supporting applications as well as competing solutions from various vendors. Beyond Hadoop, the related business intelligence, visualization, streaming, in-memory computation and the huge, diversified NoSQL space ensure that any non-expert is confused instantly.
Business goals have to drive your Big Data strategy
The solution is to start with the business goals, which usually are clear, and then discover how big data may fit into the picture as a strategy. This regularly needs education, research, and support from external experts who have experience and access to industry-specific and cross-industry big data trends. Importantly, this step should be a collaboration and contextualize big data patterns and technologies with the domain knowledge and business goals. Ideally, no solution, technology or vendor should drive the conversation at this point.
The outcome of the described exercise would be a strategic plan of employing big data pattern(s) to achieve the business goals. Now the conversation can become more specific by identifying use cases within the organization to develop proof of concepts, a big data architecture to implement them and to scale out to support the long-term strategy. After this point solutions, vendors and technologies become relevant.
Hadoop is only a part of most Big Data architectures
From experience with clients from all industries, e.g. finance, manufacturing, media, energy, E-commerce and more, we find that Hadoop often is the backbone of such an architecture although seldom the only technology required. Integrating it with other big data technologies and the existing enterprise architecture is a frequent requirement and demands a well-structured approach like the one outlined here to succeed.
This approach works equally well for small, medium and large organizations. Importantly, it gives clear guidance on the lowest level when developing proof of concepts and eventually delivering business-critical architectures and products since it is clear how they tie into a big data strategy and how it helps achieve the business goals. This is essential since the design of a big data enterprise architecture is a complex task. The technologies and patterns (e.g. Lambda Architecture) are constantly evolving and a significant amount of resources and time can be wasted. For example, choosing a suboptimal big data technology or pattern and forcing it on a use case is a subtle issue. It may only be discovered when a subsequent product has gone live and is faltering under scaling pressure or unable to accommodate new features.
A common example is choosing technology ad-hoc driven by developers’ preferences, hype or marketing. Unfortunately, the real issues often only show at scale or with increased complexity from future use cases. Proof of concepts as such can give false security without the experience and insight of how the demonstrated solution will hold up when scale-out, high availability, guaranteed response time, future functional requirements, security, integration with other systems, compliance, governance, cost, support, and all the other real-world issues become relevant.
6 Steps to Big Data success
In summary, big data success requires common sense and a structured approach. Do not get blinded by technology, hype or a ‘me too’ frenzy. The steps are simple:
- Business goals
They should be up to date and clear.
- Discover your strategy
How can big data support the business goals?
- Discover your roadmap & use cases
Plan for the long term and identify short-term use cases that are achievable.
Design an accompanying architecture and products.
- Develop your use cases
Build proof of concepts to demonstrate value, viability, and solicit buy-in.
- Deploy your architecture
Productionize your use cases with low risk employing the learning from developing the proof of concepts.
- Support and train
Focus on your business by employing services, vendors, and support where feasible, and train your staff along the above-mentioned path to bring them along with you on the big data journey.
We have employed this approach successfully with numerous clients at Big Data Partnership. Big data is here to stay and every organization should regularly evaluate it as part of its available strategies.
Note, I wrote this article for the MapR blog and published it there first.