I have the honor of representing EMC in MIT's BigData@CSAIL initiative. At a recent annual meeting, I had the opportunity to speak about the challenges enterprises are having in adopting Big Data. The talk resonated with the other industry representatives in the room so I thought it worthwhile to share these observations in this forum.
During the past decade, a handful of companies have pioneered the Big Data movement. They shared the common challenge of extracting value from unprecedented amounts and kinds of data. To achieve this, they developed new software, fueled by Moore’s law advancements, capable of processing data with substantial economies of scale. This allowed the Big Data pioneers to discover and profitably monetize previously inaccessible predictive relationships - essentially the Long Tail concept applied to analytics.
The pioneers’ extraordinary success has inspired many companies to explore using Big Data to improve or grow their business. The plethora of open-source Big Data software appears to make this an easy task. However, three fundamental challenges are making crossing the Big Data chasm difficult.
Reading the popular press, Big Data appears to be magic – just put your data in a hat, hire magicians called “Data Scientists”, and profitable insights materialize in a puff of smoke. This, however, is very far from reality. Finding obscure profitable predictive relationships may require a substantial amount of exploratory analytics. The time required to find these insights can be highly variable. So too their value. This uncertainty makes it exceedingly difficult to estimate the return on investment of implementing a Big Data platform. The traditional business school approaches to evaluating opportunities and managing investments don’t apply. Business leaders, therefore, must take a Big Data “leap of faith” which is the first of the three challenges.
The brave leaders that make the leap often land in an unfamiliar sea of technology options – Hadoop, Spark, Storm, Kafka, Solr, HBase, Redis, Cassandra, MongoDB, etc. Disruptive “born digital” companies talk of new architectural approaches like the Lambda Architecture while incumbent data warehouse vendors promote more traditional looking hybrid architectures. No two Big Data environments look alike and are difficult to compare. Put simply, the newly Big Data converted have a lot of choices but with no way to choose. This can make implementing Big Data an arduous task of evaluating various technologies and assembling them into a coherent analytics environment. Many IT organizations may not be prepared for such an involved procurement, evaluation, and integration project making this “some assembly required” aspect of Big Data the second of the three challenges.
Those that manage to build a Big Data platform need data to put on it. In established companies, that data is often in silos within the different departments and business units. Compelling the data’s migration out of the silos requires convincing their owners that the Big Data platform will provide unequaled unique value, particularly if everyone participates. To a silo owner, however, migrating all data to a single Big Data platform may seem like an unacceptable "concentration of risk”. Mitigating the impact of bad internal actors, external security breaches, and rare faults is a difficult task made even more so when the data is collected together in a single system. Therefore, big data platforms desperately need the same robust, proven data protection, management, and security capabilities that exist in contemporary enterprise storage and database systems. Unfortunately, many Big Data technologies lack these capabilities, which constitutes the third of the three challenges.
These three fundamental challenges - “leap of faith”, “some assembly required”, and “concentration of risk” – are standing in the way of many companies implementing Big Data. At EMC, the Big Data Solutions and Federation Data Lake teams are aggressively developing offerings based on our proven enterprise products to address these challenges and enable our customers to focus on using Big Data to grow their business. With EMC, you can take a step toward the future without leaving the valuable part of the past behind.