The challenge of capturing the data lies in the speed with which the data moves, the range of data types, and the complexity of managing and processing the data in more meaningful ways that can lead to better decision-making and trend discoveries.
Data sets currently available within your organization that remain unexploited are a lost opportunity.
The Cloud
4. The cloud will play an important role.
Most of the data sources for big data are outside the firewall and in the cloud. This includes external social media sources such as Facebook and LinkedIn, as well as internal social media sources such as Chatter. Because speed of analysis on a larger set of dataset is a key consideration, often big data analytics requires unique infrastructure such as Hadoop or SAP’s HANA which you are less likely to have in your in-house environment.
There is no requirement for you to invest in your own infrastructure -- it all can be delivered as a service from the cloud. As a result, in many cases, I recommend going with a cloud-based big data model, so you can enjoy the benefits without having to purchase the unique infrastructure and without needing to worry about hiring specialist skills to manage the infrastructure.
5. Data Scientist will become a precious resource.
Data Scientists are becoming a critical asset in the implementation of big data solutions. With big data, there is no shortage of data -- what you will need are people who are well-versed in sampling methodologies, algorithms, designing experiments, and working with very, very large data sets. Their unique skill set is in synthesizing various sources of data (including unstructured data from internal social media deployments and external forums), understanding trends and then selecting the right set of algorithms to drive the discovery of the right signals.
The data that generates the key signals will change over time. In addition, the needs and priorities of the business change over time as well. While the system and algorithms can evolve at a natural pace, the Data Scientists can often drive order of magnitude increases in the efficiency of solutions through rapid iteration. As big data revolution gains full steam, Data Scientists would become a precious resource and a key to your success. Talk to your analytics partner -- they can help until you build your own capabilities.
