One of the top five areas of exploration for many CIOs in 2012 was big data. And it's no wonder.
With massive volumes of valuable unstructured data permeating the average organization on a daily basis, manufacturers are in an analytics arms race to help them leverage those untapped data sets for more meaningful business insights and enhanced decision-making capabilities -- something traditional analytics solutions of a pre-big data era simply can't do.
With so much at stake, investment in big data and big data analytics is expected to see enormous growth in the next few years. Gartner, for example, predicts that big data spend will double in the three years between 2012 and 2015.
Furthermore, International Data Corp. (IDC) projected last week that the worldwide big data technology and services market will grow at a 31.7% compound annual growth rate (CAGR) over the next three years -- about seven times the rate of the overall information and communication technology (ICT) market.
As a result, we expect an increasing number of CIOs to put big data at the top of their list of strategic initiatives this year.
To help them as they explore this rich new field, I have put together five things to consider when making their 2013 big data investments.
1. Big data will create new opportunities to understand and manage things differently.
A leading MIT professor once said, “Most great revolutions in science are preceded by revolutions in measurement.”
Business analytics has given us a very clear window into measuring our customers, our business processes, our employees and our suppliers. Thanks to that, we now know what is working well within our organization and what is not.
Success with analytics technology has paved the way for the big data revolution, which promises unprecedented insights into business and provides opportunities to understand and manage things differently. But only those organizations that are comfortable with traditional business analytics will be successful with big data, due to the sheer size and scope of data they would have to manage and analyze. Getting your “foundational” business intelligence infrastructure in order is a key first step.
One of the many intriguing things about big data analysis is it often brings a view to your business that has been traditionally underrepresented in your decision making. Big data promises to bring disruptive change across all industry segments and give the companies an unfair advantage.
According to a McKinsey analysis, a retailer that can effectively derive the right insight from big data has the potential to increase its operating margin by more than 60%.
2. The correlation with social media is far greater than you think.
Social media contains a wealth of information, which if correlated with CRM data or other in-house data sets, can give you good insights into behavioral trends, customer sentiment and consumer risk.
For example, by bringing social media and CRM data together and running analytics on it, a company can use the insights to develop community-style interaction, collaboration, and camaraderie to its marketing, sales, and customer services operations.
The correlation between your in-house data and the social media data is really where the magic lies. The synthesis of those views creates a much more valid picture -- it helps to sort the noise from the key signals you are looking for.
Big data is helping a financial services company in Canada understand social relationships and create targeted campaigns to retain its customers better. MicroStrategy recently rolled out a cloud-based gateway to Facebook, that allows its clients (such as a major Telco) to enrich client and prospect data with Facebook’s Social Graph and increase its chances to cross-sell/upsell friends-and-family phone plans.
While the B2C (Business to Consumer) big data analytics applications typically bring data from Yelp, Facebook, Twitter, OpenTable and Topix; the B2B (Business to Business) big data analytics applications typically bring data from LinkedIn, GlassDoor and Business rating sites. Many organizations also bring data from internal social media sites that are typically powered by technologies such as Yammer, Jive, and Chatter.
3. You don’t have to start with big data.
Not all advanced analytics are about big data. Many business analytics needs can be met by taking data from your multiple transactional systems such as CRM, Financials and supply chain management, bringing them together into a data warehouse or a data mart, and then running analytics against them. In fact, deploying traditional business analytics solutions is the first step you need to take if you are starting your journey with analytics beyond simple dashboards and reports.
Secondly, it is important to ensure that the master data across various transactional systems is consistent, before you decide to bring big data into the equation. Otherwise the data quality issues would make it difficult to identify patterns from large data sets. Master Data Management is a good place to start to ensure your master data is consistent across systems. Just moving your current dashboards and BI reports from historical snapshots to focusing on predictive analytics can be a huge step in increasing the value of your data to your organization.
4. The cloud will play an important role.
Most of the data sources for big data are outside the firewall and in the cloud. This includes external social media sources such as Facebook and LinkedIn, as well as internal social media sources such as Chatter. Because speed of analysis on a larger set of dataset is a key consideration, often big data analytics requires unique infrastructure such as Hadoop or SAP’s HANA which you are less likely to have in your in-house environment.
There is no requirement for you to invest in your own infrastructure -- it all can be delivered as a service from the cloud. As a result, in many cases, I recommend going with a cloud-based big data model, so you can enjoy the benefits without having to purchase the unique infrastructure and without needing to worry about hiring specialist skills to manage the infrastructure.
5. Data Scientist will become a precious resource.
Data Scientists are becoming a critical asset in the implementation of big data solutions. With big data, there is no shortage of data -- what you will need are people who are well-versed in sampling methodologies, algorithms, designing experiments, and working with very, very large data sets. Their unique skill set is in synthesizing various sources of data (including unstructured data from internal social media deployments and external forums), understanding trends and then selecting the right set of algorithms to drive the discovery of the right signals.
The data that generates the key signals will change over time. In addition, the needs and priorities of the business change over time as well. While the system and algorithms can evolve at a natural pace, the Data Scientists can often drive order of magnitude increases in the efficiency of solutions through rapid iteration. As big data revolution gains full steam, Data Scientists would become a precious resource and a key to your success. Talk to your analytics partner -- they can help until you build your own capabilities.
Below are some examples of big data initiatives that can deliver big returns. If you have not created your 2013 big data plans yet, perhaps start with one of these.
Understanding your customers and hearing their feedback: There is a treasure trove of information about you and your competitors in social media. big data can combine this information with your own internal surveys, as well as your own CRM data to help you better ‘hear’ and ‘understand’ your customers. Listening to them talk about the good, the bad and the ugly about your product/pricing/packaging will tell you what is really nice and needs to be carried over into future product versions, as well as what is not as good and how it can be improved. You will be surprised at what you might learn.
Competitive Intelligence: Few things can be more valuable than knowing what people are saying about you and your competition. For many businesses, social data can provide great insight into your position relative to your competition. It can also provide amazing guidance in terms of how to position versus your competition’s weaknesses, or their perceived weaknesses. When it comes to social media, the adage ‘that perception is reality’ could never be truer.
Understanding fraudulent activity: Big data analytics can be used to identify patterns of fraudulent transactions that work against your interests. These could be fraudulent claims in your insurance business or sales transactions that create a grey market for your products, or false product returns against your product warranties.
Try doing a pilot project, see the benefits and let a full-blown program pay for itself from the cost savings. This is a section of the market that is evolving very rapidly because most companies already have some sort of analytical solution in place to help manage fraud. big data is a natural extension bringing new techniques and new sources of data into the equation.
Data sets currently available within your organization that remain unexploited are a lost opportunity.
The challenge of capturing the data lies in the speed with which the data moves, the range of data types, and the complexity of managing and processing the data in more meaningful ways that can lead to better decision-making and trend discoveries.
Start with existing data within your organization -- the real power comes in the synthesis of this in-house data with outside sources.
Come to love social media. The nuggets inside can allow you to look at your business, your customers, and your competition in new ways.
Cloud computing will play a key role, since a large part of the data you will use is already in the cloud. Take advantage of this opportunity to leverage service providers’ investments in the cloud. Don’t make building out the infrastructure your stumbling block.
Finally, build data scientists into your plan -- they will play a key role in how much value you can acquire from big data.
Todd Johnson is executive vice president and chief operating officer of Saama Technologies, a leading pure-play business analytics services firm.