Preparing to take advantage of the business value that big data can provide is going to be a multi-year process that takes place in several stages. Most of the new sources of data arriving under the banner of big data are fundamentally different than the type of data stored in most data warehouses.
A new infrastructure will have to be constructed that stores data differently and allows for different sorts of distillation of the data into a form that allows further analysis. In addition, new tools will be needed to search the data, understand it, and perform analysis on it.
This problem statement examines the nature of that transition from all relevant dimensions. We will look into the way that IT architecture and infrastructure will change, examine the relevant technology for storage and analysis, and look into the way that big data will be analyzed in real time. In addition, we will examine the organizational and change management issues that are likely to appear. (See Forbes.com: "Big Data Requires a Big, New Architecture" and "Kill Your Data Warehouse" for an overview of some of the architectural issues.)
Context and Background
Most companies have a web site or other systems that have detailed information about consumer behavior. Web server logs, logs from networking and telecommunications equipment, data from sensors of various sorts, data from e-commerce or other transactional systems all can tell a story of some sort about what his happening with customer, partners, or key business processes.
The question is: What good, if any, is such information? So far, the news about the value of big data is promising. Large e-commerce web sites are able to use big data from web server logs to better understand what consumers are doing.
This allows adjustments to be made to navigation and product offerings to take advantage of discoveries about consumer preferences. Telecom companies are using distilled call detail records to respond faster to fraudulent use of the network.
In operational environments, big data can be used to get a much more detailed picture of the state of an environment like a factory or a refinery. This more detailed picture provides early warnings of trouble and can help optimize maintenance processes.
The analysis of social media data has proven to provide detailed indications of consumer sentiment before it appears in consumer buying behavior and other forms of behavior. It is clear that there is value in big data, but the path to finding new insights that are relevant to a specific business is less clear.
This problem statement should end up creating a roadmap that a CITO can follow to understand the opportunity that big data provides and how to build the infrastructure to support it.