Marco Pacelli of ClickFox: Research Interview

One of the ways that companies are going to get value out of the deluge of big data is by buying general purpose tools to manipulate big data and perform various types of analysis. Hadoop vendors (Cloudera, MapR, Horton Works), Splunk, 1010data, Teradata Aster, EMC Greenplum, Revolution Analytics and a host of others fall into this category. Another way to put big data to work is buy an analytical applications like ClickFox, MetaMarkets, and a few others that are aimed at gaining value out of a specific type of big data that holds significant promise.

I had a great talk today with Marco Pacelli, CEO of ClickFox, along with Tom Wheeler, SVP of Research and Development, and Joe Galvin, CMO, about the present and future of analytical apps.

The conversation began with all of us agreeing that the data warehouse paradigm was changing. Marco was on board with the broad strokes of the Data Lake concept I've been developing. The Data Lake is a vision for a heterogeneous repository that accommodates unstructured, multistructured, and structured information. This problem statement gets the idea across. In a Data Lake, unlike a data warehouse, the way the information is distilled to support analysis will happen at analysis time not at design time.

Marco pointed out that companies are now struck with a sort of "data paranoia". They are obsessively storing all the information they collect without clear thinking about how they are going to put that to use. The result is that the budget for storage is going through the roof. This is a bit backward. Storing data doesn't create value. Analyzing it does. We were both in agreement that it makes much more sense to spend money on capabilities to understand the value of data rather than just storing more and more of it. As part of my work on Agile Big Data, I'm trying to identify the lowest cost and easiest to use ways of determining what questions a big data set can answer.

ClickFox was interesting to me because it naturally fit into my thinking about data lakes and agile big data. ClickFox is an analytical application for data that tracks interactions. This data includes web logs, call detail records, purchase logs, any pretty much anything that represents an interaction between a person and something else. ClickFox takes these logs and automatically distills them into a model that represents the beginning to end flow of the interaction. You can then enter the model from the consumer point of view, the product point of view, or the point of view of some aspect of the system, such as churn rates. Of course, the automatically generated model must be tuned, pruned, and polished to make it most effective, but the trip from data to useful analysis is dramatically shortened by this approach.

One powerful aspect of ClickFox is that it handles multiple stream of interactions at once. This is a huge deal for marketers who are looking to analyze multi-channel interactions with customers. It will be a big deal for other categories of executives as well as multiple streams of evidence become available. Of course, you have to create functions to map the identifying information in each log to a common name space, not usually very difficult.

The ClickFox models become more powerful when you introduce additional information about the characteristics of the individuals and products to help increase the granularity of the segmentation.

I'm planning on using what I learn about ClickFox in upcoming stories about the emergence of analytic databases and how analysts are grappling with ETL, among others.