CITO Research Product Overview: Baynote: The Power of Contextual Recommendations

This article is a CITO Research overview of Baynote, and will attempt to answer the following questions:

• What does Baynote do?
• How is Baynote different from other competing products and approaches?
• Which CITO Research technology perspectives is Baynote aimed at?
• What compelling value does Baynote provide a CTO?
• What questions should be explored in future research?

What does Baynote do?

Baynote fits into a generation of companies that helps make available to every site the most advanced capabilities pioneered by companies like Amazon.com and Google. What was once only possible on the highest traffic sites that had huge research and development budgets has now become affordable. Like other companies who have productized the capabilities of the pioneers, Baynote’s personalization products are based on new mechanisms and extend the original vision of the pioneers to meet current market requirements.

To quickly understand Baynote, it is helpful to review what Amazon.com has done with its product recommendations. Amazon began its work on product recommendations by working with a company called Net Perceptions, which was in the business of providing infrastructure that could be used to make effective product recommendations. Amazon quickly moved in its own direction, and its recommendation technology is based on proprietary methods. Amazon doesn’t say exactly how its recommendation engine works, but it is safe to say that it is an algorithm that attempts to categories consumers into many different clusters based on evidence such as what products they looked at and which products they bought. A recommendation is made when someone appears to have a strong affinity for a cluster. The general idea is that an individual may want to buy the same products that others who are in the same clusters bought. The whole process hinges on gathering evidence about a consumer and then assigning him or her to the right cluster.

Baynote’s special sauce comes from the way its collective intelligence engine gathers and distills information about what consumers are interested in. Instead of just looking at what consumers have viewed or bought, Baynote follows consumers through a web site tracking every action they take, from time on page and scrolling behavior to what links they clicked on and what words were associated with each of those links. The way users describe their intent though search queries are also critical inputs. Baynote adds any available data about consumer demographics and purchase history as well. In other words, Baynote is tracking the click stream but also the semantic stream of words associated with each search and click and how both the click stream and the semantic stream are related to the products. Baynote also focuses on analyzing activity as a consumer navigates around a site. This approach allows a larger scope of behavior to become useful in classifying consumer intent. When a recommendation engine just relies on what has been bought that leaves a lot of behavior under-analyzed. People poke around sites quite a bit before and after they buy and all of this poking around reveals what they are interested in as well.

The core of Baynote’s product is an interest graph that distills the information from all of the sources mentioned into a model of a consumer’s behavior and a prediction about their intent. This interest graph is then used to recommend products, suggest content that might be useful, create navigation on the fly, and a variety of other uses.

Most of the time, companies purchase Baynote to improve the conversion of visitors to buyers. Expanding the scope of what is recommended allows Baynote to become useful in improving other processes. Baynote can be used to enhance the quality of search results. For customer support, for example, Baynote can recommend content that may help a consumer answer a question. Or in email-marketing, Baynote may be able to help recommend content and/or products that may make the email more useful to the recipient.

Baynote Origins

Baynote’s original mission was to help people find content. Baynote was founded by Jack Jia, Rob Bradshaw, and Scott Brave. Jack Jia is the current chairman of the company, Rob Bradshaw is the chief architect, Scott Brave is the CTO, and the new CEO of Baynote is Doug Merritt. The idea for the company began when Jack Jia, who was then CTO of Interwoven, saw that, as the amount of content was growing on the Web, consumers having trouble discovering the content that they actually wanted to see. Information overload was making hard to find content that was desired and useful.

Jia had direct knowledge of this problem, because during his time as CTO of Interwoven, he had bought a company that used a meta-tagging engine that was intended to help solve this problem. When attempting to incorporate the technology of this acquisition, he discovered that the “machine approach” – using algorithms to analyze and categorize content, wasn’t really working that well.

However, in Google, he saw that a key way to present information that was interesting to people was by looking at what was interesting to other people. Google makes these determinations by using the page-rank algorithm, which obtains clues to what content is valuable and important by looking at the links that people have embedded in content. Each link essentially represents one vote that this content is important.

Jia, Bradshaw, and Brave realized that the page rank algorithm worked at scale, where millions of people voted with their links, but the same approach did not work in the context of a web site, where a much smaller number of links were available, created by a much smaller group of people. The first insight that led to the mechanisms underlying the current product was that observing the behavior of consumers as they clicked through a site provided a strong indication of what interested them. The more users clicked on a link, the more interesting that link likely was. The second insight of the product was that links are expressed in words, and words can be examined to extract clues about what attracted a consumer to click on a link or express their desires in set of words use in a search.

While the initial concept of the company was to improve search, it became clear early on that recommending content and recommending products would use many of the same principles and mechanisms. So the mission changed to focus on both types of recommendations, which is still the case today.

Applications that Baynote has constructed on its collective intelligence engine help marketers sell more products, improve search results, help customer support people provide the right information to customers, and help email marketing campaigns work more effectively.

How Baynote Works

Baynote has at its core a “collective intelligence engine” that listens to evidence of behavior of people as they visit a Web site, and it creates what they call an “interest graph” that captures and distills evidence about what may interest a consumer. The interest graph is then used to make recommendations of various types. Baynote CTO Brave calls this process “interest mining” Here is simplified step by step explanation.

1. Watch and record behavior. The first step in Baynote’s process is to watch and record behavior. The main input is user behavior. Baynote places on every page of a Web site a “JavaScript observer tag”, which is very much like the analytics tag that Google Analytics uses to track activity on a site. The observer tag allows the collective intelligence engine to record everything the user does. The links the user clicks on are analyzed as well as the words that are involved. Sometimes, when someone arrives at a page from a search engine, there are a variety of search terms that could have led them there. The terms could be words that are linked, and then clicked on. There are also aspects of time: How long did the person spend on the page? Did they scroll? A variety of behavioral information can be tracked. The collective intelligence engine watches the tagged behavior and looks for patterns using several methodologies, including Bayesian clustering, as well as proprietary algorithms.

If you don’t want to use an observer tag to provide information, or if you want to supplement the kind of information collected, or you are not using HTML to present recommendations, there’s also an API that allows events that are recognized in some fashion to be sent to the collective intelligence engine. For certain types of applications, such as mobile applications, there is no click stream, and the API is the only way to record behavior. The API also allows a request for a recommendation to be made to Baynote with the result provided in XML.

Baynote can also collect information by using log file processing, so that, instead of placing the JavaScript observer tag on the pages, the software processes the Web logs.

2. Gather information about what will be recommended. Baynote contains a database of products, content, links and a variety of other Web objects that may be recommended to users.

3. Gather information about the user. When possible, Baynote adds information about consumers that come from the records of the site in question. For example, Baynote is used at Dell. When a user is logged into a site it is possible to associate that user with the products that have been purchased in the past. This information can be used to help make recommendations of additional products or of customer support information.

4. Determine the intent and interest of the person who is using the site. Baynote’s interest graph captures the dimensions of user behavior. Those dimensions include the words that were involved in the user’s searching and clicking, and it may have other user information as well. The interest graph organizes information in clusters of users based on all of the available dimensions. A multi-dimensional picture can be assembled, which examines how the words and products cluster, how the users cluster, and then makes suggestions when a user begins exhibiting behavior similar to that of past users. Baynote can then suggest recommendations based on how similar that user’s behavior is to previous users.

Baynote’s approach is unique in that it combines a many different types of available evidence in a real time analysis. The advantage of this approach is that the categorization and clustering of a user begins as soon as a user arrives at a site. The actions and behavior start the process and as much information is added as is possible. Capturing the words that are used in a search also add semantic clues, allowing more clustering and patterns to be developed.

The big differentiator of Baynote is its semantic layer of information. This layer includes words in the click stream, providing much more powerful information than looking at click behavior alone. Many of the other engines focus on clicks and purchases. Clicks are important information, but without the semantic content, there is a lack of contextual information that you could gather about the clicks. Purchases are also very important, but there are always people are on your site conducting plenty of activity that does not include a purchase. If you only focus on purchases, you don’t learn anything about the intent of your users, and the aspirations they have for what your site provides, but currently doesn’t.

The typical downside of the collaborative filtering approach is that it delivers clusters that don’t really provide an explanation of why they were formed. It is not uncommon for a group of users to be identified because they all acted in the same way or bought the same products. The reasons why may be unclear. When you add word analysis to collaborative filtering, it provides semantic clues, resulting in far fewer unexplainable clusters. In addition, the quality of the recommendations is increased because the interest graph is based on more evidence and becomes more complete sooner.

The interest graph provides three pathways to understanding. You can start with user clusters and see how they are associated with words and with products. Or you can start with words and see which clusters and products are associated. Or you can start with products and see which words and clusters are associated. By examining your audience and their behavior in multiple ways, new insights are often gained.

Product Features

Baynote has a variety of mechanisms for putting the collective intelligence engine and the underlying interest graphs to work:

Controlling Baynote through the Optimization Center

The Optimization Center allows recommendations to be configured and inserted into web sites. The goal is to maximize conversions by providing consumers the best recommendations while at the same time living by business rules and attempting to optimize the user experience.

Recommendations must frequently follow rules. For example, a site may never want to recommend two competing products at the same time, or may always want to recommend certain sets of product together. Recommendations may be time or date sensitive. The optimization center allows all these rules to be expressed. In addition, once you have configured a recommendation profile, the optimization center generates the HTML and Javascript so that the recommendation can be inserted at the appropriate location in a web site.

There are variety of other technical details that are addressed by the optimization center, such as making sure that the HTML produced plays nicely with whatever cascading style sheets.

Types of Recommendations

Product Recommendations. At most Baynote clients, the largest economic impact comes from an increase in sales brought about by better product recommendations across customer channels.

Content Recommendations. Content recommendations (e.g., articles and videos) can be used for a variety of purposes. Two of the most popular ways are to help consumer find content that helps them solve a problem or to recommend content that may raise awareness that eventually leads to a sale or some other desired behavior.

Onsite Search. Onsite search optimization applies content and product recommendations to search results. The recommended results can be layred inside search results or can appear adjacent to search results.

E-mail Recommendations. In this approach, recommendations for products and/or content are placed in an email based on what is known about the recipient and the content of the email.

Landing Page recommendations. Baynote dynamically optimizes landing page recommendations using contextual information passed through the URL from Google, Bing, or Yahoo.

Dynamic navigation. In this approach, Baynote dynamically constructs navigation for a web site based on what the collective intelligence engine predicts the user is interested in. This approach can be used to steer a user toward content they desire or toward products that may be of interest or both.

Analytics

The second largest economic impact from using Baynote comes from the analytics produced by the collective intelligence engine. Reporting on the interest graph can reveal the ebb and flow of customer tastes. A variety of reports can be created to show the rising and falling popularity of products, content, and specific types of clusters. This information can be helpful in making merchandising decisions, constructing promotions, optimizing site navigation, and evaluating the impact of advertising and marketing campaigns.

The gap report has had a particularly large impact at many Baynote customers. Based on search engine keywords used for incoming traffic along with an analysis of the existing content and products on a site, the gap report shows what visitors to the site are looking for but not finding. This sort of analysis can lead to important insights about how consumers actually perceive a site or the products on a site. Here are three examples:

• In one case, the gap report showed that customers were looking for a product using a different word than was commonly thought to use to identify the product. By associating the product with the word actually used by consumers, sales increased.
• In another case, an on-line retailer found that consumers were looking for a specific product category that was missing from the product selection. By adding that product category, the company created a whole new line of business.
• The gap report also showed at one retailer that consumers were looking for a specific make, model, and color of a product. It turned out that sales for that product increased across all channels for all retailers. The gap report provided an early indication emergence of this trend in consumer tastes, before it showed up in sales figures.

Use cases

As the flexibility and power of Baynote’s collective intelligence engine has grown, a variety of new applications have emerged. The following use cases represent the most common ways Baynote is applied.

E-commerce. Baynote’s personalized recommendations are used to increase profitable revenue on e-commerce sites by recommending products of interest to visitors. In addition, the content and navigation recommendations can also raise awareness of desirable products.

Customer support. When a vistor to a site is trying to obtain support himself, or is perhaps getting support with the help of someone else, Baynote can recommend which support articles would be most helpful. The recommendations can be based on information about a visitor’s past buying history, the products in which they have expressed interest, or the products of interest to similar clusters in the interest graph. Recommendations can be provided via the content on pages, via navigation links, or via search results. One new application of Baynote for customer support involves monitoring chat sessions and making recommendations for content or products in adjacent windows or to the support representative participating in the chat.

Marketing. Marketing applications use Baynote’s product and content recommendation engine in the preparation of marketing content such as outbound marketing emails and other content that will be presented to consumers.

How is Baynote different from other competing products and approaches?

There are two kinds of competitors for Baynote. One is recommendation engines. These are other companies that use similar or different approaches to recommend things. Included in this category are a variety of companies who use various forms of clustering and other mechanisms to categorize consumers and make recommendations. One other type of competitor includes the multivariate testing approach, in which the basic principle of A/B testing is dramatically extended and consumer tastes are determined experimentally.

The other major competition for Baynote are analytics vendors that offer products to help understand consumer behavior on a site. Baynote can provide a rich set of reports and various kinds of analytics. Numerous other products offer a wide range of analytical capabilities.

All of the vendors in recommendations and analytics compete based on their approach to analysis. Most such vendors combine well-known techniques such as Bayesian analysis with proprietary techniques of various sorts. Baynote’s differentiation from vendors in both of these categories comes from the way it combines real-time behavioral analysis, semantic analysis, and historical information. Baynote has also been in the market for longer than most recommendations engine competitors, which shows up in features that enhance ease of use.

Which CITO Research technology perspective is Baynote aimed at?

Baynote is aimed at the departmental or corporate perspective of a company that has a significant investment in generating revenue through a consumer-focused web site. Baynote has the best chance of success at a site that already has significant traffic and revenue. Baynote will act as an amplifier to a successful site more than an accelerator of traffic growth for a newly created site.

Just as Unica, a marketing automation vendor, was purchased by IBM and made a company-wide standard, it is possible that executives operating at the multi-organization perspective may do the same with Baynote.

What compelling value does Baynote provide a CITO?

A CITO should be interested in Baynote as part of a general program of increasing the financial performance of an e-commerce site. Baynote is based on the Software as a Service delivery model, so it will not require any additional footprint in the data center. CITOs should evaluate Baynote as part of a program to increase the conversion of Web traffic into sales. Companies spend a lot of money doing marketing and promotions and a variety of other activities to get people to come to their Web sites. Increasing the conversion rates, customer satisfaction, or the ability of customers to find what they are looking for increases the return on the investment in a web site. Baynote provides, first, a way of increasing those conversions, and second, a source of sophisticated analytics that allow you to understand your audience much more completely.

What questions should be explored in further research?

How can Baynote be used to set alerts for specific levels of activity on a site to provide alerts that may lead to changes in recommendations, promotions, and site navigation?

How can automated responses be crafted to react to alerts in specific, pre-programmed ways?

What other sorts of information that may be available from mobile applications such as location can be used to improve the collective intelligence engine?

What other sources of information about real-time consumer trends, such as social media analysis could be useful in expanding the number dimensions in the interest graph?