White Paper: Incremental Search Revisited

AJAX Incremental Search Revisited

How Structured Data Matching Engines Enable
a Revolutionary Incremental Search Interface

In the process of conceiving structured data search solutions, Exorbyte engineers have experimented
with the use of AJAX interfaces (asynchronous JavaScript and XML). This paper illustrates some of their
most promising findings and how the use of the MatchMaker high-performance approximate search
server in this setting created results above all expectations. These experiments have led to the
1
development of a superior AJAX incremental search interface . This new system turns the tables on
what benefits interface designers and software developers usually expect from incremental search.

Table of Contents 
I.  Introduction and Context .......................................................................................................... 2 
II.  AJAX Incremental Search Detailed Features and Characteristics ............................................... 5 
A.  Basic AJAX Incremental Search Described ................................................................................................... 5 
Key Benefits of Basic AJAX Incremental Search ................................................................................................ 6 
Shortcomings of Basic AJAX Incremental Search.............................................................................................. 6 
B.  AJAX Incremental Search Powered a Structured Data Search Engine ......................................................... 7 
Example: Yellow‐White Pages Search Interface ............................................................................................... 7 
Interface Design Features of AJAX Incremental Search Interfaces ................................................................... 9 
Example: Advanced Incremental Travel Search ............................................................................................. 10 
C.  Systems Architecture of an Advanced AJAX Incremental Search .............................................................. 12 
Performance Issues with Incremental Search ................................................................................................ 12 
Improving, Not Replacing Delimited Search ................................................................................................... 13 
III.  Business Performance, Risks and Cost of AJAX Incremental Search Systems ........................ 14 
A.  Benefits of Various Incremental Search Systems ....................................................................................... 14 
Benefits  to the Users ..................................................................................................................................... 14 
Benefits to the Content Service Operators .................................................................................................... 15 
Example: Price Comparison Portal ................................................................................................................. 15 
B.  Cost and Risk Analysis ................................................................................................................................ 16 
Cost Matrix of Incremental Search System Types .......................................................................................... 17 
Optimizing the Cost vs. Feature Richness Dilemma of Incremental Search Interfaces .................................. 17 
IV.  Other applications of AJAX Incremental Search Powered by Data Matching Servers ........... 19 

1
Incremental search is sometimes called “incremental find”, “typeahead search”, “autosuggest”, “autocomplete”,
“combo box”, “search as you type”, “filter/find as you type” (FAYT), “inline search”, “instant search”, “word
wheeling” or even “Google suggests”:
http://en.wikipedia.org/wiki/Incremental_find

www.exorbyte.com
+1 503 616 4007 1/19
White Paper: Incremental Search Revisited

I. Introduction and Context

Incremental search is often opposed to
“delimited search” in which a user is required to
fully formulate a search query before the system
starts processing the search request. This is the
traditional way most users have gotten used to
search documents, operating systems,
databases, etc. The user has no visibility into
how effectively his search query may locate
information of interest until the processing has
ended. On the other end, Incremental search
starts right when the user types his first few
query characters. Search query suggestions or Example of Incremental Search Interface
search results are instantly presented to the user
based on his partially typed query milliseconds after each character stroke. On web sites these results
usually appear in a JavaScript layer which loads and updates itself asynchronously as the user keeps
on typing. “Asynchronously” here means without ever requiring submission (a click) and the loading of
a search results page. This technique has become known as AJAX (asynchronous JavaScript and XML).

In his book The Humane Interface, renowned user interface expert Jef Raskin compares incremental
and delimited search as such: “With a delimited search, the computer waits for the user to type a
pattern and delimit it, after which it is the user who waits while the computer does the search. When
using a delimited search the user must guess, beforehand, how much of a pattern the computer needs
to distinguish the desired target from other, similar targets. With an incremental search, he can tell
when he has typed enough to disambiguate the desired instance, because the target appeared on the
display. (..) In spite of near agreement about the desirability of incremental searches on the part of
both designers and users, almost all interface-building tools make it easy to implement delimited
searches and difficult or impossible to implement
incremental searches.”2
“Despite near agreement
about the desirability of The Internet software community has come to appreciate
incremental searches most and recognize the unique benefits of AJAX incremental
interface-building tools search interfaces. AJAX interfaces enable client-side
interactive web applications capable of retrieving data
make it difficult or from the server asynchronously in the background without
impossible to implement interfering with the display and behavior of the web page.
incremental searches” They offer users immediate suggestions for what they might
be searching for as the user types his query. Such search

2
The Humane Interface, Jef Raskin (creator of the Macintosh Interface) - 2000 -
http://books.google.com/books?id=D39vjmLfO3kC&lpg=PA126&ots=COqFf33VS9&dq=incremental%20%2
2delimited%20search%22&pg=PA126

www.exorbyte.com
+1 503 616 4007 2/19
White Paper: Incremental Search Revisited

query suggestions are usually displayed in the form of a drop-down box and have appeared on most
large online public search services in the recent years.

These incremental search interfaces are inherently limited by the speed and sophistication of the
underlying index and data recall mechanisms. Many rely on simple client-side JavaScript programs
offering access to a limited list of popular query suggestions (example current eBay.com incremental
search widget). Some are server-side databases or flat-file lists limited by their response time latency
and their capacity to display complex results sets (ranked, faceted, real-time, from large data sets, etc.)
as a user types his query.

Ever since the Internet opened worldwide access to ever “Large bodies of content
larger information sources, the need for meaningful search
solutions grew in parallel to the explosive growth of
are mostly composed of
accessible content. Search software has evolved from text and data records of
simple web-focused full-text crawler-indexers to today’s which users do not suspect
multi-segment search solutions for web pages, documents,
the existence. Only a very
databases, images, audio, video, etc. The search
software tools used for the most sophisticated enterprise, e- fast approximate search
commerce, or web search utilities are chosen from an engine can display these
expanding menu of algorithmic, semantic, phonetic, and suggested matches with
other advanced software components. AJAX incremental
the speed and relevance
search brings a sense of unity between all these tools.
These advanced search tools can now be at the service of incremental search
a natural intuitive user interface rather than the other way requires.”
around.

One of its most impressive benefits is its ability to automate the search of content users do not suspect
is in existence. “Large bodies of content are mostly composed of text and data records of which users
do not suspect the existence. Only a very fast approximate search engine can display these
suggested matches with the speed and relevance incremental search requires.”

Exorbyte’s search software expertise is in the area of structured data and semi-structured data
(database, directories, XML feeds, etc.) is recognized by a diverse worldwide customer base and has
led to the development of MatchMaker, its structured data search engine. MatchMaker’s uniqueness
comes from its ability to process very complex algorithmic, phonetic, and otherwise scripted queries at
record speeds over very large data sets. In search environments, MatchMaker usually serves as an
API-accessible backend for user-facing applications or as a search engine for structured data (e-
commerce, directories, etc.).

An AJAX search interface tied to a MatchMaker structured data search engine with a sophisticated
configuration offers users a very interactive, instant and unprecedented view of the deepest most
complex data sets possible. Much more than a “typeahead” or “autosuggest” like those offered by
most search engines today, this sort of implementation can tremendously enhance the ability of users to
validate their queries, explore deep data structures and the ability of administrators to customize search
interfaces with little development effort for both structured and unstructured data sets.

www.exorbyte.com
+1 503 616 4007 3/19
White Paper: Incremental Search Revisited

The name given to the MatchMaker extensions for

“AJAX incremental search incremental search discussed in this paper are
interfaces can give users a SearchNavigator and FlexSearch. They are two unique
AJAX frameworks3 and search interfaces combining the
stunningly informative
proven benefits of AJAX incremental search with its highly
bird’s eye view into large versatile structured data search engine: Matchmaker.
and complex data sets” Exorbyte’s AJAX frameworks’ most impressive ability is that
they give users a stunningly informative bird’s eye view into
large and complex data sets. Such a view is only achievable because of the speed and versatility of
the underlying MatchMaker engine. The possibilities offered by the infinite combination of search
query parameters available in MatchMaker and the very intuitive interface of the AJAX frameworks
appear to be limitless. Such an interface improves the traditional search experience with query
rewriting through dynamic suggestions. It also allows the display of advanced search results ahead of
query submissions. Users can finally click on the suggested results and navigate straight to content,
thereby bypassing entirely the traditionally cumbersome and costly search steps that often lead users to
dissatisfaction and online business to lost transactions and lost revenue. The experience gained with
SearchNavigator already shows tremendous search behavior changes that will profit both users and
content publishers whether they offer search within full-text web pages or enterprise-scale relational
databases (ecommerce, etc.).

3
An Ajax framework is a framework that helps to develop web applications using Ajax, a collection of
technologies used to build dynamic web pages on the client side.
http://en.wikipedia.org/wiki/Ajax_framework

www.exorbyte.com
+1 503 616 4007 4/19
White Paper: Incremental Search Revisited

II. AJAX Incremental Search Detailed Features and Characteristics

A. Basic AJAX Incremental Search Described

An AJAX incremental search interface starts with a number of simple characteristics:

• Looks and behaves like any other classical search query interface.
• Does not require software / internet users to adjust their behavior if they prefer not to.
• Does not require any user education or explanations.
• Only adds new functions to a very familiar classical search query interface.

A classical delimited search query interface

• As a user types text in a search query field, and gets instant feedback before getting to a
whole new user interface: the search results page. This simple and intuitive feedback usually
includes suggestions drawn from a limited set of normalized queries, popular search queries,
or a dictionary-style list of terms.

Basic Google search suggestions, drawn from a set of most popular queries.

There are a couple of key technical constraints posed by even the most simple incremental search
systems as described above:
• Highly increased number of queries per second generated by each keystroke (or at least each
stroke with pause4) compared to just one query by submission. This is particularly relevant for
AJAX client-server systems where server load and bandwidth have a significant cost.
• Absolute necessity of advanced approximate matching capability. An incremental search
system not capable of matching partial words or misspellings would have little to no utility.

4
Most incremental search systems only generate a suggestion if the user pauses more than a few milliseconds.

www.exorbyte.com
+1 503 616 4007 5/19
White Paper: Incremental Search Revisited

Key Benefits of Basic AJAX Incremental Search

The basic implementation of an AJAX incremental search described above offer a number of new
advantages for their users and those operating web content services. The main advantages can be
summed up as follow:
• Search suggestions: Users get suggestions for search query terms they may have misspelled (a
sort of user-assisted “query-rewriting”).
• Implementation requires little or no incremental
development by operators of content search
services since it works in parallel with underlying
“Today’s basic AJAX
search engine. incremental search
• Users get a limited view of what others might be interfaces can never give
looking for and how others formulate multi-word
users a real sense of what,
queries.
• Some query misspellings can be caught by the in the searchable content,
stemming5 effect of the interface when a word is not resembles what they are
yet fully typed. looking for”
Shortcomings of Basic AJAX Incremental Search
It is also important to note that a number of problems remain for content service operators and for users
of such basic AJAX Incremental Search systems:
• These suggestions come from a data set that needs to be regularly updated, maintained and
normalized. Not doing so, may lead to “no result” search results page, popular misspelled
queries being suggested, and no suggestions for seasonal or news-related search queries.
• Such an implementation unless in very low usage environments (intranets, single user, etc.) is
either:
o a light client-side system which can hardly be based on more than a few thousands
records or be difficult to serve as web content,
o or a client-server system requiring a structured data search engine capable of handling
each character stroke queries fast enough to diminish latency.
• This implementation is not drawing results from the actual index or an underlying searchable
database. Users could assume that each suggestion will lead to a meaningful result. That’s
often not the case.
• In general this implementation does not give users any real sense of the results they might find
if they chose any of the suggestions returned. It can mislead users in thinking suggestions
come from searchable content, but can never give them a sense of what, in the searchable
content, resembles what they are looking for.

5
http://en.wikipedia.org/wiki/Stemming

www.exorbyte.com
+1 503 616 4007 6/19
White Paper: Incremental Search Revisited

B. AJAX Incremental Search Powered a Structured Data Search Engine

The last of the shortcoming cited above for basic AJAX

“Groundbreaking
improvement: reliably
serving faceted search
suggestions to millions of
users, from millions of
searchable indexed
records, using algorithmic,
phonetic and any other
advanced scripted match
sequence.”
Incremental Search interfaces is at the core of most of the
improvements of advanced AJAX Incremental Search
interfaces like Exorbyte’s SearchNavigator. The pairing of
an AJAX Incremental Search interface with an advanced
structured data search engine like Exorbyte MatchMaker
allows a groundbreaking improvement: the incremental
search interface then becomes more than a way to
“search” for content; it also enables “direct navigation” to
target content. Such a system reliably serves faceted
search and content suggestions to millions of users, from
millions of searchable indexed records, using algorithmic,
phonetic and any other advanced scripted match sequence
in an AJAX suggestion interface.

It is also important to assess what the business benefits and costs of these sophisticated search
interfaces can be. In industries such as ecommerce an
advanced search system has a direct impact on “the incremental search
measurable business performance variables like revenues,
conversion rates, average purchase value, repeat interface then becomes
purchases, sales of rare items, sales of difficult to find more than a way to
items, user satisfaction and loyalty, etc. Implementing and “search” for content; it also
maintaining such interfaces offer measurable cost benefits
enables “direct navigation”
over other approaches. We will look at what financial
performance impact the use of such advanced search to target content”
interface can have in these settings.

Example: Yellow-White Pages Search Interface

Before highlighting specific features of advanced AJAX incremental search applications such as
Exorbyte SearchNavigator, here is a description of of these interfaces.

In this first example, we examine a public web directory called Genialotel6 which allows searching
through to about 40 million German name/phone/address listings for individuals and businesses in
Germany.

Genialotel offers a simple who/where two field search interface featured in Figure 1. Here are some
specific features which differentiate this first example from previous AJAX incremental search interfaces
discussed in this paper (see cross-references in Figure 1):

6
http://www.genialotel.com/index/en

www.exorbyte.com
+1 503 616 4007 7/19
White Paper: Incremental Search Revisited

(1) Dual purpose: This incremental search query interface is and remains a classical search form if
users choose to use is as such and ignore the suggestions presented.
(2) Virtually no performance-related query parameter limitations: In most directory search systems,
searching for a partial name plus two letters of the city name as in the example in Figure 1
(“FR”) is impossible with 40 millions records. Computing latency would be too long if at all
possible without failure. Here the results are returned fast enough to be updated at every key
stroke (few milliseconds).

Figure 1 - GenialoTel suggestions drawn from real content, faceted, ranked, taxonomized, counted, and more.

(3) Advanced approximate matching allows finding content the user didn’t suspected existed:
Results are approximate: the result set displayed is not only the product of exact match or basic
stemming and other simple close matching routines. It comes from a complex sequence of
string matching and data matching rules associated with this particular search mode.
(4) Search results are categorized, faceted, ranked, or displayed in any other advanced way.
(a) Search results are presented ranked according to relevance, proximity, or any other
parameter.
(b) The search results in this example are also presented in two possible layouts : restricted
by current “where” value or unrestricted “Best name matches in other locations”.
(c) The possible results behind each suggested query offer a count of possible matches.
(d) More results can be displayed if needed.

www.exorbyte.com
+1 503 616 4007 8/19
White Paper: Incremental Search Revisited

(e) “Best matches” suggestion are displayed with associated zip code and city for each
record to ensure disambiguation by user.
(5) Parametric incremental search results for all possible usages of the query form. Whether users
choose to start with “where” or “who/what” field, the AJAX incremental search system
(SearchNavigator) always passes on a query to the underlying engine (MatchMaker) and
returns customized suggestions results from the full data set. Not visible here on the form but
visible by visiting the Genialotel7 site.
(6) Custom interface: Interface design is fully customized: There is no risk of users thinking this is
just a Windows or Internet Explorer suggestion set from cache memory. Colors and well
structured design attract their attention to the suggestions presented.
(7) Direct Navigation – When search results can be displayed in lists of best unique matches with
little ambiguity, it is possible and of course recommended that users clicking the item be
directed straight to the target item’s content page (in this example the full directory phone data
for that record). Alternatives, such as filling the search form with the item’s data, or sending the
user to a search result page, don’t make much interface design sense. Whenever clicks and
ambiguity can be eliminated, they should be.

Interface Design Features of AJAX Incremental Search Interfaces

The possibilities offered to interface designer are limitless with advanced incremental search interfaces.
With an AJAX connection to a fast data search server, designers are capable of bringing all kinds of
information and components straight to the incremental search drop-down interface:
• Direct navigation: with an advanced incremental search system, designers can not only
provide users with search suggestions but they can also design a special navigation logic
allowing direct access to featured content. Users will be provided with enough information
about their suggestions to validate and finalize their search immediately. Instead of clicking
and choosing queries to submit to an underlying delimited search system, users click and
navigate directly to the target content.
• Flexible design choices: Incremental search dropdown interfaces include content which could
just as well be displayed anywhere in the page in a JavaScript layer or other inline le\element
• Programmable logic: the default logic of incremental search interfaces (user types >
suggestions appear > final query submitted > search results page) can be improved or
replaced by any other programmed logic. It may include more steps, more interactivity, more
content, and more intelligence (see “Advanced Incremental Travel Search Example” next).
• Biased search results using business-driven or usability-driven variables (relevance, prices,
sponsored records, etc.)
• Pictures and ads can also be displayed in the drop-down randomly or based on programmed
logic just as they would anywhere else.
• Special graphical interfaces: geographical mapping interfaces and other dynamic graphical
UIs can leverage the AJAX connection to an advanced structured data search server.
• Data mashups: multiple data sources can be organized using the logical layers of the
structured data index of the search server and be displayed and combined through the
incremental search client.

7
Visit http://www.genialotel.com/index/en to see this system working.

www.exorbyte.com
+1 503 616 4007 9/19
 White Paper: Incremental Search Revisited

Example: Advanced Incremental Travel Search

As the previous example of an AJAX incremental search shows, such systems can actually show any
number of parametric, faceted, categorized, and ranked results in a instant interactive interface during
the user query formulation stage of the search process. This immediate feedback can be tailored to
the specific form field, and combination of form fields that a user has filled. This combination of
capabilities naturally led Exorbyte engineers to a more advanced version of SearchNavigator for
particularly complex “advanced search” user interfaces: FlexSearch. Examples of applications are
notoriously difficult search interfaces such as online travel search, online job search, and any other
search requiring a large number of inter-dependent variables (form fields).

To illustrate such an application we will look at public implementation on the leading European travel
service L’Tur8. Intuitive travel search
systems are particularly difficult to
develop because they involve many
possible search scenarios that, if all
displayed in a single interface, would
become extremely confusing to users.
Most travel agencies have resolved to
offer multiple search interfaces (Flights,
Hotels, Car Rentals, Packages, etc.)
Even within each category search
remains complex. Flights search alone
often requires 15 to 20 variables to
allow users to find relevant travel
options (departure date, return date,
times, airports, direct/indirect options,
The legacy L’Tur search user interface is complex, and involves multiple tabulated passenger numbers, adults, children,
interfaces for travel types (last minute, flight + hotel, train + hotel, flights, hotel) airports, etc.)

Because the L’Tur service is focused on last minute travel packages it was important that users be able
to find travel package inventory items “buried deep” in the L’Tur catalog. It was also the desire of L’Tur
to offer a single interface for searching within flights, hotels, cars, packages. With FlexSearch below,
exorbyte was able to provide L’Tur with am AJAX-based configurable interactive search configurator
capable of guiding users through the complexity of building a travel search just like a live sales person
could.

8
Visit http://www.ltur.com/de/search.ltml?flexsearch=yes# for a live version of Exorbyte FlexSearch.

www.exorbyte.com
+1 503 616 4007 10/19
White Paper: Incremental Search Revisited

The FlexSearch Interface on L’Tur offering users an impressively simple start: one simple search field

The FlexSearch Interface on L’Tur guides users intuitively to precisely what is available in the catalog. Price, service levels, hotel and other
subcomponents, durations, locations, and all other trip variables are proposed to users based on their desired inputs.

The Exorbyte FlexSearch incremental search implementation above demonstrates the synergistic
combination of an advanced data search infrastructure (matchmaker) paired with an AJAX interface
(FlexSearch). It not only allows displaying the most advanced types of search results instantly as users
formulate their search queries, but it also:
• Changes nothing to the legacy search systems but adds value to the search experience.
• Extrapolates “long” or “advanced” advanced queries suggestions which users may not
formulate spontaneously from few characters already typed.
• Assists user in building multi-variable queries with complex decision logic preprogrammed in
the FlexSearch configuration.
• Eliminates “not founds” due to query and data misspellings or typos by relying on advanced
approximate algorithmic string matching algorithms.
• Displays suggestions from actual catalog or index data along with associated key attributes
(price, hits, and matching categories).
• Eliminates the need for Search “Back-and-Forthing” from query to results and back to refine
query enough to hit a rare content target.

www.exorbyte.com
+1 503 616 4007 11/19
White Paper: Incremental Search Revisited

• Ensures even the rarest of items are found by allowing biased ranking of items in suggestions.

C. Systems Architecture of an Advanced AJAX Incremental Search

Performance Issues with Incremental Search

As mentioned before advanced incremental search clients generate a high number of queries to the
server (potentially one call for every user keyboard stroke in search field). The design of these
interfaces is also likely to create a natural expectation of approximate matching: Partially typed search
terms are sent to the server and need to be matched. Users also expect misspelling suggestions and

Architecture of an AJAX incremental search system powered by a structured data search engine

www.exorbyte.com
+1 503 616 4007 12/19
White Paper: Incremental Search Revisited

more advanced systems such as Exorbyte SearchNavigator and FlexSearch even provide the user with
suggestions semantically relevant to their partially search term (through taxonomies, aliases, synonyms,
etc.).

These factors tend to be extremely costly in hardware resources and only a high performance
structured data search engine like MatchMaker can truly handle such throughput and scale with the
surrounding web applications. Hardware optimization and specialized indexing architecture allow
resolving many of the issues which would slow down or fail a traditional search engine. A high
performance approximate search system like MatchMaker must be architected with such considerations
in mind. The chart on the following page shows the typical layout of such a solution.

Improving, Not Replacing Delimited Search

It is also important to note that the implementation of an incremental search interface, even one
powered by an advanced data matching server like Exorbyte MatchMaker, is an addition to, not a
replacement of existing search systems. As shown on the chart on the next page, the delimited search
queries may still flow through the existing search systems after AJAX incremental search has been
implemented.

From a UI design point of view, the two search systems are very much related. For instance, the
incremental search user interface should not display search results that may not be easily found by a
similar delimited search query. However, from a systems stand point, the only two points of contact
between the delimited search systems and the incremental search systems are:
• a data integration point allowing the incremental search index to always be the same as the
delimited search index,
• and some client-side (JavaScript) widget allowing the existing search form to generate
incremental search queries so users can see real-time results ahead of submitting their final
search query and being routed to search results of the target content item itself.

While delimited search requests may still be directed to the legacy search engine, to a database or to
other content management applications, incremental search calls to the server are made through
asynchronous JavaScript and XML (AJAX) HHTP requests to the structured data search engine. That
specialized search engine holds a partial, complete or even enriched copy of the main index
(ecommerce catalog, directory, or any other structured content type). It is able to process vastly more
complex error tolerant requests with partially typed form inputs and return suggestions within a few
milliseconds for every user character stroke.

www.exorbyte.com
+1 503 616 4007 13/19
White Paper: Incremental Search Revisited

III. Business Performance, Risks and Cost of AJAX Incremental

Search Systems

An AJAX incremental search and data matching engine implementation require careful planning and
configuration to ensure optimal results. While the cost of such systems usually represents a small
portion of a larger online content delivery deployment (online media, major yellow pages sites, etc.),
many businesses today pay close attention to the costs and benefits associated with each of their IT
investments. It is therefore important to define how content services operators can approach the
questions of cost, risk and performance when planning such deployments.

In his 2000 book The Humane Interface, interface design expert Jef Raskin says "from the point of view
of interface engineering, the advantages of incremental searching are so numerous and the
advantages of delimited searches so few that I can see almost no occasions when a delimited search
would be preferred." While this qualitative endorsement by a renowned expert is significant, it doesn’t
tell business operators what the impact and cost of implementing incremental search may be.

A. Benefits of Various Incremental Search Systems

Improving content search applications (directories, ecommerce catalogs, travel services, e-government,
vertical search engines, online media, etc.) usually requires a carefully planned redeployment of
backend systems and back-office processes; thus generating significant indirect costs. However, all
costs and risks associated with such deployments should be offset by the benefits of a better search
and navigation experience. It is difficult to quantify in advance the benefits to be expected from the
redeployment as well as the maintenance cost of the systems and specialized content it may require.
Exorbyte’s experience has showed a number of constants that emerge from most implementations.
These lessons drawn from fact can help organizations conceive a framework to approach the
cost/benefits questions when conceiving a new or improved incremental search system.

Benefits to the Users

The addition of incremental search to an existing content
search interface creates some clear improvement for users:
• Incremental search diminishes or eliminates “not
“In the case of Billiger.de,
found” search results because incremental search the 1st month of
allows users to validate a query string based on implementation saw a 5%
existing matches or likely matches from the search increase in the conversion
content.
• Incremental search allows users to have a bird’s eye
rates (…) and the relative
view of complex bodies of searchable content use of search versus the
through incremental displaying of exact matches navigation of categories
and approximate matches based on string matching
(directory style taxonomy)
increased 20%”
www.exorbyte.com
+1 503 616 4007 14/19
White Paper: Incremental Search Revisited

techniques, taxonomies, and faceted search techniques.

• Incremental search eliminates misspellings, typos, and other data entry errors in the search
query strings or the searchable content.
• Incremental search diminishes user flight (dissatisfaction with search leads users to leave a site
or application before they have found what they came to look for).
• Incremental search allows users to find deeply buried content by suggesting related or similar
search queries to what the user has already typed. Such content is sometimes hardly findable
at all using delimited search if the user doesn’t suspect its existence.
• Incremental search speeds up the search process by allowing an optimal query formulation to
be submitted by users the first time around; thus eliminating “back-and-forthing” or other tedious
incremental users search processes based on elimination of irrelevant search queries rather
than on finding the right query from the start.

Benefits to the Content Service Operators

Content service operators (large web sites, directories, ecommerce companies, large enterprise
applications, etc.) can leverage the benefits of using advanced incremental search for the following
reasons:
• Reduced costs of hardware and operations of search infrastructure. With incremental search
users tend to use search only once and find their results right away. When using a structured
data search engine, this leads to dramatically decreased load on delimited search systems.
Strctured data search systems like Exorbyte’s MatchMaker are much cheaper to operate and
maintain that a traditional or full-text search engine for instance.
• Dramatically improved control over search and navigation. Search engines and content
management systems are not always easy and flexible to configure. Content service operators
can use an advanced incremental search system to assist the users with fully controllable search
and navigation suggestions before they become confused in searching and navigating the site
with other tools.
• Business driven raking and biasing of search records allows to build interfaces that both
optimize user satisfaction and business outcomes.
• Increased satisfaction generates user loyalty through improvements of users experiences in
search and navigation of the content.

Example: Price Comparison Portal

In one of its first AJAX incremental search implementation on leading German price comparison portal
Billiger.de, Exorbyte was able to collect some interesting data regarding the usage patterns of
SearchNavigator and what changes occurred after its introduction. Billiger was able to report
exceptional results9 given the small investments required by such an implementation. Implementing
SearchNavigator takes 3 to 5 days engineering and a monthly license fee to Exorbyte.

9
Billiger.de’s Managing Director, Lars Bauer discusses Billiger’s use of Exorbyte’s software in the article “Diskrete
Gedächtnisstütze” in e-commerce MAGAZIN in May 2006. See system live at http://www.billiger.de (in
German Language).

www.exorbyte.com
+1 503 616 4007 15/19
White Paper: Incremental Search Revisited

The FlexSearch Interface on L’Tur offering users an impressively simple start: one simple search field

In the case of Billiger.de, the 1st month of implementation saw a 5% increase in the conversion rates of
the site. That’s a significant revenue increase that easily offsets the costs of implementation.
Incremental search also proved a preferred method for digging deep in the multiple layers of the price
comparison portal’s categorical structure for Billiger’s users. The relative use of search versus the
navigation of categories (directory style taxonomy) increased 20% thus showing that users were more
satisfied with this method for product location.

B. Cost and Risk Analysis

As discussed previously, various types of incremental search implementation modes exist. Typical
classifications focus on their impact on the user experience but the table below (Figure 2) offers a
classification that compares various incremental search systems based on user experience benefits as
well as resources necessary to implement and maintain them.

www.exorbyte.com
+1 503 616 4007 16/19
White Paper: Incremental Search Revisited

Cost Matrix of Incremental Search System Types

Incremental Description Key Benefit Key Limitation Cost of Cost of

Search Type Implementation Maintenance

Client-side Client-side User gets Tends to focus Minimal but Medium - the
JavaScript form suggestions from a user on sets of requires much client-side data set
Incremental
suggestions limited set of most popular testing for browser used needs to be
Search based on a fixed normalized queries, not on platform maintained and
(usually limited data set. popular query the depth and compatibility for updated regularly
JavaScript) inputs. breadth of real complex or risk of sending
searchable implementations. users to not-found
content. result pages.

Server-side Client User gets Performance is Medium to High – Minimal – the

Incremental application suggestions from usually insufficient server-side data served in the
communicates full searchable for fluid suggests database or incremental search
Search using with server content usually experience, large search engine system is usually
database or through AJAX matched exactly data sets, and performance is the same as that
flat text files. connection to a to partially typed heavy usage often insufficient served in delimited
database or input. applications. requiring costly search results
traditional search Approximate hardware and pages.
engine matching is out of software upgrades
the question for if at all feasible.
lack of
performance.

Incremental Client Capable of No performance Medium – despite Minimal – the

application handling any or feature impressive data served in the
Search
communicates incremental input, limitation. performance this incremental search
powered by with server displaying any system doesn’t system is usually
Structured through AJAX kind of require a complex the same as that
Data search connection to a suggestions from data integration served in delimited
engine specialized high approximate and client-side search results
performance matching many JavaScript is only pages.
approximate millions of records as complex as its
search engine under 10 interface is rich.
(such as Exorbyte milliseconds
MatchMaker)

Figure 2 – cost-benefit comparison of 3 major incremental search systems.

As shown in the table above, it is easy to see why incremental search systems supported by a
specialized structured data search engine such as MatchMaker are optimally suited to satisfy the most
demanding business and financial constraints.

Optimizing the Cost vs. Feature Richness Dilemma of Incremental Search Interfaces
Incremental search interfaces require greater approximate search capabilities because the very premise
of a search suggestion is to help users guess what alternative content may exist relative to what they
have typed, or are about to type. Such capabilities assume that the system should be able to correct
user and content errors whenever possible. This type of “did you mean” error tolerant search software

www.exorbyte.com
+1 503 616 4007 17/19
White Paper: Incremental Search Revisited

must include facilities such as: algorithmic edit distance calculations, phonetic computations,
aliases/synonym lookup tables, custom matching scripts, and many other refinements all wrapped in
one. Such logic can be developed or configured partially into the higher-end search engines but the
hardware performance and maintenance cost is often daunting. This is why error tolerant structured
data search engines are so well suited to assist the deployment of such applications.

Figure 3 – Positioning map showing the relative cost/benefits of various incremental search approaches

An error tolerant structured data search engine such as Exorbyte MatchMaker is capable returning
pertinent approximate matches over many millions of records and within the short time constraints of
incremental search interfaces. A specialized search engine like MatchMaker requires only a small
fraction of the hardware resources required to upgrade traditional search systems to fit the demanding
performance constraints of incremental search applications. In many cases, it is not even possible to
implement incremental search over a traditional search application because of sluggish response time
and inability to handle approximate matching.

The use of an AJAX incremental search system coupled with MatchMaker offered the benefit of being
capable of running in parallel or “on top of” an existing search application. The index compiled by
the data search engine (MatchMaker) can be directly extracted from the existing search index
(catalog, directory records, etc.) The actual AJAX interface is then implemented on the existing search
web pages by simple addition of a few lines of JavaScript. The back-end systems run completely
independently: should the main search engine fail, the AJAX incremental search system would still
remain operational.

www.exorbyte.com
+1 503 616 4007 18/19
White Paper: Incremental Search Revisited

As expressed by this large body of evidence the costs and risks associated with incremental search are
best mitigated by the implementation of a structured data search engine along with the incremental
search interface. In the case of Exorbyte, monthly fees for its custom incremental search solution based
on the MatchMaker search engine start at around $2,000 per month plus approximately $6,000 of
installation and configuration services. For the smallest operators (below 1M queries per month and
100,000 records) this minimal fixed cost may be too large. In such cases a minimal client side
incremental search will offer some additional benefits for an affordable cost. In all other cases,
incremental search systems require a data matching server capable of handling high performance.

IV. Other applications of AJAX Incremental Search Powered by Data

Matching Servers
Although AJAX incremental search is an ideal fit for search interface destined for structured data
(databases, directories, ecommerce, etc.), the use of approximate data matching technology such as
Exorbyte MatchMaker opens possibilities for other types of search applications. Most unstructured
data search solutions involve searching web content such as web pages and other documents or
searching through semi-structured data (price comparison XML feeds, EDI transactions, email messages,
etc.)

It’s interesting to note that many full text search engines for instance have evolved to include a
structured index accompanied with all sorts of dictionaries and lookup tables for spellchecking,
taxonomies, and other search-related services. Each of these data sources can be leveraged in some
sort of incremental search interface if it fits with the search engine’s purposes.

The use of AJAX interfaces is only starting to take hold in web interface architectures. Implementing
fancy new search interfaces like AJAX incremental search is often a question of balancing quality of
search experience with existing or future performance limitations. Implementation of approximate high-
performance data matching servers will become a necessity if such systems need to scale and remain
optimal in search quality.

Dan Nicollet – Exorbyte Inc. 2009

All names and logos are trademarks and/or registered trademarks of their respective owners.

www.exorbyte.com
+1 503 616 4007 19/19