Building

specialized applica/ons using Solr; Migra/on from FAST ESP

Rahul Agarwalla Head of Interna/onal Business Uchida Spectrum Inc.

2011 Uchida Spectrum, Inc. All rights reserved.

Uchida Spectrum Overview

SoDware License Business

1995 ~ So)ware License Sales License Management Repor:ng License Procurement System License Adjustment Consul:ng

Network Technology Services

1997 ~ Network System Consul:ng Services
Ac:ve Directory Network Exchange Messaging Network

Enterprise Search Business

2002 ~ Enterprise Intelligence Applica:on
SMART InSight G2 Enterprise SMART InSight G2 Professional FAST ESP Lucene/Solr Lucid Works Enterprise

License Management System Consul:ng

So)ware Management Server Share Point Portal Server Websphere Portal Server

Search PlaRorm Consul:ng & Support

Portal System Consul:ng

2011 Uchida Spectrum, Inc. All rights reserved.

Page-2

Page-2

Some of Uchida Spectrums customers

2011 Uchida Spectrum, Inc. All rights reserved.

Page-3

Page-3

SMART/InSight History

Customers in Japan, China & India:

2 of top 3 Japanese car manufacturers Top consumer electronics company Large nancial ins8tu8ons Chinas biggest eCommerce rm

2005: SMART InSight 1.1 2004: PlaRorm for custom solu:ons 2003: FAST Alliance

2011 Uchida Spectrum, Inc. All rights reserved.

Page-4

Page-4

What is todays buzz word?

Smart Phone

Extreme scalability Flexibility & Extensibility Feature rich search

2011 Uchida Spectrum, Inc. All rights reserved.

Page-5

Page-5

What I learnt from the Japan catastrophe

2011 Uchida Spectrum, Inc. All rights reserved.

Page-6

Page-6

The power of community

Japanese Government [Closed/big brother]

Japanese People [Open community]

Slow, behind the curve Legacy/CYA Confusion

Quick response Disclose / Share Prac:cal Impact

Driver of innova/on

Power shift

2011 Uchida Spectrum, Inc. All rights reserved.

Page-7

Page-7

Lessons from FAST ESP Migra/on: advantage LWE/Solr Key Issues:

1. 2. 3.

Smaller record and index size enable faster index maintenance # of records per node: rule of thumb 10m vs. 2m Licensing & Maintenance Cost: less than

Scalability: 5x Cost Performance: 10x High Flexibility Lower Opera/ons Cost Faster Innova/on

2011 Uchida Spectrum, Inc. All rights reserved.

Page-8

Enterprise Search expecta/ons Big data scale Security is important Disparate data: geography, systems, languages, format, structures KM is good to have, databases are cri:cal Support dierent users & usage: department, role, tasks High recall

2011 Uchida Spectrum, Inc. All rights reserved.

Page-9

Page-9

Lessons from FAST ESP Migra/on: Filling the gaps

Security

ACL security: complex requirements File System: le & folder level control CRM/ERP : Keeping ACLs up-to-date

Content aggrega/on

Connectors Normaliza:on Open source op:ons for ESP pipeline

Openpipeline Pypes

2011 Uchida Spectrum, Inc. All rights reserved.

Page-10

Building specialized applica/ons: Content fusion Content fusion from disparate data:

Single index integra:on Modeling of content rela:onships is essen:al

2011 Uchida Spectrum, Inc. All rights reserved.

Page-11

Virtual integra/on based on search

Applica/on layer Content store

Big table, at index

Content sets and inter-rela/onships

Search Index Search Index Search Index

2011 Uchida Spectrum, Inc. All rights reserved.

Page-12

Virtual integra/on based on search2

Append Pipeline

Tagging Pipeline

Data transforma:on: -key:key, key:value, eld names Query & Result transforma:on Boos:ng / Relevancy algorithm Security Mul:-Language support Federa:on & mashups

. . . . . .

Boos:ng

Transform

LWE Adapter

SolrAdapter

Other

Search Index LWE

Solr

Result Pipeline

Query Pipeline

Search Service

Content Security

2011 Uchida Spectrum, Inc. All rights reserved.

Page-13

Building specialized applica/ons: Personaliza/on

Applica/on ow depends on the task Data Personaliza/on increases produc/vity

SMART InSight approach: Task based UI

Schema independent widgets for analy:cs & visualiza:on Portalized Personalized: widgets, func:ons, content, elds

2011 Uchida Spectrum, Inc. All rights reserved.

Page-14

Knowledge Center: made possible by Solr

Scalability and low TCO gives us ability to build new features

Knowledge Centre has logs of all user ac:vity in SMART InSight This would be too costly with a commercial Search Engine and would not be feasible in a database

Using this rich data we can:

Prole users, groups and networks Personalize Recommenda:ons Create social ranking algorithms Usage analy:cs

2011 Uchida Spectrum, Inc. All rights reserved.

Page-15

Overview of SMART InSight for Automo/ve

Task based UIs
NHTSA Internet Page EDR Repair Dealers Claims PLM Specs CAD Internal Management & Security SA SA SA SA Widgets

Ajax Portal

Personaliza/on

Benchmarking

Virtual Integra/on Framework

Contents Set

Convergent Knowledge Framework

Knowledge Centre Recommend Proling Metadata Analysis Knowledge Log

Data Chain

Design Engineering Claims

Parts Catalog

Early Defect Warning

Content Model

Claim Analysis

2011 Uchida Spectrum, Inc. All rights reserved.

Page-16

: Interac/ve Click Log Analysis System

> $50 Billion sales / year > 800 Million Items > 370 Million Users Billions of clicks per day
Access Log

Solr Hadoop

Solr, Hadoop + SMART/InSight G2

xxxxxxxx

Xxxx Xxxx Xxxx Xxxx xxxx

2011 Uchida Spectrum, Inc. All rights reserved.

Page-17

: Global Research Community

Top Academic Ins/tutes: Faculty, Research Fellows & Post graduate students Govt. Departments & Corporate R&D Scien:sts and researchers

Research Discovery & Collec/ve Intelligence (Knowledge Centre)

Broadcast Search

> 270 content sources: Socie/es, Associa/ons, Publishers & Open IEEE, ACM Elsevier, Wiley, Springer

Dynamic Result Merging

Solr
2011 Uchida Spectrum, Inc. All rights reserved.

Real /me indexing

Page-18

Demonstra/on

2011 Uchida Spectrum, Inc. All rights reserved.

Page-19

Contact Details

Rahul Agarwalla Head Interna/onal Business rahul@spectrum.co.jp www.spectrum.co.jp

2011 Uchida Spectrum, Inc. All rights reserved.

Page-20