virtual techdaysINDIA │ 18-20 august 2010

FAST Search for SharePoint 2010Allirajan Ramachandran │ Technology Specialist, Microsoft Corp

arajan@microsoft.com

Cut Costs with a Unified Infrastructure

Rapidly Respond to Business Needs

Deliver the Best Productivity Experience

Microsoft SharePoint 2010The Business Collaboration Platform for the Enterprise and the Internet

Communities

Search

Sites

Composites

ContentInsights

One platform, Multiple Service Opportunities

User Interface is visual and actionableVisual and conversational interaction with precise control

Built on SharePoint Search CenterLeverages all of innovations in SharePointOpen Web Parts, Federation, query suggestions, related queries, Did you mean?

Visual results connects users with contentThumbnails for Word and PowerPointVisual Best Bets highlight premium content Preview in browser without leaving the results

Deep Refinement

Thumbnails

Previews

Sort on any field

Similar Results

Visual Best BetsVisual cues to highlight essential content

Built on SharePoint KeywordsMatches keywords and synonyms that are contextually relevant to users. Include banners, videos, external websites.

Easy and quick to setupPoint and click setup for site admins. Set and forget with content expiration dates . Web Parts allow for easy page customization

Visual Notification

Web Part Flexibility

Deep RefinementEnables precise control of results

Contains exact countsLeads to discovering non-obvious relationships, key data trends, and deep analysis of your content

Enables conversational experience You will never miss any content; enabling better findability and exploration across the entire result set

Exact Counts

Sorted by frequency

Provides a sorted viewEach refiner is sorted by frequency, from highest to lowest, indicating the importance of each term

Customize search to meet your business needs

Key ingredients to a great customized search experience

Search in the language of your business

Deliver results that are contextually relevant

Tune relevancy to meet diverse needs

Process content with advanced linguistics

Customize the user experience to build engaging applications

Type Doc ID Title Author Date Size Keywords Companies Expertise Industry ... Body Text

123 Sales Forec… John Doe 2010-04-15 386 KB sales; pipe… Contoso; Tail.. Strategic Con.. Financial Ser… … IT Market Gr…

456 … … … … … … … … … …

963 … … … … … … … … … …

Map metadata to Managed Properties Automatic association of metadata to content

Expertise

Strategic Consulting

Market Analysis

IT Implementation

…

Industry

Financial Services

Manufacturing

Technology

...

Crawled Properties

Managed PropertiesIndex Schema:

Companies

Contoso

Tailspin Toys

Woodgrove Bank

…

Mapped in Central Admin Or PowerShell

Crawled Properties Standard document metadata discovered by the crawler or extracted from the full text by the FAST Content Processing Pipeline

Managed Properties Map one or more Crawled Properties to a single field. Enables sorting, refinement, relevance tuning and fielded searching

Title

Sales Forecast

Any data can befound and mapped

What can I do with a Managed Property?Metadata quality is critical to a good search experience

Precise hit counts in deep refiners are

computed across the whole result set.

And many more…

Concepts

Products

Companies

File Formats ,

Metadata is also used for relevancy tuning, multi-

level sorting and advanced search

Enables deep refinementMakes search conversational, guiding users to navigate and refine, while summarizing the results that are found

Enables precision relevancyManaged properties are also used for relevancy tuning & ranking, multi-level sorting, advanced (or fielded) search

How does it work?

Built on a SharePoint List or custom extractorEdit the Search Center Results PageModify the shared web part by adding tags to the refinement panel XMLCreate your own labelsSave and Publish

Custom Collections

Add refiners to user interface

Context mattersUsers need to access multiple types of content

HRLegal

Finance

Depends on role, location, responsibility and task. This can change day to day, or hour to hour.

Marketing Sales R&DCustomerSupport

ProfessionalServices

ManufacturingOperations

. . .

Ente

rpri

seC

onte

nt

Deliver results that are contextually relevantwith search that can understands your business and role

”What should I know about selling ERP?”

- Alan Brewer, Sales Lead

”What should I know about implementing ERP?”

- Renee Lo, Consultant

Role-specific relevance

Business drivenrefinement

Targeted Best Bets / Visual

Best Bets

Tune relevancy to meet diverse needsA flexible solution for your organizations, groups and individuals

Optimize Relevancy for broad intranet use

Query results with the default relevancy

”I want to know about my customer Woodgrove Bank and customers in Financial Services"

- Alan Brewer, Sales Lead

New Default Sorting

Promotes relevantResults

Quickly tailor relevancy models Deliver the right results to the right people by creating new Rank Profiles

”I want to get right to the technical documents"

- Renee Lo, Consultant

Documentation, RFPs and SOWs are now promoted Same

Results different order Users can select rank profiles in the sort by box or create their own default views by modifying the web part

Rank ProfilesTune relevancy without impacting the default algorithm

Quality Also known as static rank, consists of multiple managed properties including site, URL depth (preference for shorter URLs), and relative importance of links to this document.

Authority Applies when the query word falls in the link or anchor text.

Query Authority

Maps the popularity of a document, or the click-through rate when documents are clicked as a result of a query

Freshness Increases the relevancy if a document was recently created or modified, based on the last modified property.

Proximity Applies to where query terms fall and how close they are to each other within a document

Context Increases the rank of a document if the query term is a managed property associated with that document

Managed Property

Effects relevancy when a managed property contains a specific value, such as Woodgrove Bank or Financial Services

Out of the box relevancyTuned for great general productivity experience, relevancy improves with click-throughs and link text analysis.

Extend the default algorithmsCreate new default relevancy models. Blend static and dynamic ranking parameters to instantly improve search results.

How to create a Rank ProfileIT Pros are empowered to create new profiles quickly

Rank Profiles created in PowerShell by extending the default relevancy algorithm…

… and are exposed in the user interface by modifying the

sorting web part.

Breaks down content to the smallest addressable chunks to build meaningUnderstands file encoding, data formats, and written languages Supports 400+ file formats, 80+ languages

Process content with advanced linguisticsAutomatic and detailed analysis creates a great search experience

Map Crawled Properties

Maps all of the metadata that was discovered by the various pipeline stages

Web Link Analysis Analyzes documents for hyperlinks extracting anchor text which reinforces the authority ranking of a document.

Document Vector Creates a unique representation of a document that reflects important terms and frequency of occurrence. Used to find similar documents.

Date and Time Normalization

Converts dates and times to a standard representation, to handle locale specific representations. For example, knows that 14-Mar-10 is equivalent March 14, 2010.

Entity Extraction Finds terms in the content and maps them to predefined categories. Out of the box support for People, Companies and Locations, but can be extended to any category.

Lemmatization Finds the root of a word for a given language. For English it maps run, runs, running and ran back to a single lemma. Understands language specific grammar and context.

Tokenization Apply the language specific rules for identifying words, concepts, idioms and phrases. Also applies custom word breakers found in part numbers or telephone numbers.

Language Encoding and

Detection

Identifies the native written language and locale specific encoding so that the proper dictionaries can be used by the tokenization and lemmatization stages

Format Conversion

Extracts plain text from multiple file formats, encodings, and applications

FAST Content Processing Pipeline

FQL provides a robust and expressive query language

Wildcard support - *, ?Numeric Data types (Integer, Float, Decimal, Datetime)

OperatorsDirect field access (e.g., title:othello, author:shakespeare)Numeric (COUNT, RANGE, <, <=, >, >=)Boolean (AND, OR, ANY, NOT)Rank (RANK, XRANK)Proximity (NEAR, ONEAR)

String (operator support for strings) Boundary (starts-with, ends-with, equals)Filter

Robust query languageUse FAST Query Language (FQL) for precise query development

FAST Search Server ArchitectureCommon platform scales with your business need

One search platformwith a single UI framework, common APIs and developer experiences across products

Common connectorswith a consistent framework for capturing and indexing your information

Unified administrationwhich allows you to manage FAST Search and SharePoint in a centralized admin console

User Experience

SharePointSearch

FAST Search for SharePoint

Information

FAST Search Server 2010Summary of architectural components

Custom Front-End

OpenSearch or Other Sources

SharePoint Front-end

People Search

Qu

ery

Obje

ct M

od

el

Query and

Result Processin

g

Security AccessModule

SearchCore

Indexing

Federation Object Model

Query Web Service

AdvancedContent Processin

g

Linguistics

WebLink

Analysis

Connectors

• Web Crawler

• JDBC

Connectors

• SharePoint• File

Traverser• Web • BDC• Exchange• Notes• Documentu

m

Microsoft System Center Operations Manager

Monitoring Services

Administration and Schema Object Model

Site Collection Level Admin UI

• Keyword Management• User Context

Management• Site

Promotion/Demotion

PowerShell

• Schema configuration• Admin configuration• Deployment

configuration

Central Administration UI • Property mapping• Property extraction• Spell-checking

FAST Server(s)

SharePoint Server(s)

Other Server(s)

Content

Secure, unified access to informationIndex or federate with content, applications, and services

Access content securelywith a full-featured content crawler and both index- and query-time trimming of results

Federate queriesusing OpenSearch to quickly access existing search indexes and online information services

Extend your reachusing Business Connectivity Services to access your content sources and business applications

OpenSearch Federation

Indexing Connectors

EnterpriseContent

Business Application

s

Information Services

User Experience

Search Index

Websites and Intranet

File Shares

IBM Lotus Notes

EMC Documentum

SharePoint Server

Exchange Public Folders

AD & LDAP Profiles

Etc …

Connect to all of your enterprise contentExtend your reach with Business Connectivity Services

Build custom connectorsUse SharePoint Designer to configure data model and connect to SharePoint. Connect to WCF services, or create your own .NET assembly connector with Visual Studio.

Quickly connect to contentUse a consistent framework to quickly connect both inside and outside of SharePoint including content management systems, web services, databases, and line of business systems

Simplified, powerful administrationA high-end enterprise search solution that’s easy to deploy and manage

Deploy easilyusing wizard-driven installation, a topology designer, and native support for 64-bit virtualization

Manage efficiently with full support for Microsoft System Center and PowerShell scripting to automate tasks

Streamline administrationwith a simplified admin console that helps you manage search services across your enterprise

FAST Search for SharePoint Scaleout

Content Volume

Query Volume

Scale-out multiple “dimensions”

Query VolumeContent VolumeIndexing freshness

Redundancy optionsSearchIndexing

Performance targets*30M Docs/node50 QPS/node35 docs/sec

*Depends on content and hardware specifics

Search and Indexing

Crawling and Content

Processing

Query and Result

Processing

Back-end with extreme and flexible scale out options

No theoretical upper bounds!

virtual techdaysINDIA │ 18-20 august 2010

RESOURCES

www.sharepoint.microsoft.com

www.microsoft.com/enterprisesearch

Channel9.msdn.com/posts/matthijs/fast-search-for-sharepoint-2010-capabilities-deep-dive

virtual techdaysTHANKS│18-20 august 2010

Thank You