how to build an awesome search app - marklogic · data categorization, envelope structure query...

HOW TO BUILD AN AWESOME SEARCH APP Stu McLean, Ph.D., Principal Consultant, Public Sector - Civilian, MarkLogic Ganesh Vaideeswaran, Senior Director, Development, MarkLogic

SLIDE: 2

Agenda Components of a search application

Deep dive

SLIDE: 3

SLIDE: 5

SLIDE: 6

SLIDE: 7

SLIDE: 8

SLIDE: 9

SLIDE: 10

SLIDE: 11

SLIDE: 12

SLIDE: 13

SLIDE: 14

SLIDE: 15

SLIDE: 16

SLIDE: 17

Awesome Search for the Enterprise

WHAT MARKLOGIC PROVIDES

360° VIEW

SLIDE: 19

NBCUniversal – SNL40 App

Smart Content

Enriched and targeted content

Constantly tuned recommendations

> 5,550 sketches

SLIDE: 20

Talent Kristen Wiig

Acted in

Episode 4 Anne Hathaway and

Killers

Part of Played

Character Maharelle Sister

Season 34

Segment The Lawrence Welk Show

Aired on

Date 10/4/08

Acted in

Includes

Part of Has

Characteristic Tiny hands

NBCUniversal – SNL40 App

Smart Content: Enriched and targeted content Constantly tuned recommendations

SLIDE: 21

Fairfax County – Mapping Crime Data

Data Variety

500,000 GIS data points

Service Calls

Source Variety

800 GIS layers

Structured & Unstructured

SLIDE: 23

Mitchell1 – Streamlined Auto Repair Information

MITCHELL1 PRODEMAND

REPAIR INFO

EXPERT ADVICE

REPAIR ORDERS

Structured & Unstructured Data

Sophisticated Search

Multi-device Delivery

SLIDE: 25

Needs for an Awesome Search App

Relevance

Snippets

Highlight

Auto Complete

Spell Suggest

Synonyms

Structured & Unstructured

Multi-Model approach

Alerts

Transactions

Security

Real-time

Languages

Configurability

Development Platform

PLATFORM

Context

Presentation

SLIDE: 26

Aboutness – The Key to Successful Search What is this document about?

What is this submitted query about?

What is the user’s activity about?

What are the responses about?

SLIDE: 27

Implementing Aboutness in MarkLogic

Data What is this document about?

SLIDE: 28

SLIDE: 29

Query What is this query about?

SLIDE: 30

SLIDE: 31

Context What is the user’s activity about?

SLIDE: 32

Context

SLIDE: 33

Context

Results What are the responses about?

SLIDE: 34

Context

Results

SLIDE: 35

Implementing Aboutness in an Awesome Search App Data

Categorization, Envelope Structure

Query as a Dialogue

Context

Building an Information Resource

Results

Relevant Response

SLIDE: 36

Implementing Aboutness in an Awesome Search App Data Categorization, Envelope Structure

Query as a Dialogue

Context

Results

Relevant Response

SLIDE: 37

Domain Knowledge ‒ Tools for Establishing Aboutness Taxonomies – vocabulary of the organization

Ontologies – Tie your documents to relevant data and documents

Thesaurus

Organizational units, people, products

Public vocabularies – Geographic, DBPedia

SLIDE: 38

Terms to look for in your documents Taxonomies

SLIDE: 39

Taxonomies and Reverse Queries Categorization ‒ Finding Aboutness

Public taxonomies

Getty – art and architecture

World Bank – humanitarian projects

Library of Congress – subject headings

National Library of Medicine – medical headings

Domain taxonomies

Best Buy

Google

SLIDE: 40

Where does Aboutness reside?

title:

abstract:

glossary:

Expert searchers compose carefully crafted searches with domain knowledge

The rest of us hope for the best

Categorizing Your Documents

SLIDE: 41

Turn a Taxonomy Entry into a Query <cts:or-query xmlns:cts="http://marklogic.com/cts"> <cts:near-query distance="5"> <cts:near-query distance="5"> <cts:element-word-query weight="10"> <cts:element xmlns:_1="http://marklogic.com/solutions/obi/source">_1:title</cts:element> <cts:text xml:lang="en">Non-Government</cts:text> <cts:option>case-insensitive</cts:option> <cts:option>punctuation-insensitive</cts:option> <cts:option>stemmed</cts:option> </cts:element-word-query> <cts:element-word-query weight="10"> <cts:element xmlns:_1="http://marklogic.com/solutions/obi/source">_1:title</cts:element> <cts:text xml:lang="en">Bond</cts:text> <cts:option>case-insensitive</cts:option> <cts:option>punctuation-insensitive</cts:option> <cts:option>stemmed</cts:option> </cts:element-word-query> </cts:near-query>

Implementing Categorization

Reverse Query Categorization

Reverse Queries

Ingest mydoc.xml 1 MB document Save as

/tmp/mydoc.xml in collection tmp

Reverse Queries

In-memory copy of abridged document (10% = 100K)

Reverse Queries

cts:search(fn:doc(), cts:and-query(( cts:collection-query("topicQueries"), cts:reverse-query($reducedDoc) )), "unfiltered" )

Reverse Query Categorization ‒ Scoring

Reverse Queries

/tmp/mydoc.xml

Reverse Query Categorization ‒ Scoring

Reverse Queries

/tmp/mydoc.xml

for $q in $reverseSearchResults let $score := cts:fitness( cts:search( fn:doc(), cts:and-query(( cts:document-query($myURI), cts:query($q/query) )), ("score-logtf","unfiltered") )

SLIDE: 49

Envelope Pattern ‒ Know Your Data Homogenized Variant Data Types

Remain Schema-Agnostic, Load As-Is

Annotate Your Data

Building the Envelope

Ingested Content

Metadata

Category Annotation

SLIDE: 51

Envelope ‒ Know Your Data Standardize ingested documents of multiple types Common names may be different (e.g. Title, Heading, Subject)

Normalize your searchable elements into a common format

Multiple versions of the same document (e.g. Translations)

Preserve artifacts

Indexed search on metadata (facets)

Full-text search on content

SLIDE: 52

Query Query as a Dialogue

Context

Results

Relevant Response

SLIDE: 53

What is the Query About? Establish Aboutness Through Dialogue

Type-ahead

Match Lexicons

Match Search History (user, global)

Background Search

Phrase Identification

Object Matching

Boost Queries

SLIDE: 54

What is the Query About? Establishing Aboutness Through Dialogue (Cont’d)

Correction and Confirmation

Spell Check

Thesaurus Expansion

Semantic Expansion

Drill Down

Follow Up Search Box

Facets

Constraints on the Fly

declare function get( $context as map:map, $params as map:map ) as document-node()? { let $suggestList := ("person", "country", "organization", "project", "region", "topic") let $partial := xdmp:url-decode(map:get($params, "partial-q")) let $suggestions := map:new(( for $s in $suggestList let $term := fn:concat($partial,"*") return map:entry($s,search:suggest( $term, $options, if ($s eq "person") then 20 else 3 ))) return document {xdmp:to-json($suggestions)} };

Extending Suggest REST API extension Search multiple lexicons for

each request Type-ahead must be fast Resist temptation to put

multiple suggest calls in the app

SLIDE: 56

Query Expansion ‒ Background Search Finding phrases when there are no quotes

Approach 1 SPARQL - Match phrases in metadata, taxonomies

Put the matched text into a weighted custom constraint (e.g. phrase: )

OR the phrase: constraint with the full query

Approach 2 Build NEAR queries using pairs of nearby terms

OR the phrase and boost the weight on the NEAR query if matched

SLIDE: 57

Example – Constraints on the Fly Selected documents share common elements

Elements not known in advance but can be used as in-app facets

Set as constraints in the submitted options

SLIDE: 58

Example – Selecting from Feature Fields

SLIDE: 59

Example – Selecting from Feature Fields

SLIDE: 60

Query as a Dialogue

Context Building an Information Resource

Results

Relevant Response

SLIDE: 61

Aboutness – The Context Behind The Query Document

LexisNexis WestLaw Medline USPTO

SLIDE: 62

Aboutness – The Context Behind The Query Document Facts

WikiPedia CIA World Factbook

SLIDE: 63

Aboutness – The Context Behind The Query Document Facts Information

WikiPedia CIA World Factbook

Google Bing

Reference Data – Link or Load External Data Real Time Read

Reference Data – Link or Load External Data Ingest Retrieval Embedded Data

Reference Data – Link or Load External Data Copied Reference Data Embedded Links

additional-query SPARQL

SLIDE: 67

Query as a Dialogue

Context

Results Relevant Response

SLIDE: 68

Planning Your Data For Results When to index metadata

Facets

Sortable

Lexicon dropdowns

Document joins

When not to index

Wildcarding

Search:search constraints

SLIDE: 69

Results ‒ Focus On Relevance Boost the Document: Quality

Boost the Element: Term Weighting

Boost the Term: Query Weighting

xdmp:document-insert( "/example.xml", <a>aaa</a>, xdmp:default-permissions(), xdmp:default-collections(), 10 ) xdmp:document-load( "http://myCompany.com/file.xml", <options xmlns="xdmp:document-load"> <uri>/documents/myFile.xml</uri> <repair>none</repair> <permissions>{xdmp:default-permissions()}</permissions> < <format>xml</format> <quality>10</quality> </options>)

Boost the Document Set quality xdmp:document-insert xdmp:document-load

SLIDE: 71

In the database configuration

Select Word Query from the menu

Under the includes tab add your ranked element with its weight

Boost the Element

SLIDE: 72

Boost the Term With no Query Weighting the MarkLogic Companies document appears first

SLIDE: 73

The Boost-Query A conditional secondary query that increases the weight of results matching both

criteria

The boost-query pushes the MarkLogic World document to the top

SLIDE: 74

Aboutness – The Key to Successful Search What is this document about?

What is this query about?

What is the user’s activity about?

What are the responses about?

SLIDE: 75

Rendering Awesome Apps

SLIDE: 76

how to build an awesome search app - marklogic · data categorization, envelope structure query...

Documents

automating the game with semantics - events |...

introduction to marklogic 4.2

inside marklogic server › ... ›...

pega marklogic connector

marklogic cluster monitoring application - github...

magnetic - query categorization at scale

marklogic connector for hadoop developer’s...

javascript reference guide - marklogic · pdf filemarklogic...

marklogic server on amazon ec2...

sql data modeling guide - marklogic - marklogic 9 · pdf...

maximizing your marklogic and java investments ·...

madrid! query categorization

sql data modeling guide - marklogic · 2017. 8. 17. ·...

java application developer’s guide - marklogic ·...

continuous delivery and marklogic

query console user guide - marklogic - marklogic 9 · pdf...

inside marklogic server - marklogic developer … paper...

inside marklogic server - odbms.org · inside marklogic...

concepts guide - marklogic server overview of marklogic...

reference application architecture guide - marklogic ·...