open search david wolber. overview proliferation of digital libraries metasearch and fixed lists of...
Post on 18-Dec-2015
217 views
TRANSCRIPT
Open Search
David Wolber
Overview
Proliferation of Digital Libraries Metasearch and Fixed Lists of Sources Open Search Architecture PublishMe for P2P knowledge Sharing Webtop Metasearch Clients
Contributors
Michael Kepe Igor Ranitovic Iman Sadreddin Senior Team ’03 Ken Chong Rudd Stevens Colin Bean Tim Chan Julian Chan Pooja Garg
Information Source Explosion
Google, Amazon APIs Internet Archive Technorati– The World Live Web Domain Specific:
– ACM Digital Library for CS– Lexis-Nexis for law– MLA for literature
End-User Created Digital Libraries
Personal Web (shared Google desktop)
Personal Web Neighborhood
Topic-Specific Personal Crawlers
Ordinary people creating search engines as easily as web pages
2nd Degree
1st Degree
Nth Degree
PersonalWeb
Subsets of the Web
Motivation for Small, Independent Subsets of the Web
Avoid information being channeled through a single portal: Googleopoly
Google does no evil, but…– Censorship in China– Creeping level of commercialization– Unregulated manipulation of secret ranking
algorithms (see PageKing case)
Other media is lost, this is the last frontier
Little support for using multiple search engines
Overview
Proliferation of Digital Libraries Metasearch and Fixed Lists of Sources Open Search Architecture PublishMe for P2P knowledge Sharing Webtop Metasearch Clients
Metasearch
Help users discover and use digital libraries
Send queries to multiple, selected search engines
filter, process, and unify results
A9.com – Amazon’s metasearch
Web Services Basis
server html
server softwarexml
server
html
Web Page Model
Web Service Model
How does metasearch evolve?
New Digital library
How does metasearch evolve?
New Digital library
Metasearch clients discover it
How does metasearch evolve?
New Digital library
Metasearch clients discover it Metasearch
Programmers write adaptor/scraper
How does metasearch evolve?
New Digital library
Metasearch clients discover Metasearch
Programmers write adaptor/scraper
User can access within metasearch
SLOWLY…
Overview
Proliferation of Digital Libraries Metasearch and Fixed Lists of Sources Open Search Architecture PublishMe for P2P knowledge Sharing Webtop Metasearch Clients
Goal: Automate the Process
Metasearch engines should provide users with up-to-date lists of existing digital libraries
Digital libraries should be able to register and be made immediately available to all Metasearch clients.
Metasearch and Library development is independent.
What is Necessary?
Standard Search API– So Metasearch clients can use polymorphism to access
sources.
for each source s in sourceList {searchEngine.endPointUrl = s.endPointUrl;resultList +=
searchEngine.keywordSearch(keywords)}
Search API Registry
– Metasearch clients can get dynamic list
Web Service Standards
WSDL – Web Service Description Language
SOAP – Simple Object Access Protocol
UDDI – Universal Description, Discovery, and Integration
Standards on top of Web Services
WSDL, SOAP, UDDI basis for standards in many domains.– e.g., MS initiated for securities information
providers
Businesses agree on a standard, then client applications can use polymorphism and new businesses can register services.
In this case, we want cross-domain standard.
Open Search Architecture
Open Search Protocol (OSP)– Cross-Domain: Search-related services– Not just keyword search, but citations, authorOf, etc.
Open Search Registry– Based on UDDI– Can add customization, e.g., parsing to find out which
search operations are implemented.– Web and web service access
Open Search Architecture
OSP metasearch clientssource list
Register service
OSP-Conforming Libraries
OS Registry
User Can Choose Sources
Open Search Protocol
Keyword search
Citations (inward links, outward links)
AuthorOf and other associative operations…
Metadata object results based on Dublin Core
Restriction object for “advanced search” stuff
Publishing a Library
• Access OSP WSDL Specification from webtop.cs.usfca.edu
• Generate code in language of choice
• Implement the search operations for the digital library
• Deploy the service
• Register with Open Search registry
Deploying an Open Search Lib.
programmer 1. OS wsdl
wsdl2java
2.wsdl
3. skeleton code
Open Search
information
Registry
Library server
4. deployed service
5. registration info
Wrapping a Library
Custom search API, e.g., Google API
Open Search Wrapper
Metasearch Client
1. OSP Query 4. OSP Result
2. Custom query
3. Custom Result
Located on 3rd party server
Wrappers Developed at USF
Google Amazon (sort of) Internet Archive Technorati Feedster
Overview
Proliferation of Digital Libraries Metasearch and Fixed Lists of Sources Open Search Architecture PublishMe for P2P knowledge Sharing Webtop Metasearch Clients
PublishMe
Like Google Desktop, but shared.
Periodically updates inverse index and linkbase on PC
Deploys Web Service on User’s PC
Auto-Registers with Open Search Registry
Metasearch with P2P Knowledge Sharing
WEBTOP
Integrating Global and Personal Libraries
Motivation for Sharing Personal Webs
People create knowledge everyday when they bookmark, annotate, link, organize, and synthesize.
Communication is a separate step which often doesn’t happen
Experts Collaborative Work
Motivation for Sharing Personal Webs
Computers are designed using our brains for a model
Knowledge creation and dissemination separate
Explicit effort required to communicate Just as we model our word processors on
paper.
Additions to OSP for P2P
GetFile
OnLine(ip)– Handles user starting up– Dynamic IPs
OffLine
But What About PRIVACY?The Big Question:
How much of the information hidden
within your personal web is hidden due
to privacy concerns?
I Want you to be a Search Engine!
Overview
Proliferation of Digital Libraries Metasearch and Fixed Lists of Sources Open Search Architecture PublishMe for P2P knowledge Sharing Metasearch Clients
Goal: Implement Vannevar Bush’s Association Trails
View a document/thing in context
History of an idea
Thinkmap-like Interface
Association Types
Outward links Inward links Similar-Content links People Links
– author, people referenced in paper Domain-Specific links
– law citations– movie-actor
Associations specified by Annotators
Webtop Tree View webtop.cs.usfca.edu
Expanding a Tree
• Bird’s Eye View
• Local/Web files integrated
• Follow different Associative Trails
• Ins of Outs of Ins, etc.
• Siblings
• Weird though, as ins and outs both expand right
Webtop Side Panel View
Project Status
Too many bugs, Dad
Future Work
Open Search Protocol– In-depth study of existing search APIs– Provide Rest alternative to SOAP
Metasearch development– Complete and refine existing clients– Dream up new ones
Thinkmap Graph Automated Source Selection and Reputation System Page Ranking
Initiate grass-roots involvement
Future Work: Documents and Things
resourceassociationsannotations
document
html word pdf
person
film book
creative work
Stop talking about Webtop daddy!
webtop.cs.usfca.edu