ensuring government is only one search away...13 google confidential one of the fundamentals: index...
TRANSCRIPT
![Page 1: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/1.jpg)
Ensuring government is only one search away
Implementing the Sitemap Protocol
![Page 2: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/2.jpg)
Ensuring search “just works”
Prioritizing index comprehensiveness
![Page 3: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/3.jpg)
3 Google Confidential
Google’s relationships with government
ContentPartnerships
Government
Publisher
VendorCustomer
Distributor
Regulator Corporate citizen
Enterprise
Policy
![Page 4: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/4.jpg)
4 Google Confidential
Sitemaps.org
An open, industry standard for search engine crawling
![Page 5: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/5.jpg)
5 Google Confidential
Getting to “It just works”
• Google’s conservative side:
– The famously spare home page: “32 words”
– But also the slow evolution in the presentation of search results
– Universal Search: Years in the making, still in gradual iteration
![Page 6: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/6.jpg)
6 Google Confidential
More aggressive oneboxing
![Page 7: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/7.jpg)
7 Google Confidential
Other formats interleaved in results
![Page 8: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/8.jpg)
8 Google Confidential
Getting to “It just works”
• Google’s conservative side:
– The famously spare home page: “32 words”
– But also the slow evolution in the presentation of search results
– Universal Search, years in the making
• The ultimate test: “It just works”
– With a focus on the majority of users and the majority of searches
![Page 9: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/9.jpg)
9 Google Confidential
One of the fundamentals: Index comprehensiveness
• One way to ensure search just works is to provide users search across the broadest possible corpus of content:
– Broader than they would imagine finding useful
– Example: Google Scholar
![Page 10: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/10.jpg)
10 Google Confidential
Inclusive and comprehensive
• Commercial publishers and scholarly societies• Public A&Is and open access journals• Open web sources and institutional repositories• All formats, all languages
![Page 11: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/11.jpg)
11 Google Confidential
![Page 12: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/12.jpg)
12 Google Confidential
![Page 13: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/13.jpg)
13 Google Confidential
One of the fundamentals: Index comprehensiveness
• One way to ensure search just works is to provide users search across the broadest possible corpus of content:
– Broader than they would imagine finding useful
– Example: Google Scholar
• Relentless push to expand the index:
– Not just all books, but all library catalogs
– Not just all news sources, but all newspaper archives
– Not just earth, but the weather, sky and visible universe
![Page 14: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/14.jpg)
14 Google Confidential
Other fundamentals
• Speed
• Ranking
• Anticipating user intent, e.g. personalization
![Page 15: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/15.jpg)
15 Google Confidential
Behind the white curtain
• “Google Keeps Tweaking Its Search Engine,” New York Times, June 6, 2007:
– “[Google] is also among the world’s biggest teases. Millions of times a day, users click away from Google, disappointed that they couldn’t find the hotel, the recipe or the background of that hot guy. Google often finds what users want, but it doesn’t always.”
– Continually iterating on the identification of relevance “signals” and development of “classifiers” to refine search results
• Also continually expanding and refining crawling processes:
– “Crawl coverage” also significantly impacts “search quality”
– Communicating with website owners about bringing down barriers
![Page 16: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/16.jpg)
16 Google Confidential
The long tail of US federal government information
Obscurity of query
Num
ber
of q
uerie
s
irs refund
fda drug approval
veteran record deceased
tree faller npsyosemite
ftc hearing shermanact 1992
Questions search engines can’t completely answer
![Page 17: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/17.jpg)
17 Google Confidential
Citizens expect to find everything through search
![Page 18: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/18.jpg)
18 Google Confidential
But search engines can only index a fraction of what citizens are looking for
![Page 19: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/19.jpg)
19 Google Confidential
Many government websites are effectively invisible to search engine users
• Content “hidden” behind search forms
• Non-HTML links
• Outdated robots.txtcrawling restrictions
• Server errors
• Orphaned URLs
• Rich media: audio, video
• Paid/subscription-only access
Barriers to search engine crawling:
WEBSearchable
DEEP WEBNot searchable
![Page 20: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/20.jpg)
20 Google Confidential
Representative barriers to search engine users
• USAJOBS.gov job search (http://jobsearch.usajobs.opm.gov/):
– [statistician jobs federal government]
– Barriers: Agency information access policy and database structure
![Page 21: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/21.jpg)
21 Google Confidential
![Page 22: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/22.jpg)
22 Google Confidential
![Page 23: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/23.jpg)
23 Google Confidential
![Page 24: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/24.jpg)
24 Google Confidential
Representative barriers to search engine users
• ATF website (http://www.atf.gov):
– Sample search: [atf small business program]
– Barrier: Robots.txt (tacit agency information access policy)
![Page 25: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/25.jpg)
25 Google Confidential
![Page 26: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/26.jpg)
26 Google Confidential
![Page 27: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/27.jpg)
27 Google Confidential
![Page 28: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/28.jpg)
28 Google Confidential
![Page 29: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/29.jpg)
29 Google Confidential
Representative barriers to search engine users
• US Census Bureau Question & Answer Center (https://ask.census.gov/):
– [census america's largest minority group]
– Barrier: Database structure
![Page 30: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/30.jpg)
30 Google Confidential
![Page 31: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/31.jpg)
31 Google Confidential
![Page 32: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/32.jpg)
32 Google Confidential
![Page 33: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/33.jpg)
33 Google Confidential
![Page 34: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/34.jpg)
34 Google Confidential
Scope of the issue
![Page 35: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/35.jpg)
35 Google Confidential
Scope of the issue
• Most US federal agencies operate one or more “invisible” or un-indexable websites (e.g. databases):
– Up to 2,000 sites
– Over 1 billion documents
– DTIC, EPA, NASA, USGS
![Page 36: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/36.jpg)
36 Google Confidential
![Page 37: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/37.jpg)
37 Google Confidential
Scope of the issue
• Most US federal agencies operate one or more “invisible” or uncrawlable websites (e.g. databases):
– Up to 2,000 sites
– Over 1 billion documents
– DTIC, EPA, NASA, USGS
• State of Texas: Up to 250 uncrawlable sites
• Issue common to governments worldwide (but not the commercial web)
![Page 38: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/38.jpg)
38 Google Confidential
The good news
• The Sitemap Protocol provides a solution:
– Can make all pages, documents, and records on a website accessible to search engine crawlers
– Does not require redevelopment or redesign, just a comparably modest time investment
– Keeps the site’s owner in control
![Page 39: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/39.jpg)
39 Google Confidential
XML Sitemap
• A comprehensive list of a site’s URLs tagged with:
– Location
– Last modification date
– Change frequency
– Priority
XML sitemap
![Page 40: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/40.jpg)
40 Google Confidential
One solution for all search engines
Once implemented, Sitemaps can be made available to all major search engines
“The launch of Sitemaps is significant because it allows for a single, easy way for websites to provide content and metadata to search engines”
— Tim Mayer, Senior Director of Product Management, Yahoo Search
“We are 100% behind this protocol -- this kind of collaboration will help improve the search experience for all of our customers”
— Ken Moss, General Manager, Live Search
![Page 41: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/41.jpg)
41 Google Confidential
Success stories
• US federal government:– GAO
– GPO
– LOC
– NAL, USDA
– NARA
– NIH, HHS
– NTIS, Commerce
– OSTI, Department of Energy
• States of Arizona, California, Michigan, Virginia, Utah
• District of Columbia and other localities
• Central governments in Europe and Asia Pacific
![Page 42: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/42.jpg)
42 Google Confidential
![Page 43: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/43.jpg)
43 Google Confidential
Recent development
• As part of reauthorization of the E-Gov Act, Senate committee considering amendment adding requirements in support of web search:
– OMB to offer “guidance and best practices to ensure that publicly available online Federal Government information and services aremade more accessible to external search capabilities…”
– Agencies to “ensure compliance with guidance” within two years and report annually
![Page 44: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/44.jpg)
44 Google Confidential
![Page 45: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/45.jpg)
45 Google Confidential
![Page 46: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/46.jpg)
46 Google Confidential
Untrustworthy Trustworthy
Public sector information is a pillar of the web
Government
Generally authoritative, unbiased, free and trustworthy
![Page 47: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/47.jpg)
47 Google Confidential
Meeting citizens’ expectations of government
![Page 48: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/48.jpg)
48 Google Confidential
Join in
• Get informed: www.google.com/publicsector
• Advocate for the Sitemap Protocol at your agency/organization
• Query: John Lewis (JL) Needham, Manager, Public Sector Content Partnerships, [email protected]
WEBSearchable
DEEP WEBSearchable
![Page 49: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/49.jpg)
49 Google Confidential
Appendix
![Page 50: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/50.jpg)
50 Google Confidential
The three levels of search
WEB
SITE
INTRANET
![Page 51: Ensuring government is only one search away...13 Google Confidential One of the fundamentals: Index comprehensiveness • One way to ensure search just works is to provide users search](https://reader034.vdocuments.mx/reader034/viewer/2022042323/5f0dc9757e708231d43c1528/html5/thumbnails/51.jpg)
51 Google Confidential
Web search vs. site search
A segment of the website’s contentSearch scopeAll of the open and crawlable
deep web
VariesCostFree
More detailed, all facetsReporting toolsHigh-level stats
Limited by server capacity and costCrawlingLimited barriers to crawling,
incluidng by robots.txt
CustomizableFreshnessSearch engine crawling intervals
Professionals and citizensUserCitizens and professionals
Supporting the two levels of search