writing a search engine. how hard could it be?
TRANSCRIPT
![Page 2: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/2.jpg)
ABOUT
ABOUT ME
▸ Consultant at Compositional IT
▸ F# dev for ~3 years now
▸ Interested in Big Data, IoT, Cloud and Distributed Systems
![Page 3: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/3.jpg)
COMPOSITIONAL IT
FUNCTIONAL FIRST. CLOUD READY. @COMPOSITIONALIT
![Page 4: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/4.jpg)
HOW HARD COULD IT BE?
Every software developer ever
INTRODUCTION
![Page 5: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/5.jpg)
IT’S ONLY AN OPERATING SYSTEM, ALL IT DOES IS RUNS PROGRAMS!
Everybody when Windows blue screens
INTRODUCTION
![Page 6: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/6.jpg)
IT’S ONLY A MULTIPLAYER ONLINE VIDEO GAME!
Anybody playing a game when lag spikes hit
TEXT
![Page 7: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/7.jpg)
IT’S ONLY 2 LINES OF JAVASCRIPT
Backend developer needing to make a small API change
INTRODUCTION
![Page 8: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/8.jpg)
DUDE. HOLD MY BEER.
Drunk people 10 seconds before making a terrible mistake
![Page 9: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/9.jpg)
SATURDAY MORNING. PLANS CANCELLED.
![Page 10: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/10.jpg)
WHAT NEXT? HIT UP GOOGLE.
![Page 11: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/11.jpg)
WHAT TO DO IN LONDON THIS WEEKEND?
![Page 12: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/12.jpg)
![Page 13: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/13.jpg)
![Page 14: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/14.jpg)
WRITING A SEARCH ENGINE. HOW HARD COULD IT BE?
![Page 15: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/15.jpg)
WRITING A SEARCH ENGINE WITH AZURE AND F# IN A WEEKEND.
![Page 16: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/16.jpg)
BUT FIRST.
![Page 17: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/17.jpg)
THIS WAS A WEEKEND PROJECT.
![Page 18: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/18.jpg)
YOU SHOULD EXPECT: - HACKY CODE.
![Page 19: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/19.jpg)
YOU SHOULD EXPECT: - DEMOS TO FAIL.
![Page 20: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/20.jpg)
YOU SHOULD NOT EXPECT: - A DEEP DIVE INTO SEARCH ENGINE TECH.
![Page 21: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/21.jpg)
SEARCH ENGINE BACKGROUND
CONSTRAINTS
▸ Not a priority
▸ Can’t cost more than £85 per month
▸ No operations investment
▸ Limit to the weekend
![Page 22: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/22.jpg)
BACKGROUND
EVERYTHING I KNOW ABOUT HOW SEARCH ENGINES WORK
▸
▸
▸
▸
▸
▸
![Page 23: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/23.jpg)
THE ANATOMY OF A LARGE-SCALE HYPER TEXTUAL WEB SEARCH ENGINE
SERGEY BRIN LARRY PAGE
![Page 24: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/24.jpg)
![Page 25: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/25.jpg)
IT’S 2016. THE WEB’S CHANGED. A LOT.
![Page 26: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/26.jpg)
WHAT’S NEW? + SCALE
![Page 27: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/27.jpg)
WHAT’S NEW? + USERS
![Page 28: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/28.jpg)
WHAT’S NEW? + GLOBALISATION
![Page 29: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/29.jpg)
WHAT’S NEW? + CLOUD
![Page 30: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/30.jpg)
WHAT’S NEW? + PLATFORM AS A SERVICE
![Page 31: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/31.jpg)
WHAT’S NEW? - INFRASTRUCTURE
![Page 32: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/32.jpg)
WHAT’S NEW? - PERSONAL HOSTING
![Page 33: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/33.jpg)
SEARCH ENGINE BACKGROUND
WHAT’S IMPORTANT?
▸ Search
▸ Scraping
▸ Page rank
![Page 34: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/34.jpg)
SEARCH IMPLEMENTATION
HOW TO FIND A NEEDLE IN A HAYSTACK
▸ Take all of your documents
▸ Record all of the words which occur within a file
▸ Invert that index
▸ List of all words and the documents they appear in
▸ For all words in the search query, find the files which appear in every inverted index
![Page 35: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/35.jpg)
SOUNDS EASY RIGHT? I DON’T CARE ABOUT IT.
![Page 36: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/36.jpg)
AZURE SEARCHMANAGED SEARCH AS A SERVICE
![Page 37: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/37.jpg)
AZURE SEARCH
WHAT DOES AZURE SEARCH GIVE US?
▸ Hosted Search as a Service
▸ HTTP API for indexing and retrieving documents
▸ Ability to scale out (more replicas, more indexes)
▸ Free basic tier
![Page 38: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/38.jpg)
AZURE SEARCH IN THE AZURE PORTAL.
![Page 39: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/39.jpg)
BOOSTING DEMO.
![Page 40: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/40.jpg)
WE HAVE SEARCH. WHAT NEXT?
![Page 41: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/41.jpg)
INDEXING DATA
WHAT IS A CRAWLER
▸ Autonomously find every web page on the internet
▸ Pull the content from that web page and index it
▸ Read the links on that page and index those links
▸ Recursively process until every page on the internet has been reached
![Page 42: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/42.jpg)
THE PROBLEM? THE INTERNET’S PRETTY BIG.
![Page 43: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/43.jpg)
AZURE SERVICE BUS
DISTRIBUTED MESSAGE QUEUES
![Page 44: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/44.jpg)
INDEXING DATA
WHAT DOES AZURE SERVICE BUS GIVE US?
▸ Scalable durable queues and topics with guaranteed availability
▸ .Net APIs to communicate with the service bus
▸ Free basic tier
![Page 45: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/45.jpg)
WORKING WITH A SERVICE BUS QUEUE.
![Page 46: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/46.jpg)
WE NEED TO BE GOOD CITIZENS. WE DON’T WANT TO DDOS A SINGLE WEBSITE DURING CRAWLING.
![Page 47: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/47.jpg)
SERVICE BUS PROVIDES SUPPORT FOR MESSAGE DE-DUPLICATION BASED ON CONTENT.
![Page 48: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/48.jpg)
WE DON’T WANT TO SCRAPE THROUGH EVERY WEB PAGE IN THE WORLD.
![Page 49: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/49.jpg)
WE DON’T WANT TO INDEX: - GOOGLE SEARCH QUERIES
![Page 50: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/50.jpg)
WE DON’T WANT TO INDEX: - PROTECTED CONTENT
![Page 51: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/51.jpg)
WE DON’T WANT TO INDEX: - IRRELEVANT CONTENT
![Page 52: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/52.jpg)
DEALING WITH THE ROBOTS.TXT FILE
WRITING BASIC PARSERS IN F#
![Page 53: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/53.jpg)
BEING A WELL BEHAVED SCRAPER
WHAT IS ROBOTS.TXT?
▸ Text file standard for telling web scrapers what they should scrape
▸ Opt-in - crawlers can ignore the robots.txt file
▸ Simple file stored at the root of the web server
![Page 54: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/54.jpg)
AN EXAMPLE ROBOTS.TXT FILE.
![Page 55: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/55.jpg)
SIMPLE PARSING WITH F#.
![Page 56: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/56.jpg)
HTML AND INFORMATION RETRIEVAL
QUERYING HTML DOCUMENTS WITH HTML AGILITY PACK
![Page 57: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/57.jpg)
WE HAVE A HTML FILE. WE NEED THE CONTENT OUT OF IT.
![Page 58: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/58.jpg)
INFORMATION RETRIEVAL FROM HTML DOCUMENTS
WORKING WITH THE HTML AGILITY PACK
▸ Provides a simple query layer over HTML documents
▸ Works with well formatted and poorly formatted HTML
▸ Provides XPath support over the document
▸ Allows for querying for individual properties and elements
![Page 59: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/59.jpg)
EXTRACTING LINKS FROM A HTML DOCUMENT
![Page 60: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/60.jpg)
EXTRACTING ALL OF THE CONTENT FROM AN HTML DOCUMENT
![Page 61: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/61.jpg)
WE NOW HAVE A WEB SCRAPER. WE NEED TO RUN THE WEB SCRAPER.
![Page 62: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/62.jpg)
AZURE WEBJOBSSIMPLE HOSTING OF LONG RUNNING PROCESSES
![Page 63: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/63.jpg)
AZURE WEB JOBS
WHAT ARE WEB JOBS?
▸ A means of hosting basic executables in the cloud
▸ Provides simplified deployment and monitoring
▸ Pricing per minute of usage
![Page 64: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/64.jpg)
WE NOW HAVE A SEARCH ENGINE. KIND OF.
![Page 65: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/65.jpg)
SEARCH IS A RECOMMENDATION PROBLEM.
![Page 66: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/66.jpg)
HOW DO WE RECOMMEND CONTENT TO USERS?
![Page 67: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/67.jpg)
PAGE RANKFINDING THE MOST INFLUENTIAL SITES ON THE INTERNET
![Page 68: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/68.jpg)
PAGE RANK
WHAT IS PAGE RANK?
▸ Stanford’s patented algorithm
▸ Helps you find the most influential websites on the internet
▸ Websites with lots of links to them are more influential
![Page 69: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/69.jpg)
THE PROBLEM? THERE’S LOTS OF WEBSITES ON THE INTERNET.
![Page 70: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/70.jpg)
THERE’S EVEN MORE LINKS BETWEEN WEBSITES.
![Page 71: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/71.jpg)
WE HAVE A HUGE LINK GRAPH. WE NEED TO PROCESS THAT GRAPH.
![Page 72: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/72.jpg)
BIG DATA PROCESSING WITH MBRACE AND CLOUDFLOWS.
![Page 73: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/73.jpg)
WE HAVE A QUERY WHICH NEEDS TO RUN DAILY. WE NEED TO ORCHESTRATE IT.
![Page 74: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/74.jpg)
AZURE FUNCTIONS + AZURE RESOURCE MANAGER
USING AZURE FUNCTIONS FOR DEVOPS
![Page 75: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/75.jpg)
DEVOPS
WHAT IS AZURE RESOURCE MANAGER?
▸ Declarative way of describing Azure infrastructure
▸ REST APIs to deploy infrastructure template files
▸ APIs to see current deployment status
![Page 76: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/76.jpg)
DEVOPS
WHAT IS AZURE FUNCTIONS?
▸ Lightweight scripting of Azure web jobs
▸ Allows for running scripts in response to certain events
▸ Billing based on number of function invocations
![Page 77: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/77.jpg)
DEVOPS
USING AZURE FUNCTIONS FOR DEVOPS
▸ Set up a timer triggered Azure Function
▸ Deploy an Mbrace cluster through Azure Resource Manager
▸ Send an event when the job completes
▸ Second Azure Function for deleting the MBrace cluster
![Page 78: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/78.jpg)
AZURE FUNCTIONS AND AZURE RESOURCE MANAGER.
![Page 79: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/79.jpg)
WE NOW HAVE EVERYTHING IN PLACE FOR A SEARCH ENGINE. NOBODY CAN ACCESS IT THOUGH.
![Page 80: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/80.jpg)
AZURE FUNCTIONS
SERVERLESS WEB APIS WITH AZURE FUNCTIONS
![Page 81: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/81.jpg)
AZURE FUNCTIONS CAN OPERATE ON HTTP REQUESTS.
![Page 82: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/82.jpg)
NO LONG TERM HOSTING COSTS.
![Page 83: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/83.jpg)
AZURE FUNCTIONS HTTP API DEMO.
![Page 84: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/84.jpg)
DONE. SEARCH ENGINE COMPLETE.
![Page 85: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/85.jpg)
HTTP API
AZURE SEARCH
LINK DATABASE
PAGERANK
CLUSTER ORCHESTRATOR
AZURE SERVICEBUS
INDEXER
PAGERANK IMPORTERPAGERANK SCORE
STORE
![Page 86: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/86.jpg)
PLENTY OF ROOM FOR IMPROVEMENTS.
![Page 87: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/87.jpg)
CACHING SEARCH QUERIES.
![Page 88: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/88.jpg)
QUERY AUTO COMPLETE.
![Page 89: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/89.jpg)
SEARCH A GIVEN DOMAIN.
![Page 90: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/90.jpg)
MULTIPLE LANGUAGE SUPPORT.
![Page 91: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/91.jpg)
SUPPORT FOR OTHER DOCUMENT TYPES.
![Page 92: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/92.jpg)
BETTER INFORMATION RETRIEVAL ALGORITHMS.
![Page 93: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/93.jpg)
WHAT’S NEXT FOR IT? NOTHING.
![Page 94: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/94.jpg)
PRODUCTISING A GOOGLE COMPETITOR IS BASICALLY IMPOSSIBLE.
![Page 95: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/95.jpg)
IN SUMMARYWRAPPING UP & KEY TAKEAWAYS
![Page 96: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/96.jpg)
AZURE + F# = <3
![Page 97: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/97.jpg)
AZURE MAKES HARD INFRASTRUCTURE PROBLEMS SIMPLE.
![Page 98: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/98.jpg)
F# MAKES HARD SOFTWARE PROBLEMS SIMPLE.
![Page 99: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/99.jpg)
TOGETHER THEY MAKE HARD PROBLEMS SIMPLE.
![Page 100: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/100.jpg)
IT’S NOT GOOGLE. BUT IT TOOK 1 DEV 2 DAYS.
![Page 101: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/101.jpg)
CLOUD IS THE EPITOME OF BUSINESS AGILITY
![Page 102: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/102.jpg)
![Page 104: Writing a Search Engine. How hard could it be?](https://reader031.vdocuments.mx/reader031/viewer/2022030309/58f18dd21a28ab56018b45ef/html5/thumbnails/104.jpg)
Q&A.