Intro to Web Scraping with Python

Vancouver PyLadies MeetupMarch 7, 2016

Maris Lemba

2

Before we begin● Do consider whether the site you are interested in allows scraping by

examining its robots.txt file– This robots exclusion standard is used by web servers to communicate with web

crawlers and other robots

– Located in the top-level directory of web server

– Can specify which directories the site owner does not want robots to access

– Can have different access rules for different robots

● If you intend to publish the output from your scraping, consider whether it is legal to do so– It is generally ok to re-publish “factual data”

#Sample robots.txt fileUser-agent: Googlebot Disallow: /folder/

User-agent: *Disallow: /photos/

3

Basic components of a web page● HTML – a set of markup tags for describing web documents● CSS – style sheet language for describing the presentation of an HTML document● JavaScript – popular client-side programming language used for web applications

Sample html source

This is what it would look like in the browser

If we wanted to scrape information from the “Day of the week” table, we can locate it in the html inside the <table> ... </table> element.

If there are multiple tables, we can use attributes such as id to identify the right table.

4

How the web works

user interface

web server

web browser

database, data handling, etc

server-based system

user interface

Ajax “engine” (javascript)

web browser

HTTP requestHTTP request

HTML + CSS data

web server

database, data handling, etc

server-based system

XML/JSON or HTML/CSS or Javascript data

Schematic adapted from https://en.wikipedia.org/wiki/Ajax_%28programming%29

Cl ie

ntS

erve

r

Javascriptcall

HTML+CSS data

Conventional model of web application Ajax model of web application

5

Scraping ad hoc vs at scale

● Use case 1 (basic scraping): I need to scrape a few pages, maybe crawl a small site. I can wait to have all data retrievals done sequentially. – This is what we will focus on in this talk

– Two scenarios:● All the data you need is in the HTML files you get from the server● The page is dynamically generated using Ajax

● Use case 2: I am crawling large sites periodically. I need task scheduling, asynchronous retrieval, pipeline automation, and lots of other tools to make this easier. – You may want to look into a crawling/scraping framework like Scrapy

6

Agenda

● We'll walk through examples for the two scenarios under “use case 1”– We will use two web sites,

pacificroadrunners.ca/firsthalf/ and ultrasignup.com, to scrape the results for all years of two running races

● Pacific Road Runners First Half Marathon● Chuckanut 50K

7

Basic scraping: static HTML

● Tasks:– Inspect the HTML source to locate where in the Document Object Model (DOM) the data of

interest resides● Right-click on the page in browser window and select “View Page Source”

– Retrieve the HTML file using built-in urllib library (note that the interface to this library has changed between Python 2 and 3)

– Parse the HTML file and extract the data of interest, using either a built-in parser like html.parser, or third-party libraries such as lxml, html5lib or BeautifulSoup

● Which library you choose for parsing depends on – how sloppy your html is: if your html is “incorrect”, the parsers may handle it differently

– whether you want to maximize speed (choose lxml)

– what external dependencies you can live with (lxml is a binding to popular C libraries libxml2 and libxslt)

– how “easy” you want the interface to be (BeautifulSoup has an easy-to-get-started API and great documentation with examples)

8

urllib library● High level standard library for retrieving data across the Web

(not just html files)

● urllib library was split into several submodules in Python 3 for working with URLs: – urllib.request, urllib.parse, urllib.error, urllib.robotparser

● The urllib.request provides methods for opening URLs

● The urllib.request.urlopen() method is similar to the built-in function open() for opening files

9

BeautifulSoup library

● Python third-party library for extracting data from html and xml files

● Works with html.parser, lxml, html5lib● Provides ways to navigate, search and modify the parse

tree based on the position in the parse tree, tag name, tag attributes, css classes using regular expressions, user-defined functions etc

● Excellent tutorial with examples at http://www.crummy.com/software/BeautifulSoup/bs4/doc/

● Supports Python 2.7 and 3

10

Quick intro to BeautifulSoup

● BeautifulSoup constructor creates an object that represents original documents as a nested data structure

● The simplest way to address a tag is to use its name● find() and find_all() methods allow to search for an

element; find() returns the first element found, and find_all() a list of elements

● You can access a tag's attributes by treating the tag like a dictionary

● You can extract text from the element

Parsing samplepage.html with BeautifulSoup

11

Example: Results tab of Vancouver First Half race

The main Results page has links to yearly results

12

HTML source inspection: front page of results

Find the links in the HTML source file corresponding to the rows of yearly results you see in the browser

Note that the links we are interested in are all inside <div class=”entry-content”> ... </div>

The links to each year's page are inside an <a> tag, as a value to attribute named href

The text field of the <a> ... </a> element is a string containing the year of the race

13

Extract all links to individual year's results

Note that the collected links are relative; we can append them to the main url to get absolute links

14

HTML source inspection: individual year

Find the table of results in the DOM This is the table we want, each runner's info in <tr> ... </tr>

Each participant's entry is a table row <tr> ... </tr>.The text (participant's placement, name, finishing time, etc) are inside <td> ... </td> elements nested in the table row.

15

Scrape results for all yearsFirst five finishers of the 2016 race on the web page

Let's print out the first five of the 2016 entry in our dictionary

16

What is Ajax

● Name comes from Asynchronous JavaScript and XML, even though XML is no longer the only format for data exchange

● Refers to a group of web technologies used to implement web applications that communicate with the server in the background without interfering with the display of the page or reloading the entire page

● For web scraping, this means that the data you are interested in may not be in the source HTML file you receive from the server but is generated on the client side

● You can use browser development tools (“inspect element”) to inspect the Ajax-generated HTML that your browser displays

17

Second example: ultrasignup.comSo far, web site looks similar to previous example. We are interested in scraping results for all years of the race.

18

Results page for an individual year

This is a sample page for 2014 results. It looks like the results are in a neat table.

19

Inspecting the DOM: the result table is not filled out in the source HTML

However, the <table id=”list”> element is empty when we view source html file.This means we can't simply extract the result table from the HTML file that the server sent to us like we did in the previous example.

We see all result rows in <table id=”list”> </table> when inspecting the element in browser developer tools such as Mozilla Firebug (which shows the generated HTML)

Again, the details of the results are within the <td> ... </td> element inside each <tr> ... </tr> element

20

Scraping dynamically generated HTML

● Your scraping system needs to be able to execute JavaScript and either retrieve the results directly, or generate the HTML as seen in the browser and scrape data from that

● A popular choice for the latter is to use Selenium, a suite of tools for browser automation– Python's binding to Selenium suite is the library selenium

– Selenium supports several browsers (Firefox, Chrome, etc), but for scraping a headless non-graphical browser like PhantomJS is a good choice

– Once Selenium has generated the HTML after running JavaScript, you can either access it as a string and use this as input to BeautifulSoup for data extraction, or you can navigate the DOM and find elements within Selenium

21

Quick intro to selenium

● Selenium WebDriver accepts commands, sends them to the browser, and retrieves results● You can navigate and search the DOM

– find_element_by_id('my_id')

– find_elements_by_tag_name('table')

– find_elements_by_class_name('my_class_name')

– find_elements_by_xpath()● For example, find_elements_by_xpath(“//table[@id='list']”) will find a table with the id = “list”● Tutorial on xpath syntax: www.w3schools.com/xsl/xpath_intro.asp

– and other methods (using css selectors, link text, etc)

● You can click on links using WebDriver's click() method

● You can retrieve the generated html source by calling .page_source on webdriver instance and parse it outside of selenium

● More on Selenium with Python:

http://selenium-python.readthedocs.org/index.html

22

Scraping ultrasignup race results using Selenium and PhantomJS

23

And it works!Note that the first two lists (rows in web page table) in our dictionary corresponding to a year do not actually contain finisher results. Let's ignore those, and print out the subsequent results for the first 12 finishers.

This table is from the 2014 results page for comparison: the results match

24

Additional resources for scraping projects

Additional third-party libraries you may want to look at:

● requests library

– Filling out forms/login information before accessing data and managing cookies

● mechanize library

– Navigating through forms, session and cookie handling

– Does not yet support Python 3

– Does not run JavaScript

● scrapy library

– Framework for web crawling and scraping: managing crawling, asynchronous scheduling and processing of requests, cookie and session handling, built-in support for data extraction etc

– Does not yet support Python 3

25

References for further reading

● BeautifulSoup– http://www.crummy.com/software/BeautifulSoup/bs4/doc/

● Selenium with Python

– http://selenium-python.readthedocs.org/

● General web scraping– Web Scraping with Python by Ryan Mitchell,

O'Reilly 2015