Web Scraping 1-2-3 with Python + Scrapy

Sammy Fungsammy.hk, gownjob.com

Today Agenda

● Some Cases● Python and Scrapy

Web Scraping

● a computer software technique of extracting information from websites. (Wikipedia)

● for business, hobbies, research.......● NOT talk about business cases today.

CableTV & NOWTV Programme (Past)

● 2004.● slow, slow, slow, or worst - can't connect.● use Flash.

HK Observatory and Joint Typhoon Warning Center

● no easy data exchange format, eg. RSS/Atom.

● We won't check websites everyday.

Transportation - KMB, PTES

● no map view on KMB website for a bus route in the past.

● Exteremly Poor, Ugly (or much worse) map UI on PTES.

My experiences on web scraping

● 2004: php● year after: python● recent year: python with scrapy

Document Types

● HTML, XML,...... ● Text● Others, eg. pictures, videos,......

Web Scraping

● Look for right URLs to scrap.● Look for right content from webpages.● Saving data into data store.● When to run the web scraping program ?

What is Scrapy ?

● An open source web scraping framework for Python.

● Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

Features of Scrapy

● define data you want to scrapy● write spider to extract data● Built-in: selecting and extracting data from

HTML and XML● Built-in: JSON, CSV, XML output● Interactive shell console● Built-in: web service, telnet console, logging● Others

Installation of Scrapy

● pip● APT repo● RPM● tarball (binary/source)

Create new scrapy project

$ scrapy startproject mybotmybot/mybot/scrapy.cfgmybot/mybot/items.pymybot/mybot/pipeline.pymybot/mybot/settings.pymybot/mybot/spiders/myspider.pyetc.......

items.py

from scrapy.item import Item, Field

class HKOCurrentItem(Item): time = Field() station = Field() temperature = Field() humidity = Field() #......

spiders/hko_spider.py (1/5)

from scrapy.spider import BaseSpiderfrom scrapy.selector import HtmlXPathSelectorfrom weatherhk.items import HKOCurrentItem

import datetime, re

spiders/hko_spider.py (2/5)

class HKOCurrentSpider(BaseSpider): name = "HKOCurrentSpider" #allowed_domains = ["www.weather.gov.hk"] start_urls = [ "http://www.weather.gov.hk/textonly/forecast/chinesewx.htm" ]

spiders/hko_spider.py (3/5)

def parse(self, response): hxs = HtmlXPathSelector(response) stations = [] # Getting weather data from each stations. tx = hxs.select("//pre[1]/text()").re('[^\n]*\n')

spiders/hko_spider.py (4/5)

for i in tx: if re.search(u'\d 度',i):

data = HKOCurrentItem() data['time'] = int(dt) data['station'] = self.station.code(i) data['temperature'] = int(re.findall(u'\d+',i)[0]) stations.append(data)

spiders/hko_spider.py (5/5)

return stations

pipelines.py (1/2)

class HKOCurrentPipeline(object): def process_item(self, item, spider): station = self.db[item['station']] storeditem = dict(item.__dict__)['_values']

pipelines.py (2/2)

try: if 'temperature' in storeditem: lasttime = station.find({'temperature': {'$gt': 0}}).sort('time', -1).limit(1) if lasttime[0]['time'] != storeditem['time']: id = self.insert(storeditem)

return item