qian liu, computer and information sciences department

1

Qian Liu, Computer and Information Sciences Department

A Presentation on

Extracting Patterns and Relations from the World Wide Web

Sergey Brin

2


Problem

•The World Wide Web as an information resource:

•Huge•Widely distributed•Complex, various styles and formats•Scattered information

•So, if we could integrate the chunks of information...

3


Motivation

Discover information sources

Extract information of a particular data type automatically/with minimal human intervention

Integrate into a structured form

The largest and most diverse source of information

4


Applications

• To extract structured data from the entire World Wide Web

• Data types: books, movies, music, restaurants, etc.

5


Methods

Problem: To extract a relation of books --- (author, title) pairs from the Web.

6


Methods

Intuition:A small seed set of books (author, title pairs)

Find occurrences of them on the Web

Generate patterns

Search for books matching the patterns

Obtain a large list of books

7


Methods

Formal Definition of the Problem:

•World Wide Web

•Relation --- (author, title) pairs that occur on the Web

•Occurrences

•Every tuple of the relation occurs >= 1 times on the Web

• Consists of all fields of the tuple

• Fields --- in close proximity to one another

8


Methods

Formal Definition of the Problem (Continued):

• Patterns

•Matching one particular format of occurrences of tuples of the relation. (order, urlprefix, prefix, middle, suffix)

• Represented by a class of regular expressions

9


Methods

R’: Approximation of relation R

Coverage (recall) =

Error rate =

Precision =

|R’ + R| R

|R’ - R| R’|R’ + R| R’

10


Methods

Method: Dual Iterative Pattern Relation Expansion

Basis:

•Find tuples from patterns.

•Find patterns from tuples.

11


Methods

Set of patterns with high coverage and low error rate

Set of tuples

Find all matches to patterns

Find all occurrences of the tuples. Discover similarities in occurrences

12


Methods

1. Start with a small sample, e.g., five books.

2. Find all occurrences of the sample books on WWW.

Keep the context of every occurrence (url and surrounding text).

13


Methods

3. Generate patterns based on the occurrences.

Requirements:

• Generate patterns for sets of occurrences with similar context

• Low error rate

• Coverage

14


Methods

Procedure:

•Group the occurrences by order and middle.

• For each group: set urlprefix, prefix, suffix.

Specificity of Pattern:

• Too specific?

• Too general?

• Specificity(p)=|p.middle| |p.url| |p.prefix| |p.suffix|

15


Methods

4. Search the Web for tuples matching the pattern.

5. Is result large enough?

If yes, return.

If no, go to step 2.

16


Experiments

1st

iteration2nd

iteration3rd

iterationUnique(author,title) pairs

5 4047 9127

Occurrences 199 3972 9938patterns 3 105 346Result:unique pairs

4047 9127 15257

17


Limitations of Study

1. Scalability problem: Limited experiments due to time constraints.

2. Problem with data: duplicate books.

3. Measure of safety in matching tuples with patterns: To match a single pattern.

18


Suggestions for Future Studies

1. Scan for larger numbers of patterns and tuples over a huge repository.

2. Include methods to disregard differences such as capitalization, space, how the author is listed in the book, and so on.

19


Conclusions

• DIPRE --- a remarkable tool to extract structured data from the Web

• Minimal human intervention

• Application in domains other than books

• Finding books not listed in major online sources

--- change in information flow

20


qian liu, computer and information sciences department

Documents