comp 3331/comp 9331 week 3€¦ · v smtp: delivery/storage to receiver’s server v mail access...

Application Layer (Email, DNS, P2P, Socket Programming)

Computer Networks and Applications

Week 3

COMP 3331/COMP 9331

Reading Guide: Chapter 2, Sections 2.4 – 2.7

Announcements

v  Lab for Week 3 – Socket Programming Practice v  Sample Problem Set

§  Did anyone attempt the first problem set? Please do. Solutions soon.

§  Set for application layer is now available v  Remember mid-semester exam in Week 6 v  First Programming Assignment released this

week

2

2. Application Layer: outline

2.1 principles of network applications §  app architectures §  app requirements

2.2 Web and HTTP 2.3 FTP 2.4 electronic mail

§  SMTP, POP3, IMAP 2.5 DNS

2.6 P2P applications 2.7 socket programming

with UDP and TCP

3

4 http://arstechnica.com/business/2016/03/e-mail-inventor-ray-tomlinson-who-popularized-symbol-dies-at-74/

Electronic mail Three major components: v  user agents v  mail servers v  simple mail transfer

protocol: SMTP

User Agent v  a.k.a. “mail reader” v  composing, editing, reading

mail messages v  e.g., Outlook, Thunderbird,

iPhone mail client v  outgoing, incoming

messages stored on server

user mailbox

outgoing message queue

mail server

mail server

mail server

SMTP

SMTP

SMTP

user agent

user agent

user agent

user agent

user agent

user agent

5

Electronic mail: mail servers

mail servers: v  mailbox contains incoming

messages for user v  message queue of outgoing

(to be sent) mail messages v  SMTP protocol between mail

servers to send email messages §  client: sending mail

server §  “server”: receiving mail

server

mail server

mail server

mail server

SMTP

SMTP

SMTP

user agent

user agent

user agent

user agent

user agent

user agent

6

Electronic Mail: SMTP [RFC 2821]

v  uses TCP to reliably transfer email message from client to server, port 25

v  direct transfer: sending server to receiving server

v  three phases of transfer §  handshaking (greeting) §  transfer of messages §  closure

v  command/response interaction (like HTTP, FTP) §  commands: ASCII text §  response: status code and phrase

v  messages must be in 7-bit ASCII 7

user agent

Scenario: Alice sends message to Bob

1) Alice uses UA to compose message “to” [email protected]

2) Alice’s UA sends message to her mail server; message placed in message queue

3) client side of SMTP opens TCP connection with Bob’s mail server

4) SMTP client sends Alice’s message over the TCP connection

5) Bob’s mail server places the message in Bob’s mailbox

6) Bob invokes his user agent to read message

mail server

mail server

1

2 3 4 5

6

Alice’s mail server Bob’s mail server

user agent

8

Sample SMTP interaction S: 220 hamburger.edu C: HELO crepes.fr S: 250 Hello crepes.fr, pleased to meet you C: MAIL FROM: <[email protected]> S: 250 [email protected]... Sender ok C: RCPT TO: <[email protected]> S: 250 [email protected] ... Recipient ok C: DATA S: 354 Enter mail, end with "." on a line by itself C: Do you like ketchup? C: How about pickles? C: . S: 250 Message accepted for delivery C: QUIT S: 221 hamburger.edu closing connection 9

Try SMTP interaction for yourself:

v  telnet servername 25 v  see 220 reply from server v  enter HELO, MAIL FROM, RCPT TO, DATA, QUIT

commands above lets you send email without using email client (reader) Implication: one could send forged emailed

10

Note: Many SMTP servers will not allow the above interaction without authentication. E.g: try the above with mail.unsw.edu.au @ port 587

How to tell a fake email?

Examine Long Headers or Raw Source

Further reading: http://www.millersmiles.co.uk/identitytheft/spoofemail-060603.htm 11

Securing E-mail

v  STARTTLS: upgrade a plain text connection to TLS/SSL instead of using a separate port for encrypted communication §  Can be used for SMTP/IMAP/POP

v  PGP (later in the course)

12

SMTP: final words

v  SMTP uses persistent connections

v  SMTP requires message (header & body) to be in 7-bit ASCII

v  SMTP server uses CRLF.CRLF to determine end of message

comparison with HTTP: v  HTTP: pull v  SMTP: push

v  both have ASCII command/response interaction, status codes

v  HTTP: each object encapsulated in its own response msg

v  SMTP: multiple objects sent in multipart msg

13

Mail message format

SMTP: protocol for exchanging email msgs

RFC 822: standard for text message format:

v  header lines, e.g., §  To: §  From: §  Subject: different from SMTP MAIL

FROM, RCPT TO: commands!

v  Body: the “message” §  ASCII characters only

header

body

blank line

14

v  IF SMTP only allows 7-bit ASCII, how do we send pictures/videos/files via email?

A: We use a different protocol instead of SMTP B: We encode these objects as 7-bit ASCII C: We’re really sending links to the objects, rather than the objects themselves D: We don’t !! You have been lied to !!

15

Quiz: E-mail attachments?

Mail access protocols

v  SMTP: delivery/storage to receiver’s server v  mail access protocol: retrieval from server

§  POP: Post Office Protocol [RFC 1939]: authorization, download

§  IMAP: Internet Mail Access Protocol [RFC 1730]: more features, including manipulation of stored msgs on server

§  HTTP(S): Gmail, Yahoo! Mail, etc.

sender’s mail server

SMTP SMTP mail access

protocol

receiver’s mail server

(e.g., POP, IMAP)

user agent

user agent

16

POP3 protocol

authorization phase v  client commands:

§  user: declare username §  pass: password

v  server responses §  +OK §  -ERR

transaction phase, client: v  list: list message numbers v  retr: retrieve message by

number v  dele: delete v  quit

C: list S: 1 498 S: 2 912 S: . C: retr 1 S: <message 1 contents> S: . C: dele 1 C: retr 2 S: <message 1 contents> S: . C: dele 2 C: quit S: +OK POP3 server signing off

S: +OK POP3 server ready C: user bob S: +OK C: pass hungry S: +OK user successfully logged on

17

Self Study

POP3 (more) and IMAP more about POP3 v  previous example uses

POP3 “download and delete” mode §  Bob cannot re-read e-

mail if he changes client

v  POP3 “download-and-keep”: copies of messages on different clients

v  POP3 is stateless across sessions

IMAP v  keeps all messages in one

place: at server v  allows user to organize

messages in folders v  keeps user state across

sessions: §  names of folders and

mappings between message IDs and folder name

18

Self Study

v  Which of the following is not true?

A.  HTTP is pull-based, SMTP is push-based

B.  HTTP uses a separate header for each object, SMTP uses a multipart message format

C.  SMTP uses persistent connections

D.  HTTP uses client-server communication but SMTP does not

19

Quiz: HTTP vs SMTP






with UDP and TCP

A nice overview https://webhostinggeeks.com/guides/dns/

20

DNS: domain name system

people: many identifiers: §  TFN, name, passport #

Internet hosts, routers: §  IP address (32 bit) -

used for addressing datagrams

§  “name”, e.g., www.yahoo.com - used by humans

Q: how to map between IP address and name, and vice versa ?

Domain Name System: v  distributed database

implemented in hierarchy of many name servers

v  application-layer protocol: hosts, name servers communicate to resolve names (address/name translation) §  note: core Internet function,

implemented as application-layer protocol

§  complexity at network’s “edge”

21

DNS: History v  Initially all host-address mappings were in a hosts.txt file (in /etc/

hosts): §  Maintained by the Stanford Research Institute (SRI) §  Changes were submitted to SRI by email §  New versions of hosts.txt periodically FTP’d from SRI §  An administrator could pick names at their discretion

v  As the Internet grew this system broke down: §  SRI couldn’t handle the load; names were not unique; hosts had inaccurate

copies of hosts.txt

v  The Domain Name System (DNS) was invented to fix this

22

Jon Postel

http://www.wired.com/2012/10/joe-postel/

Example use of DNS

v  You type www.cse.unsw.edu.au in the URL window of your web browser

v  Your browser must establish a TCP connection with the CSE web server

v  To do this, your browser needs to know the IP address of the CSE web server

v  How does it obtain the IP address?? §  The browser passes on the hostname to the client side of the DNS

application running on your machine (gethostbyname() function in UNIX) §  The DNS client sends out a query for mapping the hostname to an IP

address into the DNS hierarchy black box over UDP (destination port number: 53)

§  The DNS client receives a reply with the IP address for www.cse.unsw.edu.au

§  The browser can now initiate a TCP connection with the HTTP server process located at that IP address

23

DNS: services, structure why not centralize DNS? v  single point of failure v  traffic volume v  distant centralized database v  maintenance

DNS services v  hostname to IP address

translation v  host aliasing

§  canonical, alias names v  mail server aliasing v  load distribution

§  replicated Web servers: many IP addresses correspond to one name

§  Content Distribution Networks: use IP address of requesting host to find best suitable server

•  Example: closest, least-loaded, etc

A: doesn’t scale!

24

Goals

v  No naming conflicts (uniqueness) v  Scalable

§  many names §  (secondary) frequent updates

v  Distributed, autonomous administration §  Ability to update my own (machines’) names §  Don’t have to track everybody’s updates

v  Highly available v  Lookups should be fast

25

Key idea: Hierarchy

Three intertwined hierarchies §  Hierarchical namespace

•  As opposed to original flat namespace

§  Hierarchically administered •  As opposed to centralised

§  (Distributed) hierarchy of servers •  As opposed to centralised storage

26

Hierarchical Namespace

v  “Top Level Domains” are at the top v  Domains are sub-trees

§  E.g: .edu, berkeley.edu, eecs.berkeley.edu v  Name is leaf-to-root path

§  instr.eecs.berkeley.edu v  Depth of tree is arbitrary (limit 128) v  Name collisions trivially avoided

§  each domain is responsible

root

edu com gov mil org net uk fr

berkeley ucla

eecs sims

instr

…

27

28

Hierarchical Administration

root


berkeley ucla

eecs sims

instr

root


berkeley

eecs sims §  A zone corresponds to an administrative authority that

is responsible for that portion of the hierarchy §  E.g., UCB controls names: *.berkeley.edu and

*.sims.berkeley.edu

v  E.g., EECS controls names: *.eecs.berkeley.edu

Server Hierarchy

v  Top of hierarchy: Root servers §  Location hardwired into other servers

v  Next Level: Top-level domain (TLD) servers

§  .com, .edu, etc. §  Managed professionally

v  Bottom Level: Authoritative DNS servers §  Actually store the name-to-address mapping §  Maintained by the corresponding administrative authority

29

Server Hierarchy

v  Each server stores a (small!) subset of the total DNS database

v  An authoritative DNS server stores “resource records” for all DNS names in the domain that it has authority for

v  Each server needs to know other servers that are responsible for the other portions of the hierarchy §  Every server knows the root §  Root server knows about all top-level domains

30

DNS Root v  Located in Virginia, USA v  How do we make the root scale?

Verisign, Dulles, VA

31

DNS Root Servers v  13 root servers (labeled A-M; see http://www.root-servers.org/)

B USC-ISI Marina del Rey, CA L ICANN Los Angeles, CA

E NASA Mt View, CA F Internet Software Consortium Palo Alto, CA

I Autonomica, Stockholm

K RIPE London

M WIDE Tokyo

A Verisign, Dulles, VA C Cogent, Herndon, VA D U Maryland College Park, MD G US DoD Vienna, VA H ARL Aberdeen, MD J Verisign

32

DNS Root Servers

B USC-ISI Marina del Rey, CA L ICANN Los Angeles, CA

E NASA Mt View, CA F Internet Software Consortium, Palo Alto, CA (and 37 other locations)

I Autonomica, Stockholm (plus 29 other locations)

K RIPE London (plus 16 other locations)

M WIDE Tokyo plus Seoul, Paris, San Francisco

A Verisign, Dulles, VA C Cogent, Herndon, VA (also Los Angeles, NY, Chicago) D U Maryland College Park, MD G US DoD Vienna, VA H ARL Aberdeen, MD J Verisign (21 locations)

l  13 root servers (labeled A-M; see http://www.root-servers.org/) l  Replicated via any-casting

33 Root Server health: https://www.ultratools.com/tools/dnsRootServerSpeed

Anycast in a nutshell

v  Routing finds shortest paths to destination

v  If several locations are given the same address, then the network will deliver the packet to the closest location with that address

v  This is called “anycast” §  Very robust §  Requires no modification to routing algorithms

34

TLD, authoritative servers

top-level domain (TLD) servers: §  responsible for com, org, net, edu, aero, jobs, museums,

and all top-level country domains, e.g.: uk, fr, ca, jp §  Network Solutions maintains servers for .com TLD §  Educause for .edu TLD

authoritative DNS servers: §  organization’s own DNS server(s), providing authoritative

hostname to IP mappings for organization’s named hosts §  can be maintained by organization or service provider

35

Local DNS name server v  does not strictly belong to hierarchy v  each ISP (residential ISP, company, university) has one

§  also called “default name server” v  Hosts configured with local DNS server address (e.g., /etc/

resolv.conf) or learn server via a host configuration protocol (e.g., DHCP)

v  Client application §  Obtain DNS name (e.g., from URL) §  Do gethostbyname() to trigger DNS request to its local DNS server

v  when host makes DNS query, query is sent to its local DNS server §  has local cache of recent name-to-address translation pairs

(but may be out of date!) §  acts as proxy, forwards query into hierarchy

36

requesting host wagner.cse.unsw.edu.au

gaia.cs.umass.edu

root DNS server

local DNS server cse.unsw.edu.au

1

2 3

4 5

6

authoritative DNS server dns.cs.umass.edu

7 8

TLD DNS server

DNS name resolution example

v  host at wagner.cse.unsw.edu.au wants IP address for gaia.cs.umass.edu

iterated query: v  contacted server

replies with name of server to contact

v  “I don’t know this name, but ask this server”

37

4 5

6 3

recursive query: v  puts burden of name

resolution on contacted name server


gaia.cs.umass.edu

root DNS server


1

2 7


8

DNS name resolution example

TLD DNS server

Applet: http://media.pearsoncmg.com/aw/aw_kurose_network_2/applets/dns/dns.html 38

39

Quiz: Which one would you use? Why?


gaia.cs.umass.edu

root DNS server


1

2 3

4 5

6


7 8

TLD DNS server

4 5

6 3

gaia.cs.umass.edu

root DNS server


1

2 7

8


Iterated queries Recursive queries

DNS: caching, updating records v  once (any) name server learns mapping, it caches

mapping §  cache entries timeout (disappear) after some time (TTL) §  TLD servers typically cached in local name servers

•  thus root name servers not often visited

v  Subsequent requests need not burden DNS v  cached entries may be out-of-date (best effort

name-to-address translation!) §  if name host changes IP address, may not be known

Internet-wide until all TTLs expire

40

v  The TTL value should be:

A.  Short, to make sure that changes are accurately reflected

B.  Long to avoid re-queries of higher level DNS servers

C.  Something else

41

Quiz: DNS Record TTL

DNS records

DNS: distributed db storing resource records (RR)

type=NS §  name is domain (e.g.,

foo.com) §  value is hostname of

authoritative name server for this domain

RR format: (name, value, type, ttl)

type=A §  name is hostname §  value is IP address

type=CNAME §  name is alias name for some

“canonical” (the real) name §  www.ibm.com is really servereast.backup2.ibm.com

§  value is canonical name

type=MX §  value is name of mailserver

associated with name

42

DNS protocol, messages v  query and reply messages, both with same message

format

msg header v  identification: 16 bit # for

query, reply to query uses same #

v  flags: §  query or reply §  recursion desired §  recursion available §  reply is authoritative

identification flags

# questions

questions (variable # of questions)

# additional RRs # authority RRs

# answer RRs

answers (variable # of RRs)

authority (variable # of RRs)

additional info (variable # of RRs)

2 bytes 2 bytes

43

name, type fields for a query

RRs in response to query

records for authoritative servers

additional “helpful” info that may be used

identification flags

# questions

questions (variable # of questions)

# additional RRs # authority RRs

# answer RRs

answers (variable # of RRs)

authority (variable # of RRs)

additional info (variable # of RRs)

DNS protocol, messages

2 bytes 2 bytes

44

bash-3.2$ dig www.cse.unsw.edu.au ; <<>> DiG 9.6-ESV-R4 <<>> www.cse.unsw.edu.au ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26306 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 2, ADDITIONAL: 4 ;; QUESTION SECTION: ;www.cse.unsw.edu.au. IN A ;; ANSWER SECTION: www.cse.unsw.edu.au. 300 IN CNAME albeniz.orchestra.cse.unsw.edu.au. albeniz.orchestra.cse.unsw.edu.au. 86400 IN A 129.94.242.51

;; AUTHORITY SECTION: orchestra.cse.unsw.edu.au. 86400 IN NS beethoven.orchestra.cse.unsw.edu.au. orchestra.cse.unsw.edu.au. 86400 IN NS maestro.orchestra.cse.unsw.edu.au. ;; ADDITIONAL SECTION: maestro.orchestra.cse.unsw.edu.au. 86400 IN A 129.94.242.33 beethoven.orchestra.cse.unsw.edu.au. 86400 IN A 129.94.208.3 beethoven.orchestra.cse.unsw.edu.au. 86400 IN A 129.94.242.2 beethoven.orchestra.cse.unsw.edu.au. 86400 IN A 129.94.172.11 ;; Query time: 1 msec ;; SERVER: 129.94.242.2#53(129.94.242.2) ;; WHEN: Mon Mar 18 12:36:28 2013 ;; MSG SIZE rcvd: 195 45

Your 2nd lab

Application Layer

Inserting records into DNS

v  example: new startup “Network Utopia” v  register name networkuptopia.com at DNS registrar

(e.g., Network Solutions) §  provide names, IP addresses of authoritative name server

(primary and secondary) §  registrar inserts two RRs into .com TLD server: (networkutopia.com, dns1.networkutopia.com, NS)

(dns1.networkutopia.com, 212.212.212.1, A) v  create authoritative server type A record for

www.networkuptopia.com; type MX record for networkutopia.com

v  Q: Where do you insert these type A and type MX records?

46

Reliability

v  DNS servers are replicated (primary/secondary) §  Name service available if at least one replica is up §  Queries can be load-balanced between replicas

v  Usually, UDP used for queries §  Need reliability: must implement this on top of UDP §  Spec supports TCP too, but not always implemented

v  Try alternate servers on timeout §  Exponential backoff when retrying same server

v  Same identifier for all queries §  Don’t care which server responds

47

DNS provides Indirection

v  Addresses can change underneath §  Move www.cnn.com to 4.125.91.21 §  Humans/Apps should be unaffected

v  Name could map to multiple IP addresses §  Enables

•  Load-balancing •  Reducing latency by picking nearby servers

v  Multiple names for the same address §  E.g., many services (mail, www, ftp) on same machine §  E.g., aliases like www.cnn.com and cnn.com

v  But, this flexibility applies only within domain! 48

DNS Load Balancing

49

•  (Author i t a t i ve ) DNS ser ver monitors the load of the multiple web servers

•  Replies back with the IP address for the least loaded server

•  Can also consider the location of the client (e.g., a user from Sydney directed to Australian server)

DNS for Content Distribution

50

0: cnn.com distributes images to Akamai server 1: An end user types in www.cnn.com in their browser 2: index page contains link to the a73.g.akamai.net 3: local DNS server asks root DNS server to resolve a73.g.akamai.net 4: root DNS replies with IP address of high-level Akamai DNS server, akamai.net 5: local DNS server queries this nameserver to resolve a73.g.akamai.net 6: high-level DNS server replies with IP address of the DNS server for g.akamai.net 7: local DNS queries this nameserver to resolve a73.g.akamai.net 8: low level DNS server takes into consideration the client’s location and replies with the IP address of the closest Akamai web server that hosts the image 9: end user’s machine sends a GET request for the image to this closest Akamai web server 10: Akamai webserver responds with the image

Reverse DNS

v  IP address -> domain name v  Special PTR record type to store reverse DNS

entries v  Where is reverse DNS used?

§  Troubleshooting tools such as traceroute and ping §  “Received” trace header field in SMTP e-mail §  SMTP servers for validating IP addresses of originating

servers §  Internet forums tracking users §  System logging or monitoring tools §  Used in load balancing servers/content distribution to

determine location of requester

51

Do you trust your DNS server?

v  Censorship

v  Logging §  IP address, websites visited, geolocation data and more §  E.g., Google DNS:

52

https://developers.google.com/speed/public-dns/privacy

https://wikileaks.org/wiki/Alternative_DNS

Attacking DNS DDoS attacks v  Bombard root servers

with traffic §  Not successful to date §  Traffic Filtering §  Local DNS servers cache

IPs of TLD servers, allowing root server to be bypassed

v  Bombard TLD servers §  Potentially more dangerous

Redirect attacks v  Man-in-middle

§  Intercept queries v  DNS poisoning

§  Send bogus replies to DNS server, which caches

Exploit DNS for DDoS v  Send queries with spoofed

source address: target IP v  Requires amplification

53

Want to dig deeper? http://www.networkworld.com/article/2886283/security0/top-10-dns-attacks-likely-to-infiltrate-your-network.html

DNS Cache Poisoning v  Suppose you are a bad guy and you control the name server

for drevil.com. Your name server receives a request to resolve www.drevil.com. and you respond as follows:

;; QUESTION SECTION: ;www.drevil.com. IN A ;; ANSWER SECTION: www.drevil.com 300 IN CNAME 129.45.212.42

;; AUTHORITY SECTION: drevil.com 86400 IN NS dns1.drevil.com. drevil.com 86400 IN NS google.com ;; ADDITIONAL SECTION: google.com 600 IN A 129.45.212.222

v  Solution: Do not allow DNS servers to cache IP address mappings unless they are from authoritative name servers

54

A drevil.com machine, not google.com

Dig deeper?

DNS Cache Poisoning Test https://www.grc.com/dns/dns.htm DNSSEC: DNS Security Extensions, http://www.dnssec.net

55






with UDP and TCP

56

Pure P2P architecture v  no always-on server v  arbitrary end systems

directly communicate v  peers are intermittently

connected and change IP addresses

examples: §  file distribution

(BitTorrent) §  Streaming (KanKan) §  VoIP (Skype)

57

File distribution: client-server vs P2P

Question: how much time to distribute file (size F) from one server to N peers? §  peer upload/download capacity is limited resource

us

uN

dN

server

network (with abundant bandwidth)

file, size F

us: server upload capacity

ui: peer i upload capacity

di: peer i download capacity u2 d2

u1 d1

di

ui

58

File distribution time: client-server

v  server transmission: must sequentially send (upload) N file copies: §  time to send one copy: F/us §  time to send N copies: NF/us

increases linearly in N

time to distribute F to N clients using

client-server approach Dc-s > max{NF/us,,F/dmin}

v  client: each client must download file copy §  dmin = min client download rate §  min client download time: F/dmin

us

network di

ui

F

59

File distribution time: P2P

v  server transmission: must upload at least one copy §  time to send one copy: F/us

time to distribute F to N clients using

P2P approach

us

network di

ui

F

DP2P > max{F/us,,F/dmin,,NF/(us + Σui)}

v  client: each client must download file copy §  min client download time: F/dmin

v  clients: as aggregate must download NF bits §  max upload rate (limiting max download rate) is us + Σui

… but so does this, as each peer brings service capacity increases linearly in N …

60

0

0.5

1

1.5

2

2.5

3

3.5

0 5 10 15 20 25 30 35

N

Min

imum

Dis

tribu

tion

Tim

e P2PClient-Server

Client-server vs. P2P: example

client upload rate = u, F/u = 1 hour, us = 10u, dmin ≥ us

61

P2P file distribution: BitTorrent

tracker: tracks peers participating in torrent

torrent: group of peers exchanging chunks of a file

Alice arrives …

v  file divided into 256Kb chunks v  peers in torrent send/receive file chunks

… obtains list of peers from tracker … and begins exchanging file chunks with peers in torrent

62

.torrent files

v  Contains address of trackers for the file §  Where can I find other peers?

v  Contain a list of file chunks and their cryptographic hashes §  This ensures that chunks are not modified

63

v  peer joining torrent: §  has no chunks, but will

accumulate them over time from other peers

§  registers with tracker to get list of peers, connects to subset of peers (“neighbours”)

P2P file distribution: BitTorrent

v  while downloading, peer uploads chunks to other peers v  peer may change peers with whom it exchanges chunks v  churn: peers may come and go v  once peer has entire file, it may (selfishly) leave or

(altruistically) remain in torrent

64

BitTorrent: requesting, sending file chunks

requesting chunks: v  at any given time, different

peers have different subsets of file chunks

v  periodically, Alice asks each peer for list of chunks that they have

v  Alice requests missing chunks from peers, rarest first

sending chunks: tit-for-tat v  Alice sends chunks to those

four peers currently sending her chunks at highest rate §  other peers are choked by Alice

(do not receive chunks from her) §  re-evaluate top 4 every10 secs

v  every 30 secs: randomly select another peer, starts sending chunks §  “optimistically unchoke” this peer §  newly chosen peer may join top 4

65

BitTorrent: tit-for-tat (1) Alice “optimistically unchokes” Bob (2) Alice becomes one of Bob’s top-four providers; Bob reciprocates (3) Bob becomes one of Alice’s top-four providers

higher upload rate: find better trading partners, get file faster !

Original Research Paper on BitTorrent linked to lecture notes page 66

Getting rid of the server

v  Distribute the tracker information using a Distributed Hash Table (DHT)

v  A DHT is a lookup structure §  Maps keys to an arbitrary value §  Works a lot like, well …. hash table

67

Title End-systems that have it House of Cards Season 4 Michonne, Rick, Chuck, Sansa Walking Dead Season 6 Frank, Mike, Tyrion, Gregor Game of Thrones Season Saul, Mike, Glenn Better Call Saul Season 2 Khal, Jon, Daryl

Getting rid of the server

v  Distribute the tracker information using a Distributed Hash Table (DHT)

v  A DHT is a lookup structure §  Maps keys to an arbitrary value §  Works a lot like, well …. hash table

68

Title End-systems that have it House of Cards Season 4 124.45.66.128, 88.45.34.214 Walking Dead Season 6 111.111.111.111, 432.125.43.43 Game of Thrones Season 42.12.43.32 Better Call Saul Season 2 123.134.2.123

Recall: Hash Function

v  Mapping of any data to an integer §  E.g. , md5sum, sha1, etc. §  md5: 04c3416cadd85971a129dd1de86cee49

v  With a good (cryptographic) hash

§  Hash values likely to be unique, although duplicates are possible

§  Very difficult to find collisions (hashes spread out)

69

Recall: Hash Table

v  N buckets

v  Key-value pair is assigned bucket i §  i = HASH(key)%N

v  Easy to look up value based on key

v  Multiple key-value pairs assigned to each bucket

70

Distributed Hash Table (DHT)

v DHT: a distributed P2P database v database has (key, value) pairs; examples:

§  key: TFN number; value: human name §  key: file name; value: BT tracker peer(s)

v Distribute the (key, value) pairs over the (millions of peers)

v a peer queries DHT with key § DHT returns values that match the key

v peers can also insert (key, value) pairs 71

Overlay network

v  A network made up of “virtual” or logical links

v  Virtual links map to one or more physical links (i.e. paths)

72

Overlay Network

• A network made up of “virtual” or logical links

• Virtual links map to one or more physical links

Challenges

v  How do we assign (key, value) pairs to nodes?

v  How do we find them again quickly?

v  What happens if nodes join/leave?

73

Q: how to assign keys to peers?

v  basic idea: §  convert each key to an integer §  Assign integer to each peer §  put (key,value) pair in the peer that is closest to the

key

74

DHT identifiers: Consistent Hashing

v assign integer identifier to each peer in range [0,2n-1] for some n-bit hash function §  E.g., node ID is hash of its IP address

v require each key to be an integer in same range v to get integer key, hash original key

§  e.g., key = hash(“House of Cards Season 4”) §  this is why its is referred to as a distributed “hash”

table

75

Assign keys to peers

v rule: assign key to the peer that has the closest ID.

v common convention: closest is the immediate successor of the key.

v e.g., n=4; peers: 1,3,4,5,8,10,12,14; §  key = 13, then successor peer = 14 §  key = 15, then successor peer = 1

76

1

3

4

5

8 10

12

15

Circular DHT (1)

v  each peer only aware of immediate successor and predecessor.

v  “overlay network” 77

0001

0011

0100

0101

1000 1010

1100

1111

Who’s responsible for key 1110 ? I am

1110

1110

1110

1110

1110

1110

Define closest as closest successor

Circular DHT (2)

78

v  Given N nodes, what is the complexity (number of messages) of finding a value when each peer knows its successor?

A.  O (log n)

B. O (n) C. O (n2) D. O (2n)

79

Quiz: DHT Complexity

Circular DHT with shortcuts

v  each peer keeps track of IP addresses of predecessor, successor, short cuts

v  reduced from 6 to 2 messages. v  possible to design shortcuts so O(log N) neighbours, O(log N)

messages in query

1

3

4

5

8 10

12

15

Who’s responsible for key 1110?

80

Peer churn

example: peer 5 abruptly leaves v peer 4 detects peer 5 departure; makes 8 its immediate successor; asks 8 who its immediate successor is; makes 8’s immediate successor its second successor.

1

3

4

5

8 10

12

15

handling peer churn: v peers may come and go (churn) v each peer knows address of its two successors v each peer periodically pings its two successors to check aliveness v if immediate successor leaves, choose next successor as new immediate successor

81

v What happens to 5’s value? §  A. Lost forever §  B. Lost until 5 comes back §  Still recoverable (how?)

82

Quiz: Peer Churn 1

3

4

5

8 10

12

15

v Which of the following is not true?

A.  DHT distributes portions of a hash table across peers.

B.  The key corresponding to an object (e.g., movie) depends on the current number of peers.

C.  Which peer is responsible for an object depends on the current number of peers.

83

Quiz: DHT

More DHT info

v  How do nodes join?

v  How does cryptographic hashing work?

v  How much state does each node store?

84

Research Papers (on the lectures page): Chord: A Scalable Peer-to-Peer Lookup Service for Internet Applications Dynamo: Amazon’s Highly Available Key-value Store

DHT: Applications v  File sharing and backup [CFS, Ivy, OceanStore, PAST,

Pastiche …] v  Web cache and replica [Squirrel, Croquet Media Player] v  Censor-resistant stores [Eternity] v  DB query and indexing [PIER, Place Lab, VPN Index] v  Event notification [Scribe] v  Naming systems [ChordDNS, Twine, INS, HIP] v  Distributed BitTorrent tracker [Kademlia, Vuze] v  Communication primitives [I3, …] v  Host mobility [DTN Tetherless Architecture]

85






with UDP and TCP Note: 6th Edition uses Python. Lecture notes show examples of both Java and Python

86

Socket programming

goal: learn how to build client/server applications that communicate using sockets

socket: door between application process and end-end-transport protocol

Internet

controlled by OS

controlled by app developer

transport

application

physical

link

network

process

transport

application

physical

link

network

process socket

Self Study

87

Socket programming

Two socket types for two transport services: §  UDP: unreliable datagram §  TCP: reliable, byte stream-oriented

Application Example: 1.  Client reads a line of characters (data) from its

keyboard and sends the data to the server. 2.  The server receives the data and converts

characters to uppercase. 3.  The server sends the modified data to the client. 4.  The client receives the modified data and displays

the line on its screen.

Self Study

88

Socket programming with UDP

UDP: no “connection” between client & server v  no handshaking before sending data v  sender explicitly attaches IP destination address and

port # to each packet v  rcvr extracts sender IP address and port# from

received packet

UDP: transmitted data may be lost or received out-of-order

Application viewpoint: v UDP provides unreliable transfer of groups of bytes

(“datagrams”) between client and server

Self Study

89

Client/server socket interaction: UDP (Java)

close clientSocket

Server (running on hostid)

read reply from clientSocket

create socket,

clientSocket = DatagramSocket()

Client

Create, address (hostid, port=x, send datagram request using clientSocket

create socket, port=x, for incoming request: serverSocket = DatagramSocket()

read request from serverSocket write reply to serverSocket specifying client host address, port number

Self Study

90

Example: Java client (UDP) import java.io.*; import java.net.*; class UDPClient { public static void main(String args[]) throws Exception { BufferedReader inFromUser = new BufferedReader(new InputStreamReader(System.in)); DatagramSocket clientSocket = new DatagramSocket(); InetAddress IPAddress = InetAddress.getByName("hostname"); byte[] sendData = new byte[1024]; byte[] receiveData = new byte[1024]; String sentence = inFromUser.readLine();

sendData = sentence.getBytes();

Create input stream

Create client socket

Translate hostname to IP

address using DNS

Self Study

91

Example: Java client (UDP), cont. DatagramPacket sendPacket = new DatagramPacket(sendData, sendData.length, IPAddress, 9876); clientSocket.send(sendPacket); DatagramPacket receivePacket = new DatagramPacket(receiveData, receiveData.length); clientSocket.receive(receivePacket); String modifiedSentence = new String(receivePacket.getData()); System.out.println("FROM SERVER:" + modifiedSentence); clientSocket.close(); }

}

Create datagram with data-to-send,

length, IP addr, port Send datagram

to server

Read datagram from server

Self Study

92

Example: Java server (UDP) import java.io.*; import java.net.*; class UDPServer { public static void main(String args[]) throws Exception { DatagramSocket serverSocket = new DatagramSocket(9876); byte[] receiveData = new byte[1024]; byte[] sendData = new byte[1024]; while(true) { DatagramPacket receivePacket = new DatagramPacket(receiveData, receiveData.length);

serverSocket.receive(receivePacket);

Create datagram socket

at port 9876

Create space for received datagram

Receive datagram

Self Study

93

Example: Java server (UDP), cont String sentence = new String(receivePacket.getData()); InetAddress IPAddress = receivePacket.getAddress(); int port = receivePacket.getPort(); String capitalizedSentence = sentence.toUpperCase(); sendData = capitalizedSentence.getBytes(); DatagramPacket sendPacket = new DatagramPacket(sendData, sendData.length, IPAddress, port); serverSocket.send(sendPacket); } }

}

Get IP addr port #, of

sender

Write out datagram to socket End of while loop,

loop back and wait for another datagram

Create datagram to send to client

Self Study

94

Client/server socket interaction: UDP (Python)

close clientSocket

read datagram from clientSocket

create socket: clientSocket = socket(AF_INET,SOCK_DGRAM)

Create datagram with server IP and port=x; send datagram via clientSocket

create socket, port= x: serverSocket = socket(AF_INET,SOCK_DGRAM)

read datagram from serverSocket

write reply to serverSocket specifying client address, port number

server (running on serverIP) client

Self Study

95

96

Example app: UDP client

from socket import * serverName = ‘hostname’ serverPort = 12000 clientSocket = socket(socket.AF_INET, socket.SOCK_DGRAM) message = raw_input(’Input lowercase sentence:’) clientSocket.sendto(message,(serverName, serverPort))

modifiedMessage, serverAddress = clientSocket.recvfrom(2048) print modifiedMessage clientSocket.close()

Python UDPClient include Python’s socket library

create UDP socket for server

get user keyboard input

Attach server name, port to message; send into socket

print out received string and close socket

read reply characters from socket into string

Self Study

97

Example app: UDP server

from socket import * serverPort = 12000 serverSocket = socket(AF_INET, SOCK_DGRAM) serverSocket.bind(('', serverPort)) print “The server is ready to receive” while 1: message, clientAddress = serverSocket.recvfrom(2048) modifiedMessage = message.upper() serverSocket.sendto(modifiedMessage, clientAddress)

Python UDPServer

create UDP socket

bind socket to local port number 12000

loop forever

Read from UDP socket into message, getting client’s address (client IP and port)

send upper case string back to this client

Self Study

Socket programming with TCP client must contact server v  server process must first be

running v  server must have created

socket (door) that welcomes client’s contact

client contacts server by: v  Creating TCP socket,

specifying IP address, port number of server process

v  when client creates socket: client TCP establishes connection to server TCP

v  when contacted by client, server TCP creates new socket for server process to communicate with that particular client §  allows server to talk with

multiple clients §  source port numbers used

to distinguish clients (more in Chap 3)

TCP provides reliable, in-order byte-stream transfer (“pipe”) between client and server

application viewpoint:

Self Study

98

TCP Sockets Self Study

99

Client/server socket interaction: TCP (in Java)

wait for incoming connection request connectionSocket = welcomeSocket.accept()

create socket, port=x, for incoming request:

welcomeSocket = ServerSocket()

create socket, connect to hostid, port=x

clientSocket = Socket()

close connectionSocket

read reply from clientSocket close clientSocket

Server (running on hostid) Client

send request using clientSocket read request from

connectionSocket write reply to connectionSocket

TCP connection setup

Self Study

100

Example: Java client (TCP) import java.io.*; import java.net.*; class TCPClient { public static void main(String argv[]) throws Exception { String sentence; String modifiedSentence; BufferedReader inFromUser = new BufferedReader(new InputStreamReader(System.in)); Socket clientSocket = new Socket("hostname", 6789); DataOutputStream outToServer = new DataOutputStream(clientSocket.getOutputStream());

Create input stream

Create client socket,

connect to server Create

output stream attached to socket

Self Study

101

Example: Java client (TCP), cont.

BufferedReader inFromServer = new BufferedReader(new InputStreamReader(clientSocket.getInputStream())); sentence = inFromUser.readLine(); outToServer.writeBytes(sentence + '\n'); modifiedSentence = inFromServer.readLine(); System.out.println("FROM SERVER: " + modifiedSentence); clientSocket.close(); } }

Create input stream

attached to socket

Send line to server

Read line from server

Self Study

102

Example: Java server (TCP) import java.io.*; import java.net.*; class TCPServer { public static void main(String argv[]) throws Exception { String clientSentence; String capitalizedSentence; ServerSocket welcomeSocket = new ServerSocket(6789); while(true) { Socket connectionSocket = welcomeSocket.accept(); BufferedReader inFromClient = new BufferedReader(new InputStreamReader(connectionSocket.getInputStream()));

Create welcoming socket

at port 6789

Wait, on welcoming socket for contact

by client Create input

stream, attached to socket

Self Study

103

Example: Java server (TCP), cont DataOutputStream outToClient = new DataOutputStream(connectionSocket.getOutputStream()); clientSentence = inFromClient.readLine(); capitalizedSentence = clientSentence.toUpperCase() + '\n'; outToClient.writeBytes(capitalizedSentence); } } }

Read in line from socket

Create output stream, attached

to socket

Write out line to socket

End of while loop, loop back and wait for another client connection

Self Study

104

105

Example app: TCP client

from socket import * serverName = ’servername’ serverPort = 12000 clientSocket = socket(AF_INET, SOCK_STREAM) clientSocket.connect((serverName,serverPort)) sentence = raw_input(‘Input lowercase sentence:’) clientSocket.send(sentence) modifiedSentence = clientSocket.recv(1024) print ‘From Server:’, modifiedSentence clientSocket.close()

Python TCPClient

create TCP socket for server, remote port 12000

No need to attach server name, port

Self Study

106

Example app: TCP server

from socket import * serverPort = 12000 serverSocket = socket(AF_INET,SOCK_STREAM) serverSocket.bind((‘’,serverPort)) serverSocket.listen(1) print ‘The server is ready to receive’ while 1: connectionSocket, addr = serverSocket.accept() sentence = connectionSocket.recv(1024) capitalizedSentence = sentence.upper() connectionSocket.send(capitalizedSentence) connectionSocket.close()

Python TCPServer

create TCP welcoming socket

server begins listening for incoming TCP requests

loop forever

server waits on accept() for incoming requests, new socket created on return

read bytes from socket (but not address as in UDP)

close connection to this client (but not welcoming socket)

Self Study

Application Layer: summary

v  application architectures §  client-server §  P2P

v  application service requirements: §  reliability, bandwidth, delay

v  Internet transport service model §  connection-oriented,

reliable: TCP §  unreliable, datagrams: UDP

our study of network apps now complete!

v  specific protocols: §  HTTP §  FTP §  SMTP, POP, IMAP §  DNS §  P2P: BitTorrent, DHT

v  socket programming: TCP, UDP sockets

107

v  typical request/reply message exchange: §  client requests info or

service §  server responds with

data, status code v  message formats:

§  headers: fields giving info about data

§  data: info being communicated

important themes: v  control vs. data msgs

§  in-band, out-of-band v  centralized vs. decentralized v  stateless vs. stateful v  reliable vs. unreliable msg

transfer v  “complexity at network

edge”

Application Layer: summary most importantly: learned about protocols!

108

Solve all sample problems + those posted with lectures Reading Exercise for next week: Chapter 3: 3.1 – 3.4

comp 3331/comp 9331 week 3€¦ · v smtp: delivery/storage to receiver’s server v mail access...

Documents