http - all you need to know

HTTPAll you need to

know

Who are you?

Gökhan Şengün

R&D Business Dev., New Product & Solutions Manager

www.gokhansengun.com

@gokhansengun

Aim• Better understand HTTP basics to debug problems

better• Know HTTP players to see the big picture• Know useful tools to do things faster

HTTP• HTTP is a stateless protocol. • How is being stateless like?• A stateless protocol does not require the server to retain

information or status about each user for the duration of multiple requests.

Http Proxy

Popular Http Proxies• Fiddler• Burp Suite• Browser Developer Tools (Embedded Proxy)

Demo – Bare Metal - Using Telnet

Demo – Browser Developer Tool

Demo – Fiddler

Demo – Burp Suite

Http Protocol – Important Parts

MethodsMethod Used for

GET Retrieve a resource

POST Create / Update a resource [Not Idempotent]

PUT Create / Update a resource [Idempotent]

DELETE Delete a resource

HEAD Retrieve a resource except the body

Response CodesCode Meaning

1xx Informative

2xx Success

3xx Requires Additional Action

4xx Client Error (It is your fault)

5xx Server Error (It is my fault)

Accept (Req)

MIME used for media-type. Client gives hint about the types that it understands well and preference.

Syntax:

• Accept: <MIME_type>/<MIME_subtype>

Examples:

• Accept: application/json, text/xml;q=0.9, */*;q=0.8

Content-Type (Req / Resp)

MIME used for media-type

Examples:

• Content-Type: text/html; charset=utf-8 • Content-Type: application/json• Content-Type: text/xml

Demo – Accept and Content-Type

Host (Req)

• Hints the web server about the domain name requested• Optionally includes port, default• HTTP: 80• HTTPS: 443

Examples:

• Host: www.gokhansengun.com• Host: localhost:8090

Connection (Req / Resp)

• Hint from both client and the web server about TCP connection• close: if either party for some reason wants to close• keep-alive: if either party want to keep open for further

requests• Persistent connection (default in HTTP/1.1

• RFC 2616 limits 2 connection per host, browsers have 6 now.

Examples:

• Connection: close• Connection: keep-alive

BTW: Http Pipelining• Only Idempotent

requests allowed (GET, HEAD)• Guess why?

• Has benefit only on high latency setups.

Accept-Languge (Req)

• Hint from client about its language preference

Examples:

• Accept-Language: en-US,en;q=0.8• Accept-Language: tr-TR, tr;q=0.9, en;q=0.8, *;q=0.5

Demo – Accept-Language

Accept-Encoding (Req)

• Hint from client about its encoding preference

Examples:

• Accept-Encoding: Accept-Encoding: gzip, deflate, sdch• Omit for non-encoding

Demo – Accept-Encoding

Referer (Req)

• Hint from client about the last page user navigated from.• Allows analytics, caching, logging

Examples:

• Referer: http://ads.xyz.com

User-Agent (Req)

• Hint from client about the type of client

Examples:

• User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36

Cache-Control (Req / Resp)

• Hint from server to all over the world about resource’s cache eligibility.

• Cache-Control: no-cache• Cache-Control: public• Cache-Control: private• Cache-Control: no-store• Cache-Control: max-age=300• Cache-Control: public, max-age=31536000

Post / Redirect / Get Pattern (1)• Problem (Multiple Post requests)

Post / Redirect / Get Pattern (2)

Post / Redirect / Get Pattern (3)• Solution

Demo – Mix

HTTP Players• Web Servers• Load Balancers• DDoS Protection and WAF Systems• Cache Server• CDN (Content Delivery Networks)• Cloudflare

Web Servers• Nginx• Apache• IIS

Load Balancers• Balance HTTP load between servers• Balance statefully (needs your SSL private key)• Cache responses• Alters requests and responses• Blocks, rate-limits requests• Does SSL-offloading (needs your SSL private key and

beneficial only if you have HW LB)

DDoS Protection Systems and WAF• Observes traffic (needs your SSL private key)• Detects malicious activity – several attacks• Blocks IP, IP Range• Redirects to No CAPTCHA or reCAPTCHA• Rate-limits requests

Cache Servers• Caches any type of HTTP responses from origion• Could be static file or reference data• Like very very simple KV store• Powerful if scripting allowed

Examples:

• Varnish• Nginx

CDN (Content Delivery Network)• Caches the content on the edges• Request does not enter your data center• Very very efficient

Cloudflare• CDN• Load Balancing (Cloud – Region Based through DNS)• DDoS• WAF• Rate Limiting• Website Optimization• Cache Header Optimization• AutoMinify• Aggressive Gzip• Automatic Content Caching

Cookies• Helps stateless HTTP protocol statefulness when

necessary,• Has restrictions in EU.

Types:

• Session Cookies• Persistent Cookies

Authentication and Tokens• Basic Authentication• Forms Authentication• Token Authentication

Session Cookie vs Token Auth

HTTP Security• Use SSL/TLS for transport layer security (HTTPS

everything)• Why?

• Set Cookies with HttpOnly• Avoid Cross Site Scripting

• Set Cookies with Secure• Avoid sending cookies in HTTP requests

• Use HSTS (HTTP Strict Transport Security) header• Instruct browser to comm only with HTTPS for a period of time• Avoid SSL-stripping attacks

HTTP Performance Measurement• Use Apache ab• Use Apache JMeter (blogs from

www.gokhansengun.com)• http://loader.io/• https://www.blazemeter.com/• Use APM (Application Performance Monitoring) tools• NewRelic, Dynatrace, Riverbed, App

Scaling HTTP• Use Cache Server• Use CDN• Cache Aggressively• Use DNS load balancing• Use SPA (Single Page Application) Technique• Minify and bundle JS / CSS

Questions?