proxy recita-on

CarnegieMellon

1

ProxyRecita-on

JeffreyLiuRecita4on13:November23,2015

CarnegieMellon

2

Outline¢  Ge3ngcontentontheweb:Telnet/cURLDemo

§  Howthewebreallyworks¢  NetworkingBasics¢  Proxy

§  DueTuesday,December8th§  Gracedaysallowed

¢  StringManipula-oninC

CarnegieMellon

3

TheWebinaTextbook¢  Clientrequestpage,serverprovides,transac-ondone.

¢  Asequen-alservercanhandlethis.Wejustneedtoserveonepageata-me.

¢  Thisworksgreatforsimpletextpageswithembeddedstyles.

Web server

Web client

(browser)

CarnegieMellon

4

Telnet/CurlDemo¢  Telnet

§  Interac4veremoteshell–likesshwithoutsecurity§  MustbuildHTTPrequestmanually

§  Thiscanbeusefulifyouwanttotestresponsetomalformedheaders

[rjaganna@makoshark~]%telnetwww.cmu.edu80Trying128.2.42.52...ConnectedtoWWW-CMU-PROD-VIP.ANDREW.cmu.edu(128.2.42.52).Escapecharacteris'^]'.GEThttp://www.cmu.edu/HTTP/1.0HTTP/1.1301MovedPermanentlyDate:Sat,11Apr201506:54:39GMTServer:Apache/1.3.42(Unix)mod_gzip/1.3.26.1amod_pubcookie/3.3.4amod_ssl/2.8.31OpenSSL/0.9.8e-fips-rhel5Location:http://www.cmu.edu/index.shtmlConnection:closeContent-Type:text/html;charset=iso-8859-1<!DOCTYPEHTMLPUBLIC"-//IETF//DTDHTML2.0//EN"><HTML><HEAD><TITLE>301MovedPermanently</TITLE></HEAD><BODY><H1>MovedPermanently</H1>Thedocumenthasmoved<AHREF="http://www.cmu.edu/index.shtml">here</A>.<P><HR><ADDRESS>Apache/1.3.42Serverat<AHREF="mailto:[email protected]">www.cmu.edu</A>Port 80</ADDRESS></BODY></HTML>Connectionclosedbyforeignhost.

CarnegieMellon

5

Telnet/cURLDemo¢  cURL

§  “URLtransferlibrary”withacommandlineprogram§  BuildsvalidHTTPrequestsforyou!

§  CanalsobeusedtogenerateHTTPproxyrequests:

[rjaganna@makoshark~]%curlhttp://www.cmu.edu/<!DOCTYPEHTMLPUBLIC"-//IETF//DTDHTML2.0//EN"><HTML><HEAD><TITLE>301MovedPermanently</TITLE></HEAD><BODY><H1>MovedPermanently</H1>Thedocumenthasmoved<AHREF="http://www.cmu.edu/index.shtml">here</A>.<P><HR><ADDRESS>Apache/1.3.42Serverat<AHREF="mailto:[email protected]">www.cmu.edu</A>Port 80</ADDRESS></BODY></HTML>

[rjaganna@makoshark~]%curl--proxylemonshark.ics.cs.cmu.edu:3092http://www.cmu.edu/<!DOCTYPEHTMLPUBLIC"-//IETF//DTDHTML2.0//EN"><HTML><HEAD><TITLE>301MovedPermanently</TITLE></HEAD><BODY><H1>MovedPermanently</H1>Thedocumenthasmoved<AHREF="http://www.cmu.edu/index.shtml">here</A>.<P><HR><ADDRESS>Apache/1.3.42Serverat<AHREF="mailto:[email protected]">www.cmu.edu</A>Port80</ADDRESS></BODY></HTML>

CarnegieMellon

6

HowtheWebReallyWorks¢  Inreality,asingleHTMLpagetodaymaydependon10s

or100sofsupportfiles(images,stylesheets,scripts,etc.)¢  Buildsagoodargumentforconcurrentservers

§  Justtoloadasinglemodernwebpage,theclientwouldhavetowaitfor10sofback-to-backrequest

§  I/Oislikelyslowerthanprocessing,soback¢  Cachingissimplerifdoneinpiecesratherthanwhole

page§  Ifonlypartofthepagechanges,noneedtofetcholdpartsagain§  Eachobject(image,stylesheet,script)alreadyhasauniqueURL

thatcanbeusedasakey

CarnegieMellon

7

HowtheWebReallyWorks¢  Excerptfromwww.cmu.edu/index.html:

<htmllang="en"xml:lang="en"xmlns="http://www.w3.org/1999/xhtml"><head>...<linkhref="homecss/cmu.css"rel="stylesheet"type="text/css"/><linkhref="homecss/cmu-new.css"rel="stylesheet"type="text/css"/><linkhref="homecss/cmu-new-print.css"media="print"rel="stylesheet"type="text/css"/><linkhref="http://www.cmu.edu/RSS/stories.rss"rel="alternate"title="CarnegieMellonHomepageStories"type="application/rss+xml"/>...<scriptlanguage="JavaScript"src="js/dojo.js"type="text/javascript"></script><scriptlanguage="JavaScript"src="js/scripts.js"type="text/javascript"></script><scriptlanguage="javascript"src="js/jquery.js"type="text/javascript"></script><scriptlanguage="javascript"src="js/homepage.js"type="text/javascript"></script><scriptlanguage="javascript"src="js/app_ad.js"type="text/javascript"></script>...<title>CarnegieMellonUniversity|CMU</title></head><body>...

CarnegieMellon

8

Sequen-alProxy

CarnegieMellon

9

Sequen-alProxy¢  Notetheslopedshapeofwhenrequestsfinish

§  Althoughmanyrequestsaremadeatonce,theproxydoesnotacceptanewjobun4litfinishesthecurrentone

§  Requestsaremadeinbatches.ThisresultsfromhowHTMLisstructuredasfilesthatreferenceotherfiles.

¢  Comparedtotheconcurrentexample(next),thispagetakesalong-metoloadwithjuststa-ccontent

CarnegieMellon

10

ConcurrentProxy

CarnegieMellon

11

ConcurrentProxy¢  Now,weseemuchlesspurple(wai-ng),andless-me

spentoverall.¢  No-cehowmul-plegreen(receiving)blocksoverlapin

-me§  Ourproxyhasmul4pleconnec4onsopentothebrowsertohandle

severaltasksatonce

CarnegieMellon

12

HowtheWebReallyWorks¢  AnoteonAJAX(andXMLHZpRequests)

§  Normally,abrowserwillmaketheini4alpagerequestthenrequestanysuppor4ngfiles

§  AndXMLHapRequestissimplyarequestfromthepageonceithasbeenloaded&thescriptsarerunning

§  Thedis4nc4ondoesnotmaaerontheserverside–everythingisanHTTPRequest

CarnegieMellon

13





14

Carnegie Mellon

Sockets

¢  Whatisasocket?§  Toanapplica4on,asocketisafiledescriptorthatletstheapplica4onread/

writefrom/tothenetwork§  (allUnixI/Odevices,includingnetworks,aremodeledasfiles)

¢  Clientsandserverscommunicatewitheachotherbyreadingfromandwri-ngtosocketdescriptors

¢  ThemaindifferencebetweenregularfileI/OandsocketI/Oishowtheapplica-on“opens”thesocketdescriptors

15

Carnegie Mellon

OverviewoftheSocketsInterface

Client/ServerSession

Client Server

socket socket

bind

listen

rio_readlineb

rio_writen rio_readlineb

rio_writen

Connec4onrequest

rio_readlineb

close

close EOF

Awaitconnec4onrequestfromnextclient

open_listenfd open_clientfd

accept connect

getaddrinfo getaddrinfo

Carnegie Mellon

16

HostandServiceConversion:getaddrinfo ¢  getaddrinfoisthemodernwaytoconvertstringrepresenta4onsof

host,ports,andservicenamestosocketaddressstructures.§  Replacesobsoletegethostbyname-unsafebecauseitreturnsa

pointertoasta4cvariable

¢  Advantages:§  Reentrant(canbesafelyusedbythreadedprograms).§  Allowsustowriteportableprotocol-independentcode(IPv4andIPv6)§  Givenhostandservice,getaddrinfo returnsresultthat

pointstoalinkedlistofaddrinfostructs,eachpoin4ngtosocketaddressstruct,whichcontainsargumentsforsocketsAPIs.

¢  getnameinfoistheinverseofgetaddrinfo,conver-ngasocketaddresstothecorrespondinghostandservice.

SocketsAPI¢  intsocket(intdomain,inttype,intprotocol);

§  Createafiledescriptorfornetworkcommunica4on§  usedbybothclientsandservers§  intsock_fd=socket(PF_INET,SOCK_STREAM,IPPROTO_TCP);§  Onesocketcanbeusedfortwo-waycommunica4on

¢  intbind(intsocket,conststructsockaddr*address,socklen_taddress_len);§  AssociateasocketwithanIPaddressandportnumber§  usedbyservers§  structsockaddr_insockaddr–family,address,port

17

Carnegie Mellon

SocketsAPI¢  intlisten(intsocket,intbacklog);

§  socket:sockettolistenon§  usedbyservers§  backlog:maximumnumberofwai4ngconnec4ons§  err=listen(sock_fd,MAX_WAITING_CONNECTIONS);

¢  intaccept(intsocket,structsockaddr*address,socklen_t*address_len);§  usedbyservers§  socket:sockettolistenon§  address:pointertosockaddrstructtoholdclientinforma4onamer

acceptreturns§  return:filedescriptor

18

Carnegie Mellon

SocketsAPI¢  intconnect(intsocket,structsockaddr*address,socklen_t

address_len);§  aaempttoconnecttothespecifiedIPaddressandportdescribedin

address§  usedbyclients

¢  intclose(intfd);§  usedbybothclientsandservers§  (alsousedforfileI/O)§  fd:socketfdtoclose

19

Carnegie Mellon

SocketsAPI¢  ssize_tread(intfd,void*buf,size_tnbyte);

§  usedbybothclientsandservers§  (alsousedforfileI/O)§  fd:(socket)fdtoreadfrom§  buf:buffertoreadinto§  nbytes:buflength

¢  ssize_twrite(intfd,void*buf,size_tnbyte);§  usedbybothclientsandservers§  (alsousedforfileI/O)§  fd:(socket)fdtowriteto§  buf:buffertowrite§  nbytes:buflength

20

Carnegie Mellon

CarnegieMellon

21





CarnegieMellon

22

ByteOrderingReminder¢  So,howarethebyteswithinamul--bytewordorderedin

memory?¢  Conven-ons§  BigEndian:Sun,PPCMac,Internet

§  Leastsignificantbytehashighestaddress§  LialeEndian:x86,ARMprocessorsrunningAndroid,iOS,and

Windows§  Leastsignificantbytehaslowestaddress

CarnegieMellon

23

ByteOrderingReminder¢  So,howarethebyteswithinamul--bytewordorderedin

memory?¢  Conven-ons

§  BigEndian:Sun,PPCMac,Internet§  Leastsignificantbytehashighestaddress

¢  Makesuretousecorrectendianness

CarnegieMellon

24

Proxy-Func-onality¢  Shouldworkonvastmajorityofsites

§  Twitch,CNN,NYTimes,etc.§  SomefeaturesofsiteswhichrequirethePOSTopera4on(sending

datatothewebsite),willnotwork-  Loggingintowebsites,sendingFacebookmessage

§  HTTPSisnotexpectedtowork§  Google,YouTube(andsomeotherpopularwebsites)nowtry

topushuserstoHTTPsbydefault;watchoutforthat¢  Cachepreviousrequests

§  UseLRUevic4onpolicy§  Mustallowforconcurrentreadswhilemaintainingconsistency

§  Detailsinwriteup

CarnegieMellon

25

Proxy-Func-onality¢  Whyamul--threadedcache?

n  Sequen4alcachewouldboaleneckparallelproxy

n  Mul4plethreadscanreadcachedcontentsafely

n  Searchcachefortherightdataandreturnitn  Twothreadscanreadfromthesamecacheblock

n  Butwhataboutwri4ngcontent?

n  Overwriteblockwhileanotherthreadreading?

n  Twothreadswri4ngtosamecacheblock?

CarnegieMellon

26

Proxy-How¢  Proxiesareabitspecial-theyareaserverandaclientatthesame

4me.¢  Theytakearequestfromonecomputer(ac4ngastheserver),and

makeitontheirbehalf(astheclient).¢  Ul4mately,thecontrolflowofyourprogramwilllooklikeaserver,but

willhavetoactasaclienttocompletetherequest

¢  Startsmall§  Grabyourselfacopyoftheechoserver(pg.946)andclient(pg.

947)inthebook§  Alsoreviewthe4ny.cbasicwebservercodetoseehowtodeal

withHTTPheaders§  Notethat4ny.cignoresthese;youmaynot

CarnegieMellon

27

Proxy-How¢  Whatyouendupwithwillresemble:

Server(port80)Client

Clientsocketaddress128.2.194.242:51213

Serversocketaddress208.216.181.15:80

Proxy

Proxyserversocketaddress128.2.194.34:15213

Proxyclientsocketaddress128.2.194.34:52943

CarnegieMellon

28

Summary¢  Step1:Sequen-alProxy

§  Worksgreatforsimpletextpageswithembeddedstyles

¢  Step2:ConcurrentProxy§  mul4-threading

¢  Step3:CacheWebObjects§  Cacheindividualobjects,notthewholepage§  UseanLRUevic-onpolicy§  Yourcachingsystemmustallowforconcurrentreadswhile

maintainingconsistency.Concurrency?SharedResource?

CarnegieMellon

29

Proxy–Tes-ng&Grading¢  New:Autograder

§  ./driver.shwillrunthesametestsasautolab:§  Abilitytopullbasicwebpagesfromaserver§  Handlea(concurrent)requestwhileanotherrequestiss4llpending

§  Fetchawebpageagainfromyourcacheamertheserverhasbeenstopped

§  Thisshouldhelpanswertheques4on“isthiswhatmyproxyissupposedtodo?”

§  Pleasedon’tusethisgradertodefini4velytestyourproxy;therearemanythingsnottestedhere

CarnegieMellon

30

Proxy–Tes-ng&Grading¢  Testyourproxyliberally

§  Thewebisfullofspecialcasesthatwanttobreakyourproxy§  Generateaportforyourselfwith./port-for-user.pl[andrewid]§  Generatemoreportsforwebserversandsuchwith./free-port.sh§  Considerusingyourandrewwebspace(~/www)tohosttestfiles

§  Youhavetovisithaps://www.andrew.cmu.edu/server/publish.htmltopublishyourfoldertothepublicserver

¢  Createahandinfilewithmakehandin§  Willcreateatarfileforyouwiththecontentsofyourproxylab-

handinfolder

CarnegieMellon

31





CarnegieMellon

32

Stringmanipula-oninC¢  sscanf:Readinputinspecificformat

intsscanf(constchar*str,constchar*format,…);Example:buf=“213isawesome”//Readintegerandstringseparatedbywhitespacefrombuffer‘buf’//intopassedvariablesret=sscanf(buf,“%d%s%s”,&course,str1,str2);Thisresultsin:course=213,str1=is,str2=awesome,ret=3

CarnegieMellon

33

Stringmanipula-on(cont)¢  sprine:Writeinputintobufferinspecificformat

intsprinI(char*str,constchar*format,…);Example:buf[100];str=“213isawesome”//Buildthestringindoublequotes(“”)usingthepassedarguments//andwritetobuffer‘buf’sprinI(buf,“String(%s)isoflength%d”,str,strlen(str));Thisresultsin:buf=String(213isawesome)isoflength14

CarnegieMellon

34

Stringmanipula-on(cont)Otherusefulstringmanipula-onfunc-ons:¢  strcmp,strncmp,strncasecmp¢  strstr¢  strlen¢  strcpy,strncpy

CarnegieMellon

35

Aside:Se3ngupFirefoxtouseaproxy

¢  Youmayuseanybrowser,butwe’llbegradingwithFirefox

¢  Preferences>Advanced>Network>Se3ngs…(underConnec-on)

¢  Check“Usethisproxyforallprotocols”oryourproxywillappeartoworkforHTTPStraffic.

CarnegieMellon

36

Acknowledgements¢  Slidesderivedfromrecita-onslidesoflast2yearsby

§  Shiva§  HartajSinghDugal§  IanHartwig§  RohithJagannathan

CarnegieMellon

37

Ques-ons?

proxy recita-on

Documents