Networkingstuff,fromtherealworld

RobertGrahamDartmouth,Spring,2017

OSIMODELTRUTHS

TheOSIisalie•  Createdinlate1970stodescribehowterminalstalkedtomainframes

•  TCP/IPmodelshoehornedintoit–  TheOSIModelhasbeen“retconned”tofitTCP/IP

•  Thereisnosessionlayer–  Yes,sessionconcepts,butnolayer

•  ThereisnopresentaNonlayer–  Yes,presentaNonconcepts,butnolayer

•  ThereisnotevenanapplicaNonlayer–  ApplicaNonsareareontopoftheapplicaNonlayer,nottheapplicaNonlayerthemselves

Thereareonly4layers

•  Transport(TCP,UDP,SCTP)•  Internetwork(IPv4,IPv6)•  Localdatalink(Ethernet,WiFi)•  Localphysical(Ethernet,WiFi)

Localphysicallayer

•  ASFARAS:thelocalwire(orintotheair)•  UNIT:bits

Locallink

•  ASFARAS:nexthop•  UNIT:frames(localaddress,CRCchecked)

•  STRIPPEDOFFBEFORENEXTHOP•  Othernon-Ethernetlinksexist– MPEG-TS,ATM,FrameRelay,PPP,etc.

Internetworklayer

•  ASFARAS:otherend(end-to-end)•  UNIT:packet

Transportlayer

•  ASFARAS:remoteapplicaNon•  UNIT:stream,datagram

Sideways/up-downs•  Onlytwoup/down

APIs–  TCP/IPisaunified

wholeintheoperaNngsystemwithsocketsasAPIontop

–  Ethernetisaunifiedwholewithpacketdriverontop

•  There’snosideways–  It’sabadanalogy

thatleadstoconfusion

Physical

Link

Network

Link

Physical

Link Link

Network Network

Transport Transport

ThereusedtobemanyInternets

•  XeroxIDP/SDP•  NovelIPX/SPX•  AppleTalk•  DECnet•  SNA•  BanyanVines•  GOSIP–  ISO/OSItofitthemodel

GOSIPfailed

•  Itreallywasdesignedtofitthatmodel•  OpNmizaNonstoovercomewhat’sbroken– SessionsetupinsideTransportsetuppackets– Sessionlayeraddedinvisiblebytestopackets

•  Somuchoverheadcouldneverworkright

•  YetwesNllgetX.509intoday’snetworks•  AndLDAP

WHATDOESIPV6SOLVE?

TrickquesNon

•  Already10billiondevicesontheIPv4Internet•  Soobviously,“moreaddresses”isnotsomethingthatneededtobesolved

WHERECOOKIESCOMEFROM

Wheredocookiescomefrom?

•  Mostweb-appwritersaren’ttoclearonthis– “It’spartofPHP”

•  TheycomefromtheHTTPheader

GET / HTTP/1.0 Host: www.example.com Cookie: foo=bar;

200 OKServer: Apache/1.0Set-Cookie: foo=bar2

Whythisimportant

•  Everythinggoesacrossthewireinaconcreteform–  It’snevermagic–  It’salwayssomethingthatfollowsconcreterules

•  Hackerscanmanipulatethisonthewire•  OrhackerscanmanipulatethisfromhosNlesystems

•  Nothingcanbetrusted

ThefailureofRPC

•  Remoteprocedurecall– SunRPC(ONCRPC)withNFS– MS-RPC(DCERPC)withWindows

•  DCOMobjectorientedRPC•  Passedinternaldatabetweenmachinesinvisibly– BlasterWorm– \\machinename– Evenpointervalues

TCPCHECKSUMS

TCPchecksums

•  Detectsall1-biterrors•  Detectsmost2-biterrors– “most”isn’tenough–  Itmeans“some”aren’tdetected

Everystepshouldbeprotected

•  Ethernet/linksareCRCprotected•  PCIetransfersareCRCprotected•  CPUcachesareparityorECCprotected•  Intrachiptransfersareprotected•  RAMisECCprotectedonhigh-endsystems

Howitreallyworks

•  Notsomuch•  Especiallyincheaperdevices•  EspeciallyRAM•  EspeciallypermanenterrorsinRAMcells•  Visiblecomparingpacketswithretransmits–  Errorssmearedacrossadjacentbytes

BadRAMinnon-ECCdevicesisthe#1causeofundetectedTCPerrors

Bit-rot

•  gmail.com->gmakl.com

Bit-rotcomesfromeverywhere

•  Bitsflippedonthenetwork•  BitsflippedinRAM•  Bitsflippedonharddrives

•  Consequence:– Gmakl.comgetssteadystreamofspam

SoluNon

•  Independentchecksums– BitTorrent– Bitcoin– AnythingSHA2

•  Youreallyneedtodothisinyourcustomsolware

•  Googledoeswiththeirinternalstuff

SMALLPACKETS

Thesmallpacketproblem

•  Thesameas“buyinginbulk”problem– Lotsofsmallpacketsmoreexpensivethanfewerlargepackets

•  Typicalsmallpacketproblems– Portscanning– VoIPaudiotraffic– SIP– DNS

Benchmarks

•  Largepacketperformancesayslinleaboutsmallpacketperformance

•  Example:USBEthernet– Fullbandwidthatlargepacketsizes

•  400-mbpsonUSB2.0•  1000-mbpsonUSB3.0

– Noteven100mbpsonsmallpacketsizes•  10,000to100,000packets-per-second•  …ratherthan1,500,000packets-per-second

Ethernetmaxpacketratefor1gbps

•  hnp://blog.erratasec.com/2013/10/whats-max-speed-on-ethernet.html

•  1.488millionpacketspersecondat64-bytesperpacket–  Interframegap,Preamble,CRC,padding,etc.

•  476-mbpsusingminimumpacketsizes•  ISPmeasuredbandwidth!=Ethernetbandwidthatport–  ISPuplinksdon’tincludingEthernetheader,padding,etc.

YourUDPapp

•  ~250kto~700kpackets-per-secondnaïve•  2-millionwithLinuxmulNcoreopNmizaNons– SO_REUSEPORT:manysocketshandles,oneUDPport

– MulNpleEthernetreceivequeues•  7-millionwithmanymorecoresandextremeLinuxopNmizaNons

•  FYI:30-millionifyoubypasstheLinuxkernel

LINUXOPTIMIZATIONHOW-TO

BasicLinux

•  Increasefiledescriptors•  RecompilekernelforopNmizaNons•  EthernetopNmizaNon•  TCP/IPopNmizaNons

Perftools

•  Use“perf”tofindwhereinkernelthingsarestuck

•  Usuallyturnitoff•  E.g.– Turnoffnevilterfor4%

SEND()DOESN’TSEND

send()

•  bytes_sent=send(bytes_to_send)–  Ifsocketisnon-blocking,bytes_sentmaybefewerthanbytes_to_send

– Thereisalimitinoutgoingkernelsendbuffers– Thereisalimitonincomingkernelreceivebuffersontheotherside

•  Ithappensatscale– Youwon’tseeitunNlitmaners– Reallyhardtocreatetestcasefor

WhereIseethis

•  Short/longlinesinemailmessages•  hnp://harryponer.wikia.com/wiki/Splinching--20cf307813b8ac926404b1628ab5Content-Type: application/msword;

name="Prospectus for a Transportation Technologies Incubator v4.doc"Content-Disposition: attachment;

filename="Prospectus for a Transportation Technologies Incubator v4.doc"Content-Transfer-Encoding: base64X-Attachment-Id: e953d37b7ed4bd1f_0.1

0M8R4KGxGuEAAAAAAAAAAAAAAAAAAAAAPgADAP7/CQAGAAAAAAAAAAAAAAABAAAANAAAAAAAAAAAAAAAAAAAqAMAAAAAAACoAwAAAAAAAKgDAAAAAAAAqAMAABQAAAAAAAAAAAAAALwDAAAAAAAAAAgAAAAAAAAACAAAAAAAAAAIAAAAAAAAAAgAACwAAAAsCAAAFAAAALwDAAAAAAAAKg8AAG4BAABMCAAAAAAAAEwIAAAAAAAAAAAAEwIAAAAAAAAJwkAAAAAAAAnCQAAAAAAACcJAAAAAAAAqQ4AAAIAAACrDgAAAAAAAKsOAAAAAAAAqw4AAAAAAACrDgAAAAAAAKsOAAAAAAAAqw4AACQAAACYEAAAaAIAAAATAADUAAAAzw4AABUAAAAAAAAAAAAAAAAAAAAAAAAAqAMAAAAAAABWCgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAnCQAAAAAAACcJAAAAAAAAVgoAAAAAAABWCgAAAAAAAM8OAAAAAAAA

Projectidea:DartmouthSMTP

•  MonitorallDartmothincoming/outgoingemail

•  Count%splinchedemails•  CountamountofTCPreceive-windows-fullpackets

RECV()DOESN’TRECEIVEENOUGH–ORRECEIVESTOOMUCH

recv()

•  bytes_recvd=recv(bytes_to_recv)– Othersidemaynothavesentenoughbytes– Othersidemighthavesenttoomanybytes

•  Ithappensatscale•  Ithappensbecauseofoddsolwareonotherside

WhereI’veseenthis

•  Line-orientedprotocols– HTTP,FTP,SMTP

•  TypicalFTPsolware– AssumesenNrelinehasbeenreceived

•  Shortlineswithout\nthengettruncated•  Remainderisassumedtobestartofnextline

– Assumesnomorethanonelinereceived•  ParsesunNl\n,discardsremainder•  Nextpacketassumesstartofnextline

HTTPservers

•  SendingmulNplerequestsbeforearesponsehasbeenreceived

“pipelining”

•  Meansyoucansendmoredatathanexpectedandit’snotlost

•  Notpipeliningmeansyoucan’t

Projectidea:masscan

•  Idea– MasscanFTPport(21)– Sendtruncatedpacketswithout\nfollowedbyrestofline

– Sendexcesspacketswithdataaler\n– Testhowmanyhaveflawedresponses

TCPISASTREAM

Example:SnortrulesforINTERNALBLUE(WannaCry)

•  Testsforpacketpayloadsthatstartwithstring“SMB”

•  ButTCPisastream–  Icansplitpayloadsarbitrarily–  IcansNckSMBattheendofthepreviouspacketinsteadofthestartofthispacket

– EspeciallysinceSMBsupports“pipelining”

alerttcp$EXTERNAL_NETany->$HOME_NET445(msg:"OS-WINDOWSMicrosolWindowsSMBremotecodeexecuNonanempt";flow:to_server,established;content:"|FF|SMB3|00000000|";depth:9;offset:4;byte_extract:2,26,TotalDataCount,relaNve,linle;byte_test:2,>,TotalDataCount,20,relaNve,linle;metadata:policybalanced-ipsdrop,policyconnecNvity-ipsdrop,policysecurity-ipsdrop,rulesetcommunity,servicenetbios-ssn;reference:cve,2017-0144;reference:cve,2017-0146;reference:url,isc.sans.edu/forums/diary/ETERNALBLUE+Possible+Window+SMB+Buffer+Overflow+0Day/22304/;reference:url,technet.microsol.com/en-us/security/bulleNn/MS17-010;classtype:anempted-admin;sid:41978;rev:3;)

“Packets”arearbitrary

•  Thepacketislayer3•  Layer2andbelowarenotpartofthepacket•  Layer4andabovearenotpartofthepacket–  Inthatwherelayer3boundariesmatchlayer4boundariesispurelycoincidental

Packetsareasingleblockofdata,butof

independentparts

BYTE-ORDER

ntohs()iswrong

•  Youshouldbehandlingbyte-orderthesamewayaswitheveryotherlanguage– n=buf[0]<<256|buf[1];– n=buf[0]*256+buf[1];

•  Neverusentohs()stylefuncNonswhenparsing– Nevercast/overlaypackedstructures

ntohs()iswrong

•  Neverstoreintegersinverted– The‘int’typealwaysmeansinthemachinebyte-order

–  Ifyoumust,thencreateanewtype,suchas“external_int”tohold(possibly)invertedintegers

–  [byte-orderproblemisatypeproblem]•  UseitonlywhendictatedbysocketsAPI– sin.sin_port=ntohs(80);– butIPv6getaddrinfo()getsridofthis

noths()neverworkedanyway

•  Fordecades,Solarisappsmysteriouslyfailedwith“buserror”becausewhilentohs()solvesbyte-order,itdoesn’tsolvealignment

external!=internal

•  Internalbyte-orderisunknownandunknowable–  It’sabstract

•  Externalbyte-orderisknown–  It’sconcrete–  Even:don’tfear“magicnumbers”,becauseit’sthatconcrete•  if(ip_ver==4)…elseif(ip_ver==6)…else…

–  Ifyouchangethevalueina.hfile,yourcodewillfailtointeroperatewiththeotherside

PARSINGISATHING

Wherehackedvulnscomefrom

•  Becauseschoolsdon’tteachhowtoparseinput– …sopeoplecomeupwithadhocsoluNonsthemselves

•  Allthesevulns(liketheoneinWannaCry)comesfromfailuretoparsecorrectly

•  Distrustallinputyoureadfromthenetwork– Assumethesenderisahackertryingtotrickyou–  Validatefirstbeforeusingit– …andstuff

IPV6APPSANDGETADDRINFO()

YourapphastoresolvenamestoIPaddresses

•  Getaddrinfo()doesDNSresoluNon•  AlsoparsesIPaddressesfromtexttobinary•  Nolongerusegethostname()

Usegetaddrinfo()

•  MagicallymakesyourcodesupportbothIPv4andIPv6

Don’tusegetaddrinfo()

•  It’snotthreadsafe– UseonlyfromtheconfiguraNonthread– Maycrashotherwise

•  It’snotscalable– Don’tusewhenusertriestoconfigurethousandsofaddresses

– Don’tuseittoreverse-lookupincomingIPaddressesonaserverinordertologDNSnames

–  Considerusinginet_pton()whenparsingnumericaddresses,maybe

hnps://blog.powerdns.com/2014/05/21/a-surprising-discovery-on-converNng-ipv6-addresses-we-no-longer-prefer-getaddrinfo/

Family

•  freeaddrinfo()•  getpeername()•  Inet_pton()•  Inet_ntop()

WhatisthedealwithDNSanyway?

•  Howlongdoyoucachethenamereturnedbygetaddrinfo(),beforerefreshingit?

•  Whatifitreturnsanerror?Doyouaskforitagain?

•  CanIreuseanoldoneifrefreshingfails?•  WhenabotnettakesdownDNS,doesthismeanyourinternalappfails?

INTERNETSCALEANDASYNCHRONOUS

Howyoulearn

•  Hownetworkingworksatall– Spawnthreads–  recv()/send()withblockingcalls– ThisisbadbecauseitsupportsonlyafewthousandconnecNons

– BecausetheoperaNng-systemcanonlyscheduleafewthousandacNvethreads

Threadschedulerispacketscheduler

•  ThismeansthateveryincomingTCPpacketcausesthethreadassociatedwiththesockettobescheduledasrunnable

•  ServicesexposedtotheInternet– WithmillionsofincomingTCPconnecNons– Withhackerstryingtomessthingsup

Notevenkidding

Asynchronous

•  Onethread– …oronethreadperCPU

•  epoll(),libevent,orlibuv– Selectswhicheversockethaspendingdata– Don’tuseselect()orpoll()astheyaren’tscalableeither

•  Canhandle100,000– Or1-millionwithOStuningandhelyCPU

Project:testwithmasscan

•  masscancangeneratemillionsofTCPconnecNons

•  Icouldgetnginxto450,000TCPconnecNons

ISLANDOFMISFITPACKETS

Projectidea

•  ResponsesfromwrongIPaddress–  BothUDPandTCP–  ShouldbeimpossibleforTCP

•  Checksumerrors•  Constantreplies–  SomeNmesapplicaNonlayer–  SomeNmesunderlyingstack

•  2millionaddressesrespondtoanySYN•  Somany“accelerators”

SOMECODE

Someofmygithubstuff

•  RunsonWindows,Linux,andmacOS•  WrineninC•  Clientsandservers•  Virtuallynohtons()stylefuncNons–  Justforse�ngsin_port

Telnetlogger(server)

•  UsedfortheMiraiIoTbotnet•  LogspasswordsforincomingTCPconnecNons

•  hnps://github.com/robertdavidgraham/telnetlogger/

BINDtkill

•  Simpleclient,sendsDoStobind•  hnps://github.com/robertdavidgraham/cve-2015-5477/

•  getaddrinfo()example– Connectstoallhostsreturnedbygetaddrinfo()DNSquery,IPv4orIPv6

•  Nohtons()stylefuncNons

Heartleech(client)

•  ExploitsHeartbleedtoscrapeSSLcerNficatesfromvulnerablesystems

•  ExamplehowtouseSSL– Warning:needsspecialversionofSSLtocompiletoexploitheartbleed

•  hnps://github.com/robertdavidgraham/heartleech

•  (nohtons()atall)

masscan

•  hnps://github.com/robertdavidgraham/masscan

•  Millionsofpackets-per-second•  MillionsofconcurrentTCPconnecNons•  CustomTCP/IP–  verylimited– Likeyourhomework

HOWBIGIS4BILLION?

Masscandemo

•  masscan0.0.0.0/0–p<something>--banners–rate<something>