the network files, case #53: diagnosing diseases of dns presented by mark minasi [email protected] ...
TRANSCRIPT
The Network Files, Case #53: Diagnosing diseases of DNSPresented by Mark [email protected] for newsletters, audio sets etc
WSV313
Introduction
Both software like AD and humans like us much prefer to refer to network systems by names than by IP addressesAs you know, the thing that translates host names into addresses is DNSSo when DNS fails, anything can break from Facebook to Active DirectoryIn this session, we'll do some quick review and then get intermediate/advanced in how DNS works and how to troubleshoot it
2
Agenda
Review: queries and recursion in DNSExamine a particular query in-depth: ports, TXIDs and moreHow DNS uses UDP versus how it uses TCP
Tracking DNS with Network MonitorDNS details: a Q and three A'sEDNS, Extensions to DNSDNS tools that are way better than nslookup
3
Pretty much all of my important points and what you need to duplicate the demos are all in these slides so don't worry about taking notes
First: DNS Logs
Let's take a look at an actual log from a running DNS serverYou do not get this log by default; rather, you get it by enabling it in the DNS server's properties
4
Clear as a bell, eh?
Actually, it is, once you know how DNS "thinks," under the hood, so in this talk we're going to spend some time making this sort of thing both familiar and readable.
Once you understand DNS packets and protocols, though, all the good tools start making sense.
DNS Queries and Recursion
To make sense of that log, then, we need some more backgroundLet's say that PC1 wants to look up the IP address of "a.bigfirm.com"We'll see two things:
How many queries and how many DNS servers are involved in answering the questionWhat's inside each DNS query
Looking at this simple query shows the same tools and approaches we use for all DNS troubleshooting
7
The DNS Hierarchy: Review
The public DNS system comprises zillions of DNS servers in a pyramid-like hierarchyAt the top are the root servers ("."), which point to the next level downThe next level down are the Top Level Domains (TLDs), like .com, net, .us, .biz, .tv etc … all of them have DNS servers and they point to the next-lower level of domainsCreate your own generic TLDs (gTLD) for a mere $185K
8
The Public DNS Hierarchy
9
. (root)
.com .org .net .uk.gov .ca
minasi.com microsoft.com doj.gov
hq.minasi.com waco.doj.gov
test.minasi.com
mswatch.doj.gov
Top level domains
Second level domains
“sub-domains” or “child domains”
The DNS Hierarchy: Review
The next level down are the domains that organizations and individuals use, like minasi.com, manybooks.net, google.cn, bigfirm.biz and so onAnd as you know, organizations further subdivide their domains with subdomains/child domains like technet.microsoft.com… those subdomains need DNS servers as wellA single DNS server can serve many domains
10
Quick Quizwhat's involved with DNS server setup?
Suppose I set up a DNS server inside my home's network behind some cheap NAT router, and the DNS server has an address like 10.1.1.17I do no other configuration than to simply enable the DNS role on the serverI then query that DNS server to resolve, say www.yahoo.com… can my new DNS server resolve it?
11
Recursion and DNS Queries
PC1 finds its local DNS server (call it ISPDNS), which is either configured statically or via DHCPPC1 asks ISPDNS to look up the IP address for a.bigfirm.comISPDNS goes to the top of the DNS hierarchy first, asking one of the 13* DNS root servers, "what's the IP address for a.bigfirm.com?"
12
* (it's not really 13 but we'll see that in a bit)
Recursion and DNS Queries
The root servers have no time to do ISPDNS's work, so they brush it off, saying "I dunno… why not go ask the .com DNS servers? There are 13 of them – here are their names and addresses"So ISPDNS takes those addresses and asks one of the .com DNS servers, "what's a.bigfirm.com's IP address?"
13
Recursion and DNS Queries
The .com DNS server isn't about to do ISPDNS's job either, and replies, "I dunno… why not ask bigfirm.com's DNS servers? Here are their names and addresses"ISPDNS now asks one of bigfirm.com's DNS servers, web2.minasi.comWeb2.minasi.com actually has a copy of all of the bigfirm.com DNS info on its hard disk, and answers the questionNow ISPDNS can answer PC1
14
Review: "Authoritative"
The root and .com servers knew where to find the a.bigfirm.com record, but they did not have the recordSo DNS searches until it finds the DNS server that contains a copy of the bigfirm.com zone right on its hard driveThat DNS server is said to be "authoritative" for a.bigfirm.com; in other words, it was the first server that didn't have to "guess" where it was
15
So How Many Queries?
PC1 -> ISPDNSISPDNS -> rootroot -> ISPDNSISPDNS -> .com DNS.com DNS -> ISPDNS
ISPDNS -> web2.minasi.comweb2.minasi.com -> ISPDNSISPDNS-> PC1
16
Inside a Query: Ports and TXIDs
Let's look at just one of those queries, the one from ISPDNS to web2.minasi.comISPDNS chooses a "transient port," a TCP or UDP port above 1024, and asks web2.minasi.com a question from that port to the other server's port 53ISPDNS also keeps track of the question – because DNS servers often have many outstanding questions – by assigning a random "transaction ID" or TXID
17
ISPDNS
web2.minasi.com
What's the IP address for a.bigfirm.com? Send it to my port 3351 and specify transaction ID (TXID) 279 when you do.
"Answer: 73.165.73.5"
sent to port 3351, TXID 279
(The port number and TXID are random numbers with values ranging up to 65,535.)
18
TCP and UDP
DNS is sort of unusual in that it's a protocol that is equally capable of functioning over TCP port 53 or UDP port 53What makes it even more unusual is that for most of its work, DNS heavily favors UDP, partly because of the sheer volume of DNS traffic and in particular the load on the root servers
19
TCP and UDPside-effects: the 13 conundrum
Ever noticed that you never see more than 13 DNS servers, even on a big site?It's because early RFCs (883 and 1035) mandated a maximum packet size on UDP DNS communications of 512 bytes, and that's just about a safe size to store 13 host names and addresses(That's only basically a safe size, there are exceptions)
20
TCP and UDPsecondary effects: firewall troubles
512 byte UDP packets should never fragment, and most DNS traffic is UDP, leading to some firewall rules like
If it claims it's a DNS packet but it's fragmented, block itIf it claims it's a DNS packet but it's TCP, block itIf it claims it's a DNS UDP packet but it's larger than 512 bytes, block it
We'll see some effects of this later
21
Okay, One Quick Example…(based on a true story)
On Monday, folks at Bigfirm.com could resolve yahoo.com addressesTuesday, no more… but they could resolve all other Internet addressesSo what happened?
22
Answer
A few years ago, Yahoo briefly added a 14th publicly-advertised DNS serverThat pushed them over the top from a 512 byte UDP packet, forcing any DNS queries for anything.yahoo.com to use TCPBut Bigfirm's firewall folks had never built a "DNS using TCP is OK" rule, thinking that simple DNS lookups never need TCPI've actually seen this happen a few times both on inside and outside DNS
23
The Magic Troubleshooting Keywait, don't run away…
The way to crack DNS problems is oftimes to drill down to the actual network trafficThe logs (if you check the boxes) are pretty goodBut the ultimate answer is to use Network MonitorHonest, it's not that bad
24
Get Ready to Netmon…
Create a Server 2008 R2 system(the free evaluation copy works fine)Add the DNS rolePoint the server to itself for DNSAnd then let's cut down the network chatter…
25
Shutting down IPv6just to keep things clean
Simplify the NIC list and some of the network chatter by zapping IPv6 entirelyNo, not forever, just for testing, and the GUI can't really do this… you need the Registryreg add hklm\system\currentcontrolset\services\tcpip6\parameters /v DisabledComponents /t REG_DWORD /d 255Needs reboot; zero it to re-enable
26
Then, start up Network Monitor and see a screen that looks something like this (without the Teredo and Isatap NICs):
Select the NICs you care about and whether or not you want "promiscuous mode"
Phase 2: window cleanup
Now to work…
Do whatever you want to do to try out DNS; a simple ping –n 1 a.bigfirm.com is fine from a command prompt on Server(Remember to first do a "ipconfig /flushdns" beforehand so you get DNS traffic)Clean up the columns to your liking – I zap "process," "time offset," and "TimeDateLocalAdjusted"AND you want to remove the clutter, so it's time for filters
29
The Big Netmon Magic
Build a DNS-only filter:Click in the "Display Filter" text field; it's a "pane" in the Netmon windowsType "DNS" and click "Apply"
This says, "only show me packets that are recognizably part of DNS communication"Things then clarify….
30
Just the good stuff. 31
Sidebar: Root Servers
Note the traffic to 192.203.230.10… it's not a non-routable address, it's one of the 13 IP addresses where you can find the root serversIn actuality there are (as of 28 May 2012) 312 root servers sharing those addressesYou can get more details on them at www.root-servers.org… scroll to the bottom of the page to get the actual root server total
32
Drilling Down Further
Here, we're resolving "a.bigfirm.com," so we seeA request to a root server and responseA request to a .com server and responseA request to the bigfirm.com DNS server and response
So let's look at the details and how they're formatted by DNS
33
DNS Details: Q & The Three A's
Every DNS packet has zero or more of four parts:"Question" section"Answer" sections: the answer"Authority" sections (Netmon calls them "Name server"): relevant name servers"Additional" sections: extra information, answers to questions raised by the original question
34
Query to Root
Question: "hey, root, ever heard of a.bigfirm.com?"
One question, no answers, no authority ("Name ServerCount," no additionals
The Question!
35
Response from Root
36Response: "um, no, but you should next check the .com DNS servers – here are their names and IP addresses"
The question count just parrots back the question.
There is no answer.
The "authority section" offers hints about where to ask the question NEXT with the .com DNS server names.
The "additional" section saves you the trouble of having to look up their IPv4 and IPv6 addresses.
Next…
DNS server makes an identical query for an A record for a.bigfirm.com, but this time to a .com DNS serverThat .com DNS server will respond with the names of any DNS servers for "bigfirm.com"The QAAA tally will be similar to before – 1Q, no A's on the question, 1Q, no answer, two authorities (bigfirm has only two DNS servers), two additional (IP addresses)
37
Response From .com DNS Svr
38
"I don't have the answer, but go ask web2.minasi.com, it will be able to answer your question"
Finally…
Now that the system knows where to find the DNS server for bigfirm.com, it queries thatThen the response arrives, and now the original DNS query is resolvedThe general approach with Netmon is to build and test a properly working queryKeep that as a reference and compare it when examining a troubled system
39
Example 2: DDNS Registration
Next, here's how we'd tackle an AD-related DNS annoyance: dynamic DNS registrationDynamic DNS registration fails either because of security (an AD issue, if the zone is AD-integrated) or a DNS failureSo try out a dynamic DNS registration on Network MonitorHere are the steps
40
DDNS registrationQuery for SOA record for domain
Query to local DNS serverResponse from local DNS server(Why isn't there the query to root and .com or other TLD?)SOA returns name of "primary" DNS server
Query for IP address of primary DNS server
Query to local DNS serverResponse from local DNS server
DDNS RegistrationDDNS request to primary DNS serverSuccess/failure response
41
Again, get a dynamic DNS registration working, then use its structure to examine what happens in a failed registration
Extensions to DNS
a 2008 R2 issue, sort of
Understanding Extended DNS
You've probably noticed by now that DNS needs a bit of modernizationDoing that, however, means changing protocol format and that could break tens of billions of network operations world-wide So 1999 introduced RFC 2671, "Extension Mechanisms for DNS" or "EDNS"Windows DNS has supported it since 2003, but it's been blamed (wrongly) for problems in 2008R2, so here's the story
43
EDNS Goals
Original DNS leaves seven bits for flagsAll but one are used up nowEDNS creates space for more flagsUDP limits of 512 bytes are goofy in today's InternetEDNS lets EDNS-aware DNS servers negotiate larger UDP packet sizes(Remember why UDP is so important to DNS and the annoying 13-server limit)
44
How EDNS Works
EDNS-aware DNS servers want to find other EDNS-aware serversAgain, any such method mustn't break EDNS-dumb DNS serversAnswer: always add an extra query record called an "OPT" record which shows up in the "additional" sectionIf the responder answers the OPT query, it's EDNS-aware; otherwise, it just ignores it
45
Example
I created a host "a.bigfirm.com" and gave it 50 A records, so there's no way the "A" record query for a.bigfirm.com can fit in 512 bytesAs my DNS server uses EDNS, however, it can stay with UDPHere are some bits from the Netmon trace of the lookup and then a similar lookup without EDNS
46
Original Query OPT Section
47
Response, Part 1
48
OPT Response on EDNS System
49
Compare w/non-EDNS
50
Note the "DnsOverTcp" protocol reference
EDNS-Related Problem
Suppose EDNS negotiates a UDP packet larger than 512 bytesThen suppose it runs into one of those routers with a stupid firewall ruleResult: you can't resolve things like Yahoo, Microsoft or the likeThe problem is the firewall, but EDNS gets blamed for it
51
Supposed "Workaround"
dnscmd /config /enabeednsprobes 0(Note: I strongly recommend you not do this!)This causes your DNS server to never offer OPT records, but it doesn't stop it from responding to themOnly 2008R2 difference is that now R2 DNS servers have probes set to "1," not "0"Don't disable EDNS probes, find out what firewall or router is causing the problem
52
Why Bother with EDNS?several reasons, actually
On 31 March of this year, a really big thing happened: VeriSign finished signing the .com domain, paving the way to making DNS hijacking flatly impossible via DNSSECThe 512-byte "soft" limitation is silly in 2011DNSSEC and OPT let us get past these old problemsBut you need EDNS to make those things possible
53
Tools
Some DNS test tools, in brief
Dump Nslookup, Get DIG
Windows comes with NSLOOKUP, but it's got any number of problemsThe non-Windows world has been using a better tool called the "Domain Internet Groper" or "DIG," and you can do that as wellGo to http://www.isc.org/downloads and get the latest version of BINDExtract its files and keep the DLLs, dig.exe and dig.html – put them all on the path
55
Basic Dig Syntax
dig record [@dnsserver] [recordtype] [+option1, +option2…]examples:dig www.bigfirm.com
queries for www.bigfirm.com's A recorddig bigfirm.com mx
Gets bigfirm's MX record
56
More Dig Examples
dig minasi.com mx +norecurseasks the DNS server not to recurse and to just respond with what it knows (note that the server may choose to ignore that command)
dig minasi.com mx +traceTells dig to do the recursion and track every step along the way
dig –hGets help on other Dig options
57
Some Dig Options
+vc: force TCP+novc: force UDP+dnssec: request DNSSEC-related records+fail: don't try next DNS server in search list if the first fails
58
Basic DIG
NetMon Frame for Comparison
60
DIG's Reported Status Values
On ->>HEADER<<-, you'll see "STATUS"NOERROR: no errorNXDOMAIN: "no such record" query failSERVFAIL: some DNS server configuration errorNOIMP: "not implemented," server doesn’t understand somethingREFUSED: query refused by queried server
61
Nice DNS diag tool
dnslint /d domainnameIf split-brain, specify the DNS server to ask with /s: dnslint /d domainname /s dnsipex: dnslint /d bigfirm.biz /s 192.168.0.2Or check a DC’s SRV records:dnslint /ad /s localhost /vAdd /y to automatically overwrite old outputKB 231045 has download link
62
DCDIAG and DNS
Offers (since 2003 SP1!) a series of useful tests targeted at DNS and ADBasic syntax:dcdiag /test:DNS [/e] [/dnstestoption1]….Be careful about /e… it means to run those tests on every DC in the forestUses a lot of remote control and therefore requires RPC access
63
DCDIAG
Simplest command:dcdiag /test:dns /v
pings DNS server, checks it's in ADChecks access to forwarders or, if no forwarders, the root serverscreates a dynamic entry in DNSLooks for certain SRV records
IPv6 can throw false warningsExternal DNS servers can also
64
More Resources
My DNS articles in Windows IT Pro over the yearsMy newsletter 30 (how to set up an AD-friendly DNS subsystem)Newsletter 31 (Island DNS)My presentation at 2010 TechEd on DNSSEC
65
Thanks!
Thank you for staying to this last sessionPLEASE take a moment, do an evaluationTry out Netmonning your DNS!Get the free tools and get comfy with 'emI'm at [email protected] can find my newsletters, online forum, seminar information at www.minasi.comSee you next year!
66
SIA, WSV, and VIR Track Resources
Talk to our Experts at the TLC
#TE(sessioncode)
DOWNLOAD Windows Server 2012 Release Candidate
microsoft.com/windowsserverHands-On Labs
DOWNLOAD Windows Azure
Windowsazure.com/teched
Resources
Connect. Share. Discuss.
http://northamerica.msteched.com
Learning
Microsoft Certification & Training Resources
www.microsoft.com/learning
TechNet
Resources for IT Professionals
http://microsoft.com/technet
Resources for Developers
http://microsoft.com/msdn
Complete an evaluation on CommNet and enter to win!
MS Tag
Scan the Tagto evaluate thissession now onmyTechEd Mobile
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to
be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS
PRESENTATION.