dovecot imap server · rackspace • rackspace email uses dovecot to serve imap for over a million...
Post on 14-Oct-2018
283 Views
Preview:
TRANSCRIPT
Dovecot IMAP Server
http://www.dovecot.org/
Date: September, 2009
Rackspace
• RackspaceEmailusesDovecottoserveIMAPforoveramillionpaidmailboxes– MSExchangealsoavailable
• RackspacehassponsoredDovecotdevelopmentforyears– Andemployedmefull@meforyear2009
Overview
• Dovecothistory&howAppleusesit• Dovecotfeatures• IMAP&Dovecotperformance
• Troubleshoo@ng• Futurefeatures
WhatisDovecot?• Dovecotis
– IMAPserver
– POP3server– LocalmaildeliveryagentwithSievefiltering– Managesieveserver
– LMTPserver(v2.0+)
• DovecotisNOT– SMTPserver
• Soitneitherreceivesnorsendsmailsdirectly
Dovecot
Pictures from Wikipedia, by Cyril Thomas and Carcharoth
History
• DovecotdesignwasstartedaroundJune2002• FirstreleasewasJuly2002• Late2003aredesignstarted• v1.0.0releasedApril13th2007• v1.1.0releasedJune21st2008• v1.2.0releasedJuly1st2009• v2.0betashopefullythisyear
WhyDidAppleSwitchtoDovecotfromCyrus?
•
Apple’sDovecot• Mytestversion:WWDCdeveloperspreview• dovecot‐‐versionsays:1.1.14apple0.5• Applepatches:
– OpenDirectoryauthen@ca@on– Mul@pleconnec@onspermailprocess
• Configura@onin/etc/dovecot/dovecot.conf– Serveradminchangessomese_ngs
– Defaultsmoretowardsperformancethansecurity
Features
• O`enhasbeaerperformancethancompe@@on.– Op@mizedforminimizingdiskI/O(index/cachefiles)– Hos@ngmyownmailson10yearsoldSparchelps
• Highlyconfigurablefordifferentenvironments– StandardmboxandMaildirwithtransparentindexing(externalmailboxmodifica@onsareok)
– dbox:Dovecot’shigh‐performancemailboxformat– Manydifferentwaysofclustering
– Extremelyflexibleauthen@ca@on• PoseixandEximsupportDovecotforSMTPAUTH
Features• Admin‐friendly/self‐healing
– Allerrorsarelogged– Understandableerrormessages
– Improvedconstantly(toreducemyemailload)
– Detected(index)corrup@ongetsfixedautoma@cally
• file_dotlock_create(/home/@mo/Maildir/dovecot‐uidlist)failed:Permissiondenied(euid=1000(@mo)egid=1000(@mo)missing+xperm:
/home/@mo)
• chown(/home/@mo/Maildir/.box,‐1,0(root))failed:Opera@onnotpermiaed(egid=1000(@mo),groupbasedon/home/@mo/Maildir)
Authen@ca@on
• Passwordanduserdatabasesepara@on– Passdbforverifyinguser’spassword– Userdbforlookinguphowtoaccessmailbox
• Supportforalmosteverything:SQL,LDAP,PAM,checkpasswordscripts,etc.– Everythingisconfigurable(e.g.fullSQLqueries)– Supportsmul@pledbs(e.g.system+virtualusers)
• Authmechanisms:PLAIN,CRAM‐MD5,DIGEST‐MD5,Kerberos,OTP,etc.
• Passwordschemes:Plaintext,CRYPT,MD5,SHA1,SHA256,SSHA,SSHA256,etc.
Authen@ca@onCache
• Passdbanduserdblookupscanbecached• Passwordchangesareautoma@callydetected:Ifauthisunsuccessful,andpreviousauthwasa) successful:douncachedpassdblookupb) usuccessful:faillogin
• Nega@vecachingcanbedisabled– Userdoesn’texistcaching– Passwordfailures(v1.2+)
• Avoidsaneedforimapproxywithwebmails?
Maildir
• Apple:/var/spool/mail/dovecot/<user‐id>/• Maildirbasics:
– Onefile=onemail
– Filenamegloballyunique– Messageflagsstoredinfilename
• 1250461029.M8247P5745.host,W=1279,S=1243:2,S– W=Virtualmessagesize(CRLFlinefeeds)
– S=Physicalmessagesize(exactlythesameasindisk)–forspeedingupquotarecalcula@on
– :2,justmeans“version2”andflagsfollowthecomma.S=Seen
• Messagesmustneverchange!
Maildir++DirectoryLayout
• Maildir/‐INBOX– cur/,new/,tmp/
• Maildir/.foo/–foldercalled”foo”– cur/,new/,tmp/
• Maildir/.foo.bar/–foo’schildfolder”bar”– cur/,new/,tmp/
• ’.’beginsallfolderdirectorynamesandseparateshierarchies
MaildirDirectories
• Savingmessages:– firstmailiswriaentotmp/
• Onceinawhileoldfiles(fromcrashes,etc.)aredeleted
– mailismovedtonew/tofinishsaving
• Dovecotlooksformailsinnew/andmovestocur/– Scanningnew/isfasterthanscanningcur/– Socur/willeventuallycontainallmessages
DovecotFiles
• dovecot‐uidlistmapsfilenamestoIMAPUIDs• dovecot‐keywordsmapsa..zflagsinfilenamestoIMAPkeywords(aka.customflags,labels)
• subscrip:onstracksIMAPsubscrip@ons
Nostateislostifdeleted:
• dovecot‐uidvalidity*‐forgenera@nguniqueIMAPUIDVALIDITYvalues
• dovecot.index*‐Indexfiles• maildirsize–Tracksquotausage
IMAPProtocol
• Baseprotocoliscomplex–difficulttoimplementitcorrectly(bothclient&server)
• Flexible–manydifferentwaystoimplementaclient(online&offlineclients)
• Extensible–therearealotofextensions– Clientsrarelysupportmorethansomebasicextensions,suchasIDLE.
– Thunderbirdv3addssupportforseveralnewextensions,suchasCONDSTORE.
ImapTestIMAPServerTester
• WriaenoriginallyforDovecotstresstes@ng– Foundalotofcrashes,hangsandmailboxcorrup@ononotherIMAPserversaswell
• TestsIMAPservercompliancewithscriptedtestsanddynamicrandomstresstes@ng.
• DovecotiscurrentlytheonlyIMAPserverthatfullypassesallofImapTesttests.• PandaIMAPisprac@callytheretoo
• Mostotherserversfailinmanydifferentways.• hap://imapwiki.org/ImapTest
OfflineIMAPClients
• Typicallydownloadnewlyseenmessages’bodiesonceandcachethemlocally
• O`encanbeconfiguredtodownloadimmediatelyvs.downloadwhenreading
• Someuseserversidesearches(Thunderbird)andsomedon’t(Outlook–ifsomemessageshaven’tbeendownloaded,thosearen’tsearched)
• Usuallyalsofetchmessages’metadataonce(headers,receiveddate)
• Server‐sidecachingmayhelp,butnotthatmuch– It’sextradiskI/O‐>morelikelyjusthurts
OnlineIMAPClients
• Webmailso`enkeepaskingforthesameinforma@onoverandoverandoveragain
• Pineandsomewebmailscachewhatthey’vealreadyseen,butnotpermanently
• Mua(withoutlocalcache)andsomeothersfetchallmessages’metadataevery@mewhenopeningamailbox
• Cachingisveryuseful,butdifferentclientswantdifferentmetadata
IMAPServerPerformance• Difficulttobenchmark• Dependsalotonclients:Whetherclientsusealocalcachemakesahugedifference.– Onlinevs.offlineclients
• Whatdatatoindex/cache?• SPECmail2009addssupportforIMAP
– EmulatesdifferentIMAPclients.Clientamountsareconfigurable.
– Theonlybenchmarkgivingrealis@cresults.– Publishedresultsallrunondifferenthardware‐>resultsunusableforcomparingso`ware
DovecotCacheFile
• dovecot.index.cachefiles• ThemainreasonforDovecot’sgoodperformance• Dynamic:cachesonlywhatclientswant.
– Specificmessageheaders(From:,Subject:,etc),– MIMEstructureinforma@on,– Sent/receiveddate,etc.
• Cachingdecisionsforeachfield:“no”,“temporary”,“permanent”
• Unusedfieldsdroppeda`eramonth.• Cacheddataneverchanges(IMAPguarantees)• Cachefilegets“compressed”onceinawhile• O`enabout10‐20%ofmailboxsize
DovecotIndexFiles
• dovecot.indexcontainsmessages’metadata– IMAPUniqueIDnumber(UID)iden@fiesmessages
– Flags(\Seen,\Answered,keywords,etc.)– Extensiondata:mboxfileoffsets,cachefileoffsets,modseqnumber(v1.2CONDSTORE),etc.
• Lazilycreated/updatedsincev1.1– dovecot.index.loghasallthelatestchanges.dovecot.indexisupdateda`er8kBofnewdatahasbeenwriaentothe.log
DovecotIndexFiles
• dovecot.index.logisamailboxtransac@onlog– Somewhatsimilartodatabases’transac@onlogsorfilesystemjournals.
– Containsallchangestobedonetodovecot.index.• dovecot.indexisreadtomemoryonceandthenupdatedfromdovecot.index.log– VeryefficientwithNFS/clusteredfilesystems!– Veryefficienttofindoutwhatchangesanothersessionhaddone!
Plugins
• DovecotpluginscanhookintoalmostanythingandmodifyDovecot’sbehavior.Someexis@ngfeaturesimplementedasplugins:– AccessControlLists– Quota– Fulltextsearchindexes– Readingcompressedmbox/maildirfiles
• CanaddnewIMAPcommands• Implementnewmailstoragebackends(virtual,SQL,IMAPproxying)
DovecotClustering
• Twodifferentwaystodoit:• Globallysharedfilesystem
– ManyIMAPservers,eachabletohandleanyuser
– NFS,clusterfilesystems
• Sharding– Eachuser’sdataindifferentservers
• maybemirroredto2‐3servers
– IMAPproxyforwardsuserstocorrectserver(s)
AppleClustering
• I’veonlygoogledthisinforma@on..• Xsan,clusterfilesystem
• Mul@plemailserversconnectedtoXsan– Ac@ve‐ac@vesetup– Loadbalancingwithhardware,DNS,..?– Performanceprobablybestifuserusuallyredirectedtothesameserver• Orifnotuser,atleastthesameIP
Troubleshoo@ng
• Logs!Dovecotlogsallerrors!• top• rawlog• dtruss
DovecotProcesses
• Something’sslow?Isolateittoaspecificprocessfirst,thenusee.g.dtruss:
• dovecot–masterprocess,createsallotherprocesses,alllogginggoesthroughit
• dovecot‐auth–ODlookups• imap‐login,pop3‐login–acceptsnewconnec@ons,handlescommandsun@lsuccessfullogin,SSLproxyingevena`erlogin
• imap,pop3–post‐loginhandling
ClientTroubleshoo@ng
• LookattheIMAP/POP3protocoltrafficbetweenDovecotandclient– Dovecot’srawlogtool
• worksalsowithSSLconnec@ons– SomeothernetworksniffersuchasWireshark
• imap/pop3_client_workaroundsse_ngsnotenabledinApple’sdefaultconfig(?)
v1.2NewFeatures
• Virtualmailboxes(searchviews)– ”Allunreademailsinallmailboxes”
– Allmessagesinallmailboxes(exceptTrash)• VirtualPOP3INBOX• Forsearchingmessagesfromallmailboxes• gmail‐likeconversa@onviews
• Userscansharemailboxestoeachothers– IMAPACLcommands
• NewIMAPextensions,performanceimprovements
Dovecotv2.0
• Somenewfeaturesalreadyimplemented:– Redesignedmasterprocess
• Easytoaddexternalservices,e.g.ManageSieve– Redesignedconfigura@on
• Local/remoteIP/mask‐specificconfigura@on– SSLcerts
• Allowchangingconfigdatasource(e.g.SQL?)– LMTPserverandproxy– dsync:Realiablyandefficientlysynctwomailboxes(e.g.viaSSH)
– dbox–highperformancemailboxformat
Dovecotv2.x
• Featuresnotyetimplemented,buthopefullywillbebytheendofthisyear:– Indexfileimprovements
• Nolocking(withatomicappends)
• Smallchecksumsallaroundfordetec@ngcorrup@on• Ingeneralmakethecodesimplerandmorerobust
– Mul@‐masterreplica@on• dboxcloudstorage(forsomeexis@ngcloudAPI(s)?)
• Indexsharing/replica@onbetweenservers
Ques@ons?
top related