Ops for Developers

Download Ops for Developers

Post on 18-Dec-2014




5 download

Embed Size (px)


Ops for Developers, training session, presented by Ben Klang at Lone Star Ruby Conference 6 2012


<ul><li> 1. spkr8.com/t/13191 Ops for Developers Or: How I Learned To Stop Worrying And Love The Shell Ben Klang bklang@mojolingo.comFriday, August 10, 12 </li> <li> 2. Prologue IntroductionsFriday, August 10, 12 </li> <li> 3. Who Am I? Ben Klang @bklang Github/Twitter bklang@mojolingo.comFriday, August 10, 12 </li> <li> 4. What are my passions? Telephony Applications Information Security Performance and Availability Design Open SourceFriday, August 10, 12 </li> <li> 5. What do I do? Today I write code and run Mojo Lingo But Yesterday...Friday, August 10, 12 </li> <li> 6. This was my worldFriday, August 10, 12 </li> <li> 7. Ops CultureFriday, August 10, 12 </li> <li> 8. I am allergic to downtimeFriday, August 10, 12 </li> <li> 9. Its About Risk If something breaks, it will be my pager that goes off at 2am New software == New ways to break If I cant see it, I cant manage it or monitor it and it will breakFriday, August 10, 12 </li> <li> 10. Agenda 9:00 - 10:30 1:30 - 3:00 Operating Systems &amp; Autopsy of an HTTP Hardware Request All About Bootup Dealing with Murphy 10:30 - 11:00: Break 3:00 - 3:30: Break 11:00 - 12:30 3:30 - 5:00 Observing a Running Scaling Up System Deploying Apps Optimization/Tuning Audience Requests 12:30 - 1:30 LunchFriday, August 10, 12 </li> <li> 11. Part I Operating Systems &amp; HardwareFriday, August 10, 12 </li> <li> 12. OS History Lesson BSD, System V, Linux and WindowsFriday, August 10, 12 </li> <li> 13. UNICS Soon renamed Unix (Sep. 1969) Time Sharing System Version 1 UNIX Time Sharing System Version 5 (Jun. 1974) UNIX Sys III 1BSD (Nov. 1981) (Mar. 1978) UNIX Sys V 4.3BSD (Jan. 1983) (Jun. 1986)Friday, August 10, 12 </li> <li> 14. Friday, August 10, 12 </li> <li> 15. Hardware ComponentsFriday, August 10, 12 </li> <li> 16. Common Architectures Intel x86 (i386, x86_64) SPARC POWER ARM But none of this really matters anymoreFriday, August 10, 12 </li> <li> 17. CPU Congurations Individual CPU SMP: Symmetric Multi-Processing Multiple Cores Hyperthreading/Virtual CoresFriday, August 10, 12 </li> <li> 18. (Virtual) Memory RAM + Swap = Available Memory Swapping strategies vary across OSes What your code sees is a complete virtualization of this x86/32-bit processes can only see 3GB of RAM from a 4GB address spaceFriday, August 10, 12 </li> <li> 19. Storage Types Local Storage (SATA, SAS, USB, Firewire) Network Storage (NFS, SMB, iSCSI, AOE) Storage Network (FibreChannel, Fabrics)Friday, August 10, 12 </li> <li> 20. Networking LAN (100Mb still common; 1Gbit standard; 10Gb and 100Gb on horizon) WAN (T-1, Frame Relay, ATM, MetroE) Important Characteristics Throughput Loss DelayFriday, August 10, 12 </li> <li> 21. Part II All About BootupFriday, August 10, 12 </li> <li> 22. Phases BIOS Kernel Bootstrap Hardware Detection Init SystemFriday, August 10, 12 </li> <li> 23. System Services Varies by OS Common: SysV Init Scripts; /etc/inittab; rc.local Solaris: SMF Ubuntu: Upstart Debian: SysV default; Upstart optional OSX: launchd RedHat/CentOS: SysV Init ScriptsFriday, August 10, 12 </li> <li> 24. SysV Init Scripts Created in /etc/init.d; Symlinked into runlevel directories Symlinks prexed with special characters to control startup/shutdown order Prexed with S or K to start or stop service in each level Numeric prex determines order /etc/rc3.d/S10sshd -&gt; /etc/init.d/sshdFriday, August 10, 12 </li> <li> 25. rc.local Single dumb startup script Run at end of system startup Quick/dirty mechanism to start something at bootupFriday, August 10, 12 </li> <li> 26. /etc/inittab The original process supervisor Not (easily) scriptable Starts a process in a given runlevel Restarts the process when it diesFriday, August 10, 12 </li> <li> 27. Supervisor Processes Solaris SMF Ubuntu Upstart OSX launchd daemontoolsFriday, August 10, 12 </li> <li> 28. Ruby Integrations Supervisor Processes Bluepill God Startup Script Generator ForemanFriday, August 10, 12 </li> <li> 29. Choosing a Boot Mechanism Is automatic recovery desirable? (Hint: sometimes its not) Does it integrate with monitoring? Is it a one-off that will get forgotten? Does it integrate into OS startup/shutdown? How much work to integrate with your app?Friday, August 10, 12 </li> <li> 30. Part III Observing a Running SystemFriday, August 10, 12 </li> <li> 31. Common Tools top free vmstat netstat fuser ps sar (not always installed by default)Friday, August 10, 12 </li> <li> 32. Power Tools lsof iostat iftop pstree Tracing tools strace tcpdump/wiresharkFriday, August 10, 12 </li> <li> 33. Observing CPU Go-to tools: top, ps CPU is not just about computation Most Important: %user, %system, %nice, %idle, %wait Other: hardware/software interrupts, stolen time (especially on EC2)Friday, August 10, 12 </li> <li> 34. The Mystical Load Avg. Broken into 1, 5 and 15 minute averages Gives a coarse view of overall system load Based on # processes waiting for CPU time Rule of thumb: stay below the number of CPUs in a system (eg. a 4 CPU host should be below a 4.00 load average)Friday, August 10, 12 </li> <li> 35. When am I CPU bound? 15 minute load average exceeding the number of non-HT processors %user + %system consistently above 90%Friday, August 10, 12 </li> <li> 36. Observing RAM Go-to tools: top, vmstat Available memory isnt just Free Buffers + Cache ll to consume available RAM (this is a good thing!)Friday, August 10, 12 </li> <li> 37. RAM vs. Swap RAM is the amount of physical memory Swap is disk used to augment RAM Swap is orders of magnitude slower Some VM types have no meaningful swap Rule of thumb: pretend swap doesnt existFriday, August 10, 12 </li> <li> 38. Paging Strategies Solaris: Page in advance Linux: Page on demand (last resort) Windows: CrazinessFriday, August 10, 12 </li> <li> 39. When am I memory bound? Free + buffers + cache &lt; 15% of RAM...</li></ul>