1. Headless Host ScanningJan Schaumann 99CE 1DC7 770A C5A8 09A6 0DCD 66CE 4FE9 6F6B D3D7@jschauma

2. Headless Host Scanning -- @jschauma 2 11/19/12 3. Life Lessons: Pen testing cant solve everything.Headless Host Scanning -- @jschauma 3 11/19/12 4. Inside-OutHeadless Host Scanning -- @jschauma 4 11/19/12 5. Distance to Yahoo! ad in feet: 404Construction cost: $357 millionCapacity: 41,915Headless Host Scanning -- @jschauma511/19/12 6. Ballpark numbersDistance to Yahoo! ad in feet: 30927Construction cost: $1.5 billionCapacity: 52,325Headless Host Scanning -- @jschauma 6 11/19/12 7. Life Lessons: Size mattersHeadless Host Scanning -- @jschauma 7 11/19/12 8. It all began some time in January of 2007...Yahoo! Presentation; Confidential11/19/12 9. Initial Problem StatementHey, we need to know which hosts cant deal with the new Daylight Saving Time. Headless Host Scanning -- @jschauma 911/19/12 10. A Systems Administrators ToolboxHeadless Host Scanning -- @jschauma 10 11/19/12 11. A simple tool perl python /bin/shHeadless Host Scanning -- @jschauma 11 11/19/12 12. Initial "Solution"#!/bin/shwhile read host; dossh $host "env TZ=PST8PDT perl -e @t=localtime 1173682800; exit 1 - pop @t"doneHeadless Host Scanning -- @jschauma 1211/19/12 13. Problem Statement Please run this script on these hosts.Headless Host Scanning -- @jschauma 1311/19/12 14. 2nd Pass#!/bin/shwhile read host; doscp script.sh $host:/tmp/ssh $host "/tmp/script.sh"doneHeadless Host Scanning -- @jschauma 14 11/19/12 15. Mensch, nutz doch ein Konfigurationsmanagementtool!Headless Host Scanning -- @jschauma 15 11/19/12 16. Configuration ManagementHeadless Host Scanning -- @jschauma 16 11/19/12 17. Configuration ManagementGood at: keeping state ensuring flat text file propertiesNot so good at: deploying applications executing (one-time) scriptsTerrible at: reporting output of individual commands or scriptsHeadless Host Scanning -- @jschauma1711/19/12 18. Life Lessons: Sturgeons Law (1st adage)"Nothing is always absolutely so."Headless Host Scanning -- @jschauma 18 11/19/12 19. Mensch, nutz doch Deine Logaggregationsinfrastruktur!Headless Host Scanning -- @jschauma 19 11/19/12 20. Log Aggregation makes assumptions about message format frequently unreliable and/or asynchronous adds additional complexityHeadless Host Scanning -- @jschauma 2011/19/12 21. 3rd Pass#!/bin/shwhile read host; docat script.sh | ssh $host "bash s"doneHeadless Host Scanning -- @jschauma 21 11/19/12 22. 3rd Pass#!/bin/shwhile read host; dossh $host "bash s" < script.shdoneHeadless Host Scanning -- @jschauma 22 11/19/12 23. Problem Refinement Run this set of arbitrary commands on our hosts.Yes, on all of them.Headless Host Scanning -- @jschauma 23 11/19/12 24. Wait a second all hosts?$ wtf hostwtf: I dont know what host means!$ a computer connected to a network a virtual machine a container jails, zones, ... chrootHeadless Host Scanning -- @jschauma 24 11/19/12 25. Sturgeons Law (1st adage) "Hosts" on the network"Hosts" in asset database "Hosts" in DNSYahoo! Presentation; Confidential 11/19/12 26. 3rd Pass#!/bin/shwhile read host; dossh $host "bash s" < script.shdoneHeadless Host Scanning -- @jschauma 26 11/19/12 27. 4th PassHeadless Host Scanning -- @jschauma 27 11/19/12 28. Headless Host Scanning -- @jschauma 28 11/19/12 29. Life Lessons: Causality (Duh!)$ w10:45AM up 51 days, 22:19, 11 users, load averages: 976.23, 618.17, 1020.25$ time ls13.354 real0.001 user 0.002 sys$ Headless Host Scanning -- @jschauma29 11/19/12 30. Horizontal Scalability ScanmasterScanslaves Scanslaves TargethostsHeadless Host Scanning -- @jschauma30 11/19/12 31. SSH Complications: assword promptsInteractive connections?Password:$USER@hostnames password:Headless Host Scanning -- @jschauma 31 11/19/12 32. SSH Complications: assword promptsInteractive connections begat password injectors.Headless Host Scanning -- @jschauma 32 11/19/12 33. SSH Complications: assword prompts proc respond { pw } { global pws if {![info exists pws($pw)]} { send_user " (autopw) " stty -echo expect_user -re "(.+)n" stty echo set pws($pw)[exec perl -ne "print pack(q{u}, $_)" tkill(1) cron scheduledkilling of processes)Headless Host Scanning -- @jschauma 4411/19/12 45. Yay, Open Source!Headless Host Scanning -- @jschauma 45 11/19/12 46. Scanmaster Scripts checkhosts(1) run given script on all hosts in input via ssh(1) (in background) scanhosts(1) invoke checkhosts(1) against split input (again in background) defaults to 150 invocations of checkhosts(1) in parallel scanslave(1) starts ssh-agent(1) and adds keys adds tkill(1) crontab invoke scanhosts(1) against split input scanmaster(1) splits input into number of scanslaves chunks ssh to each scanslave and kicks off scanslave(1) in screen session aggregates and post-processes resultsHeadless Host Scanning -- @jschauma 4611/19/12 47. Scanmaster Scripts$ wc -l *330 checkhosts259 scanhosts325 scanmaster179 scanslave 1093 total$ le *checkhosts: Bourne shell script text executablescanhosts: Bourne shell script text executablescanmaster: Bourne shell script text executablescanslave: Bourne shell script text executable$Headless Host Scanning -- @jschauma 47 11/19/12 48. "Every item on the menu at Taco Bell is justa different configuration of roughly eightingredients. With this simple periodic tableof meat and produce, the company pulleddown $1.9 billion last year."http://teddziuba.com/2010/10/taco-bell-programming.htmlHeadless Host Scanning -- @jschauma4811/19/12 49. Miscellaneous Security Issues ssh-agent exposes our keys to target hosts use ForwardAgent=no, ClearAllForwardings=yes ssh-agent on scanslaves exposes our privkeys to anybody withsuper-user privileges on the scanslaves use a scan-only key with from= and various no- restrictions password injector will respond to anything asking for your assword uhm... dont use password auth? use of a "real" user is stupidHeadless Host Scanning -- @jschauma 49 11/19/12 50. Going topless. T.O.P.L.E.S.S.Headless User Requirement #1: selective access (via ssh keys)Headless User Requirement #2: a restricted shellHeadless User Requirement #3: an un-restricted shellHeadless User Requirement #2 and #3: allow arbitrary commands, provided theywere sanctioned by somebody we trustHeadless Host Scanning -- @jschauma 5011/19/12 51. Windows PowerShellHeadless Host Scanning -- @jschauma 51 11/19/12 52. Windows PowerShell ExecutionPolicy Restricted No scripts can be run. Windows PowerShell can be used only in interactive mode. Unrestricted No restrictions; all Windows PowerShell scripts can be run. RemoteSigned Downloaded scripts must be signed by a trusted publisher beforethey can be run. AllSigned Only scripts signed by a trusted publisher can be run.Headless Host Scanning -- @jschauma 52 11/19/12 53. The (TacoBell) SigshNAMEsigsh -- a signature verifying shellSYNOPSISsigsh [-c certs] [-x] [-p prog]DESCRIPTIONsigsh is a non-interactive, signature requiring and verifyingcommand interpreter. More accurately, it is a signature verication wrapper around a given shell. It reads input in PKCS#7 format from standard in, veries the signature and, if the signature matches, pipes the decoded input into the command interpreter.Headless Host Scanning -- @jschauma5311/19/12 54. Yay, Open Source!Headless Host Scanning -- @jschauma 54 11/19/12 55. The (TacoBell) Sigsh$ nd . ( -type f -a ! -path *.svn/* ) | xargs wc -l 50 ./test/sigsh.test.pl36 ./certs/Makele 5 ./certs/README 121 ./doc/sigsh.117 ./doc/Makele 176 ./src/sigsh.sh12 ./README29 ./LICENSE 446 total$Headless Host Scanning -- @jschauma55 11/19/12 56. Yay, Open Source!Headless Host Scanning -- @jschauma 56 11/19/12 57. The (TacoBell) Sigsh$ egrep -v "^[ ]*#" sigsh.sh | wc -l 93$Headless Host Scanning -- @jschauma 57 11/19/12 58. The (TacoBell) SigshEXAMPLESThe following examples illustrate possible usage of this tool.To execute the commands in the le script.bash:openssl smime -sign -nodetach -signer mycert.pem-inkey mykey.pem -in script.bash -outform pem | sigshTo execute the perl code contained in the signed PKCS#7 le code.pem:sigsh -p /usr/bin/perl we have builtin time-bomb powers! multiple people can have signing-power independent of each other we can scan over XXXK "hosts" in approximately 15 minutes we have the flexibility to collect all sorts of interesting dataHeadless Host Scanning -- @jschauma 60 11/19/12 61. Scanning over 400 hosts/second[...]16:09:14: 0 / 30 done - waiting...16:10:16: 0 / 30 done - waiting...16:11:18: 1 / 30 done - waiting...16:12:20: 6 / 30 done - waiting...16:13:22: 20 / 30 done - waiting...16:14:24: 27 / 30 done - waiting...16:15:25: 29 / 30 done - waiting...16:16:27: 29 / 30 done - waiting...16:17:30: 30 / 30 done16:17:30: Scanned XXXXXX hosts in XX minutes and 32 seconds.Aggregating data...Yahoo! Presentation; Confidential11/19/12 62. The Bad this is still a TacoBell sigsh, not a powershell we have not solved the ssh-key access problem (though a leaked ssh-key cannot allow arbitrary commandexecution, only execution of previously approved commands) multiple people can have signing-power independent of each otherHeadless Host Scanning -- @jschauma62 11/19/12 63. Life Lessons: Paretos PrincipleThe "trivial many" The "vital few" Headless Host Scanning -- @jschauma 63 11/19/12 64. Life Lessons: Milton Friedman was right.Theres nothing so permanent asa temporary government programsolution.Yahoo! Presentation; Confidential11/19/12 65. The Ugly Only ~80% of our hosts have sigsh => we need to combine headlesswith "headful" scans. Thus, we currently: scan headlessly (80%, in