Don’t Mind the GapDoing DevOps in an Air-gapped World

Galen EmeryFederal Solutions Architect3/8/2017

Who am I?I work with the Federal Government and IntegratorsDoD, Civilian, or IC? Yes.I help users understand how to build and structure their infrastructure (using Chef) to solve their problemsFormer Windows Server AdminBeen with Chef since March 2014Born and raised in Seattle (Nathen, you’re wrong about Seattle crab)

What we coverWhat is “air-gapped”?What issues do we encounter?AssumptionsHow to solve it (technical)How to solve it (process)What about … ?Food for Thought

What is air-gapped?air-gappedadjective(of a computer) having no direct connection to the Internet or to any other computer that is connected to the Internet, for security reasons."a USB drive or other hardware approach would be required to infect the air-gapped machine"

What about mitm?• Corporate man-in-the-middle• Terminates SSL sessions and re-initiates• Often includes packet inspection• Requires you to trust its generated certificates• May still include a firewall blocking access to sites

Maybe its just a firewall• Restricts access to:• Github or gist.github.com• slack• Pastebin• Rubygems.org / some other artifact repository• etc

No restrictions?• Congratulations!• You might still have a

firewall/proxy, its just permissive

• Do you trust those systems will be up when you need them?

Issues

Getting stuff into production• How do we “cross the gap”?• How do we distribute it?• How do we ensure its integrity?

Process?!You want to talk about PROCESS? You kidding me?!

I just hope we can make one change!

VelocityHow do we achieve velocity if we burn a DVD for every change?Or if a proxy is inspecting all of our packets, every time we download a package?

Okay, I’m not air-gapped but…• I have a firewall between my systems and the internet• I have a proxy that inspects all traffic and slows me down• I don’t have {level of access necessary} to make changes• Tool {X} does not understand authenticating proxies

Ground Rules

You know that I know that you know…• How you get code into your high-side environment

Everything is code• If its source code, it goes into SCM• If it is an artifact, it goes into artifact store, and has a checksum• We can (and do) write tests• We build a pipeline

Solution (High-Level)

Go from Low to High

Assume you don’t have internet• Even if you do• Build this into your pipeline• Test this on the low side• Your systems should NOT assume internet access• Most tooling by default assumes it does

Do this work on the low-sideBuild your pipeline in a way that removes internet access for your systems, they must grab their code and artifacts locallyIf you can build your infrastructure without internet access in Dev, you can absolutely do it in Prod

Protect the Data, not the InfraThe data is sensitive, not the infrastructure itself. Keep the data in the high-side, but ensure that you build your infrastructure with the same code in Dev and Prod

WorkstationThis is your “loading dock” for the rest of the infrastructureEverything comes through hereIt needs:• A way to serve files (directly or indirectly)• A way to create artifacts (zip, tar, etc)• built programmatically

WorkstationUse the workstation to:• Stand up your artifact repo• Stand up your configuration management infra• Publish your artifacts• Run tests

Workstation (Chef Example)Bring in to the workstation a zip/tar with the following• ChefDK• A FTP/SSH/SCP Server binary (if it doesn’t already exist on your box)• Your cookbooks• Any extra gems necessary• Chef packages (client, server)

Use Chef Zero to stand up your FTP Server, populate it with the artifacts and prepare it for use in the rest of the process

What about dependencies?I could download, transfer, attempt to install, download, transfer, attempt to install..

I could skip using the gem and instead write it myself

I can create a full gem mirror of all 80,000+ gems on rubygems.org

I can install what I need into a directory, and then move that directory over as an artifact

Simplest is to create an artifactI used “gem install –i $PATH $GEM” && tar cf $PATHIts not pretty but it worksBenefit is: I have a moment-in-time artifact of the gems I’m usingIdeally you’d create this at the end of your development pipeline

You should be using an artifact repo• Use something that can store and manage your artifacts• You can version your artifacts• Often supports the correct dependency structure for your artifacts

Once you have Workstation + Artifact• You can setup your configuration management• Use a tool, running from your workstation to setup the infrastructure

you need

• Example• Chef Provisioning SSH• Doesn’t require internet access• Can bootstrap Chef from a FTP or SCP server

Everything is set, right?• Have workstation, Config Mgmt + Artifact store

• But I haven’t told any of my infrastructure that it shouldn’t reach out to the internet for X, Y or Z

Tell your systems to stay inside• Remove unreachable Satellite repositories• If using ruby, remove rubygems from your sources. Add your artifact

store• If using Chef, update berksfile to an internal supermarket• If using X, update Y to Z

• Don’t do this manually, do it with code and test it on the low side!

What if I have a proxy instead?Most of the tooling supports HTTP_PROXY and HTTPS_PROXYSome of it does notIt is often easier to design the system to assume no network access than to keep fighting proxiesThat said, if you can poke holes out to slack, rubygems, github, chef, etc you’ll be much happier (unless they go down)

Changing the Process

Do this in Dev• Do not wait for Prod to test your systems without internet access• Along these lines, harden your Dev systems to the same standards

as Prod

Create and Use• CI/CD Pipelines• Unit and Integration Tests

Identify where you assume risk• If you let users ssh/rdp into production, that is where the risk lies• If you only let users make changes to production through a pipeline,

that’s where the risk lies• If the risk lies in a pipeline and someone else has to approve code

changes, the risk of any single change/actor is much lower

Imagine This Scenario• Nobody can ever log into Production• All changes flow through a pipeline that tracks who committed,

reviewed and shipped the change• All changes are tested through Dev, QA, etc before deploying to Prod• We build a new Prod every time we make a change

What About … ?

My laptop can’t reach X• If you can’t reach GitHub, or Slack, etc ever; not just in Production

that’s an issue

Security won’t let me!

• Work with them. They are a vital component of your business (or should be)

• Identify where the risk is. The risk is with the data, not with the user.

• Consume Risk in Dev, not in Prod (Fail Fast)• Progress is coming

How often do we update our packages?

Where are the gates?

What are the metrics?Velocity: How fast we shipEfficiency: How good we are at shippingRisk: How long does it take us to fix

Food for Thought

Alaskan King Crab• Biggest crab, legs alone can be a meal• Served steamed or chilled with drawn butter and lemon• Fishing Season:• October• January

Dungeness Crab• Similar size to blue crab, hard shell• Served steamed or chilled with drawn butter and lemon• Fishing Season:• Starts in November• Ends in June/July

Blue Crab• #CrabFeast• Old Bay• Fishing Season:• Starts In April

Proof of Concept codewww.github.com/galenemery/chef-mreWant a USB Key with it?

Google “stuxnet” or “vault 7”If you asked for a USB Key, please let me know who holds your clearance so we can have it revoked

ACKNOWLEDGEMENTSRobb KiddChefConf 2014: Mind the Gap: Deployments with Chef in unforgiving Environments

ACKNOWLEDGEMENTsThank you JOKES ON YOU, THERES NOTHING HERE

Thank you DevOpsDays Baltimore