Don’t Break the Glass!In case of emergency…

Intro• Apache CloudStack committer, PMC member• Security Guy• Founder of Stratosec• …hacker, reverse-engineer• @johnlkinsella, #CCCNA14

Goal

Provide bread crumbs for you to gain expertise on CloudStack’s data model so you can react quickly in an emergency

situation.

TOC• What Should Production Look Like?• My Nightmare• My response• How to prepare• How we’re trying to make things better• Homework

What Should Production Look Like?• Briefly:

– Redundant management servers– Redundant databases– Redundant storage (primary and secondary)– Redundant infrastructure– Data backups – encrypted

– Pro tip: store backups outside CloudStack.

One Painful February Day…• Made a change in control panel prior to giving customer

access

• Control panel interpreted this as “Delete the Customer.”

• More precisely: “Delete the Customer, VMs and all storage. Now.”

Wait, What??• Took about 15 seconds to realize what happened

• Looking in CloudStack Management UI, saw VM shutting down, then being expunged.

Periodic Table…

Emergency Steps• Shut down ACS management server immediately

• Fire up your favorite SQL tool (I recommend Squirrel SQL)

With your Data Recovery Shrine of Choice

Immediately Establish Contact

Establish Contact With Data Shrine

Know how CloudStack Works• Steps to delete a VM:

– Shut down– Destroy– Expunge volumes– Expunge VM

The “SQL API”• DB Tables:

– vm_instance– volumes– nics– user– account

• Common Fields:• account_id• instance_id• created/updated/removed• state• removed

BACK UP DB BEFORE MAKING CHANGES!

Actual SQL

Make change in SQL.

Bonus: changing VM’s IPUPDATE user_ip_address SET allocated=(SELECT allocated FROM (SELECT allocated, public_ip_address FROM user_ip_address) AS oldip WHERE public_ip_address='184.172.14.194') WHERE public_ip_address='184.172.14.193';UPDATE user_ip_address SET account_id=(SELECT account_id FROM (SELECT account_id, public_ip_address FROM user_ip_address) AS oldip WHERE public_ip_address='184.172.14.194') WHERE public_ip_address='184.172.14.193';UPDATE user_ip_address SET domain_id=(SELECT domain_id FROM (SELECT allocated, public_ip_address FROM user_ip_address) AS oldip WHERE public_ip_address='184.172.14.194') WHERE public_ip_address='184.172.14.193';UPDATE user_ip_address SET state='Allocated' WHERE public_ip_address='184.172.14.193';UPDATE user_ip_address SET network_id=(SELECT network_id FROM (SELECT network_id, public_ip_address FROM user_ip_address) AS oldip WHERE public_ip_address='184.172.14.194') WHERE public_ip_address='184.172.14.193';UPDATE nics SET ip4_address='184.172.14.193' WHERE instance_id=1797;UPDATE vm_instance SET private_ip_address='184.172.14.193' WHERE instance_name='i-2-1797-VM';

Move VM between accountsUPDATE user_ip_address SET account_id=10 WHERE id=29;UPDATE user_ip_address SET domain_id=8 WHERE id=29;UPDATE user_ip_address SET network_id=217 WHERE id=29;UPDATE vm_network_map SET network_id=217 WHERE vm_id=144;UPDATE vm_instance SET domain_id=8 WHERE id=29;UPDATE op_networks SET nics_count=nics_count+1 WHERE id=217;UPDATE op_networks SET mac_address_seq=mac_address_seq+1 WHERE id=217;UPDATE nics SET broadcast_uri='vlan://230' WHERE id=179;UPDATE nics SET network_id=217 WHERE id=179;UPDATE nics SET isolation_uri='vlan://230' WHERE id=179;UPDATE firewall_rules SET account_id=10 WHERE ip_address_id=29;UPDATE firewall_rules SET domain_id=8 WHERE ip_address_id=29;UPDATE firewall_rules SET network_id=217 WHERE ip_address_id=29;

How you can prepare• Create written cloud policy, audit production to match

• Test your backups

• Become familiar with CloudStack internals

• Do you really need frequent data expunge processes?

Next: Production Lock• Feature: Allow our customers to ensure that no

automations will modify or delete their compute or storage instances

– Working for VM– Adding for storage, networks– Need to add to UI

Homework• Pick a user VM in your Cloud. Via the database:

– Find the VM instance record– Find the VM’s nics in the database– Find the VM’s storage in the database– Try deleting a VM via normal means, then before

expunge, restore via SQL

kthxbye!• http://cloudstack.apache.org

jlk@stratosec.co

@johnlkinsella

http://theresnomon.co