automa tion manager backup /restore, ha and dr for ibm cloud · 9/24/2018 backup/restore, ha and dr...

9/24/2018 Backup/Restore, HA and DR for IBM Cloud Automation Manager - Cloud Automation

https://developer.ibm.com/cloudautomation/2018/05/08/backup-ha-dr/ 1/18

Blog

Backup/Restore, HA and DR for IBM CloudAutomation Manager

IBM Cloud Automation Manager (CAM) is designed for production orchestration of cloud workloads and builds off theenterprise capabilities of IBM Cloud Private and the Kubernetes environment. In this article we describe the capabilities ofCAM and strategies around Backup/Restore, High Availability (HA) and Disaster Recovery (DR) for CAM instances that can andshould be taken advantage of when running CAM in a production environment.

Backup and RestoreLets first start with the basic backup and restore of a CAM environment. CAM deploys and manages workloads to and acrossmultiple cloud environments. The data that is used to create the deployments and manage the deployments across theclouds is stored inside CAM’s databases. It is important that a complete and recurring backup strategy be used to backup allCAM data so that in the event of either data loss or data corruption the CAM data can be restored.

Hugh Hockett and Barry Gower Published on May 8, 2018 / Updated on May 23, 2018

0

Cloud Automation Documentation Tutorials Blogs Videos Why IBM?

https://developer.ibm.com/cloudautomation/category/blog/

https://developer.ibm.com/cloudautomation/2018/05/08/backup-ha-dr/

https://developer.ibm.com/cloudautomation/

http://www.ibm.com/support/knowledgecenter/en/SS2L37

http://developer.ibm.com/cloudautomation/tutorials/

https://developer.ibm.com/cloudautomation/blogs/

http://developer.ibm.com/cloudautomation/videos/

https://www.ibm.com/us-en/marketplace/cognitive-automation



Backing up CAM data:CAM data resides in four Persistant Volumes (PVs) that are pre-created prior to the installation of CAM. More on PVs from Kubernetes: https://kubernetes.io/docs/concepts/storage/persistent-volumes/

The default names of the four CAM PVs and their usages are:

Volume Persistant Volume (PV)name

DefaultSize

Usage

Mongo DB cam-mongo-pv 15 GB MongoDB volume for the main CAM database

Logs cam-logs-pv 10 GB Logs volume for CAM pod logs

Terraform Plugins cam-terraform-pv 15 GB Terraform volume for default and user added

plugins

Template Designer

DB

cam-bpd-appdata-pv 15 GB Template Designer MariaDB

For more information on CAM PVs see: https://www.ibm.com/support/knowledgecenter/SS2L37_2.1.0.2/cam_create_pv.html

As a best practice these four CAM volumes should reside in a storage location where consistent snapshots can be taken sothat data is consistently backed up from all four volumes simultaneously. If the underlying storage system does not supportconsistent snapshots, manual backup actions can still be performed using the procedure below.

https://kubernetes.io/docs/concepts/storage/persistent-volumes/

https://www.ibm.com/support/knowledgecenter/SS2L37_2.1.0.2/cam_create_pv.html



Finally, an optional custom encryption password can be configured at CAM chart deploy time to encrypt database contentsand Terraform container contents with a custom password. It is important that this custom encryption password also bebacked up separately (in parallel with the instructions in this document) since it will be needed to decrypt the data during therestore scenario. Without this password a new install of CAM (for restoring from a backup) will not be able to read the existingencrypted CAM data.

Manual backup procedureStop CAM First, stop CAM to ensure there are no modifications during the backup. stopCAM.sh

Logs To backup the CAM logs, simply make a copy of the logs volume mount using a shell command like cp or rsync.

Terraform Plugins To backup the Terraform Plugins, simply make a copy of the Terraform plugins volume mount using a shell command like cpor rsync.

Mongo DB If CAM was deployed with an external MongoDB, best practices for MongoDB backup strategies should be utilized,particularly if the MongoDB instance is replicated or sharded. Find backup and restore best practices in the MongoDBdocumentation: https://docs.mongodb.com/manual/core/backups/

Backup bundled CAM MongoDB database

Start just the cam-mongo database. kubectl scale -n services deploy cam-mongo --replicas=1

1.

https://docs.mongodb.com/manual/core/backups/



It is a good idea to separate the system being backed up from the system capturing the backup. Start a temporarystandalone MongoDB container in the same ICP cluster where CAM is deployed. This container will be used to performthe backup. Note, the port numbers for this command may need to be changed to avoid conflicts. docker run -d -p 27018:27018 --name cam-mongo-backup mongo:3.4

Next, execute the mongodump command, pointing to the cam-mongo service. export MONGO_IP=$(kubectl get -n services svc/cam-mongo --no-headers | awk '{print $3}') docker exec -it cam-mongo-backup mongodump --uri mongodb://$MONGO_IP:27017 --archive=/tmp/mongo_backup.gz --gzip

Once the dump is complete, copy the backup archive off of the standalone MongoDB backup container. Make sure to storethis someplace safe. docker cp cam-mongo-backup:/tmp/mongo_backup.gz .

Finally, cleanup the standalone MongoDB backup container: docker stop cam-mongo-backup docker rm cam-mongo-backup

Template Designer There are two different actions to backup the CAM Template Designer data: the MariaDB and some repositories on the cam-bpd-ui container.

Backup bundled CAM MariaDB database If CAM was deployed with an external MariaDB, best practices for MariaDB backup strategies should be utilized,particularly if the MariaDB instance is replicated or sharded. Find backup and restore best practices in the MariaDBdocumentation: https://mariadb.com/kb/en/library/backup-and-restore-overview/If CAM was deployed with the bundled MariaDB, follow this procedure:

Start just the cam-bpd-mariabd database. kubectl scale -n services deploy cam-bpd-mariadb --replicas=1

2.

3.

4.

5.

•

1.

https://mariadb.com/kb/en/library/backup-and-restore-overview/



It is a good idea to separate the system being backed up from the system capturing the backup. Start a temporarystandalone MariaDB container in the same ICP cluster where CAM is deployed. This container will be used to performthe backup. Note, the port numbers for this command may need to be changed to avoid conflicts. docker run -d -p 3307:3307 --name cam-maria-backup -e MYSQL_RANDOM_ROOT_PASSWORD=yes -d mariadb:10.1.16

Next, execute the mysqldump command, where is the password for the MariaDB instance. To find the password, lookin the CAM install chart for a file called cam-bpd-mariadb-secret.yaml. The password value will be underdata.password. If you are not comfortable specifying password on the command line, you can use an options fileinstead. Learn more about options files in the mysqldump documentation:https://mariadb.com/kb/en/library/mysqldump/#options export MARIA_IP=$(kubectl get -n services svc/cam-bpd-mariadb --no-headers | awk '{print $3}') docker exec -it cam-maria-backup mysqldump --host=$MARIA_IP --port=3306 --all-databases --user=ucdpadmin --password= --result-file=/tmp/mariadb_backup.sql

Copy the backup archive off of the standalone MariaDB backup container. Make sure to store this someplace safe. docker cp cam-maria-backup:/tmp/mariadb_backup.sql .

Finally, cleanup the standalone MariaDB backup container: docker stop cam-maria-backup docker rm cam-maria-backup

Backup CAM BPD UI repositories

Start just the cam-bpd-ui service. kubectl scale -n services deploy cam-bpd-ui --replicas=1

Create an archive of the important files export BPD_UI=$(kubectl get -n services pods | grep cam-bpd-ui | awk '{print $1}') kubectl exec -it -n services $BPD_UI -- tar -cvzf /tmp/cam-bpd-ui-backup.tgz -C /opt/ibm-ucd-patterns/ workspace repositories

2.

3.

4.

5.

•1.

2.

https://mariadb.com/kb/en/library/mysqldump/#options



Then, copy the archive off the container. Make sure to save this someplace safe. kubectl cp -n services $BPD_UI:/tmp/cam-bpd-ui-backup.tgz .

Finally, cleanup the created archive on the container. kubectl exec -it -n services $BPD_UI -- rm -f /tmp/cam-bpd-ui-backup.tar

Content Runtime If you have deployed a Content Runtime, follow the steps here to backup the Chef Server, Software Repository Binaries andPattern Manager:https://www.ibm.com/support/knowledgecenter/SS2L37_2.1.0.2/content/cam_content_runtime_backup.html

Start CAM Once all manual backup steps are complete, start CAM. startCAM.sh

Manual restore procedurePre-requisite: These manual restore steps assume there is a deployed instance of CAM. This can either be the same instancethat the backup was created from which is being repaired, or it can be a new instance of CAM on a separate installation of IBMCloud Private as part of a larger disaster recovery scenario. If a new instance of CAM is being deployed, input the “customencryption password” when deploying the CAM chart if one was specified during the initial deployment.

Stop CAM First, stop CAM to ensure there are no modifications during the restore. stopCAM.sh

Logs To restore the CAM logs, simply make a copy of the backed up log files onto the logs volume mount using a shell commandlike cp or rsync.

3.

4.

https://www.ibm.com/support/knowledgecenter/SS2L37_2.1.0.2/content/cam_content_runtime_backup.html



Terraform Plugins To restore the Terraform Plugins, simply make a copy of the backed up Terraform plugins onto the Terraform plugins volumemount using a shell command like cp or rsync.

Mongo DB Restoring bundled CAM MongoDB database

Start just the cam-mongo database kubectl scale -n services deploy cam-mongo --replicas=1

It is a good idea to separate the system being restored from the system performing the restore. Start a temporarystandalone MongoDB container in the same ICP cluster where CAM is deployed. This container will be used to performthe restore. Note, the port numbers for this command may need to be changed to avoid conflicts. docker run -d -p 27018:27018 --name cam-mongo-backup mongo:3.4

Copy the desired backup archive to be restored into the standalone container docker cp mongo_backup.gz cam-mongo-backup:/tmp/mongo_backup.gz

Next, execute the mongorestore command, pointing to the cam-mongo service. export MONGO_IP=$(kubectl get -n services svc/cam-mongo --no-headers | awk '{print $3}') docker exec -it cam-mongo-backup mongorestore --uri mongodb://$MONGO_IP:27017 --archive=/tmp/mongo_backup.gz --gzip --drop

Finally, cleanup the standalone MongoDB container docker stop cam-mongo-backup docker rm cam-mongo-backup

Template Designer

Restoring bundled CAM MariaDB database

Start just the cam-bpd-mariadb database kubectl scale -n services deploy cam-bpd-mariadb --replicas=1

1.

2.

3.

4.

5.

•1.



It is a good idea to separate the system being restored from the system performing the restore. Start a temporarystandalone MariaDB container in the same ICP cluster where CAM is deployed. This container will be used to performthe restore. Note, the port numbers for this command may need to be changed to avoid conflicts. docker run -d -p 3307:3307 --name cam-maria-backup -e MYSQL_RANDOM_ROOT_PASSWORD=yes -d mariadb:10.1.16

Copy the desired backup archive to be restored into the standalone container docker cp mariadb_backup.sql cam-maria-backup:/tmp/mariadb_backup.sql

Next, execute the restore command, where is the password for the MariaDB instance. To find the password, look inthe CAM install chart for a file called cam-bpd-mariadb-secret.yaml. The password value will be under data.password.If you are not comfortable specifying password on the command line, you can use an options file instead. Learn moreabout options files in the mysqldump documentation: https://mariadb.com/kb/en/library/mysqldump/#options export MARIA_IP=$(kubectl get -n services svc/cam-bpd-mariadb --no-headers | awk '{print $3}') docker exec -it cam-maria-backup sh -c "mysql -v --host=$MARIA_IP --port=3306 --user=ucdpadmin --password= < /tmp/mariadb_backup.sql"

Finally, cleanup the standalone MariaDB container docker stop cam-maria-backup docker rm cam-maria-backup

Restore CAM BPD UI repositories

Start just the cam-bpd-ui service. kubectl scale -n services deploy cam-bpd-ui --replicas=1

Copy the desired backup archive to be restored into the BPD UI container export BPD_UI=$(kubectl get -n services pods | grep cam-bpd-ui | awk '{print $1}') kubectl cp -n services cam-bpd-ui-backup.tgz $BPD_UI:/tmp/

Expand the archive on the container kubectl exec -it -n services $BPD_UI -- tar -xvf /tmp/cam-bpd-ui-backup.tgz -C /opt/ibm-ucd-patterns/ workspace repositories

2.

3.

4.

5.

•1.

2.

3.

https://mariadb.com/kb/en/library/mysqldump/#options



Finally, cleanup the archive on the container. kubectl exec -it -n services $BPD_UI -- rm -f /tmp/cam-bpd-ui-backup.tar

Content Runtime If you have deployed a Content Runtime, follow the steps here to restore the Chef Server, Software Repository Binaries andPattern Manager: https://www.ibm.com/support/knowledgecenter/SS2L37_2.1.0.2/content/cam_content_recovery.html

Start CAM Once all manual restore steps are complete, start CAM. startCAM.sh

High AvailabilityLet's discuss building a CAM environment with high availability to address single points of failure for a basic deployment ofCAM with a single set of pods. First it is important that an ICP be created with multiple worker nodes that can supportmultiple instances of CAM pods with an instances of CAM pods on each node. For more information about deploying an ICPHA environment see the following ICP knowledge center links: https://www.ibm.com/support/knowledgecenter/SSBS6K_2.1.0.2/supported_system_config/hardware_reqs.html https://www.ibm.com/support/knowledgecenter/SSBS6K_2.1.0.2/installing/high_availability.html

CAM is composed of a set of pods or microservices. A CAM pod of each type can be run on each worker node. For exampleone of the CAM pods is called cam-iaas which services the core APIs in CAM. If you deploy an ICP with three worker nodes, acam-iaas pod can run on each of the three worker nodes. Running multiple cam-iaas pods helps increase the capacity forhandling CAM API calls as well as provide high availability of the CAM APIs themselves. If a worker node were to go down, thesurviving two worker nodes and their cam-iaas pods could continue to handle the API requests for CAM.

One note here: technically a single worker node can be used to provide pod HA by deploying more than one instance ofspecific CAM pod on a single worker node. This would allow for surviving the loss of a specific CAM pod, however it would not

4.

https://www.ibm.com/support/knowledgecenter/SS2L37_2.1.0.2/content/cam_content_recovery.html

https://www.ibm.com/support/knowledgecenter/SSBS6K_2.1.0.2/supported_system_config/hardware_reqs.html

https://www.ibm.com/support/knowledgecenter/SSBS6K_2.1.0.2/installing/high_availability.html



provide high availability for losing a worker node with all running pods on it (e.g. the VM or physical hardware failing). It isrecommended that CAM pods run on multiple worker nodes to provide high availability that covers the most failure scenarios.

In CAM there are multiple types pods that handle the front end proxy, API calls, UI calls, service broker, databases and more.Each type of pod can be configured to run on more than one worker node via a "replica count" configuration option at CAMinstall time. This picture shows when deploying the CAM chart configuring a replica count of 3 for the CAM microservices:



Each type of pod translates to multiple underlying microservices. For example there are multiple UI microservices andmultiple API microservices. Configuring a replica count of 3 for APIs sets the number of pods to 3 for all microservices that



are used to handle APIs. The types of pods to configure include:

Broker

Proxy

APIs

UI

Template Designer CDS

Today when a replica count greater than 1 is configured and there are multiple worker nodes, the pods are spread across allavailable worker nodes. For example if a "Replica Count" of 3 is configured and there are 3 worker nodes, a single pod for thattype will be started on each worker node for high availability. For more information on how Kubernetes scheduling works seehttp://blog.kubernetes.io/2017/03/advanced-scheduling-in-kubernetes.html

Pods can also be scaled up or down during runtime to address high availability concerns or address performance/loadcharacteristics of CAM. To scale a specific CAM pod during runtime navigate to the ICP UI under Workloads > Deployments.Find the specific pod (for example cam-iaas in the screenshot below) and under the action menu select "Scale"

•••••

http://blog.kubernetes.io/2017/03/advanced-scheduling-in-kubernetes.html



Then enter the number of instances to scale to (3 in the example below) and click "Scale Deployment"



By default CAM ships with a single MongoDB database for storing CAM data, a single Redis for session management and asingle MariaDB for storing template designer data. To support a highly available CAM environment external clusters of



MongoDB, Redis and MariaDB can be setup outside of CAM and then be configured for use by the CAM deployment at installtime.

Cloud Automation Manager can be configured to use an external MongoDB and an external database for Cloud AutomationManager Template Designer by following the instructions here:https://www.ibm.com/support/knowledgecenter/SS2L37_2.1.0.2/cam_using_external_mongodb.html

Similarly an external Redis cluster that can provide high availability can also be configured for use with CAM by following theinstructions here: https://www.ibm.com/support/knowledgecenter/SS2L37_2.1.0.2/cam_using_external_redis.html

Note: Since these external databases and Redis clusters are created and managed outside of CAM ensure you have anappropriate backup and disaster recovery strategy for these external components in addition to the practices outlined here.

Disaster RecoveryDisaster recovery (DR) typically covers scenarios where a datacenter is no longer available in either a planned or unplannedevent, and services must be restored at another site that is geographically separated. Implementations of disaster recoverytypically vary based on corporate requirements, locations of data centers, available technology and more.

Common technologies used involve site to site continuous replication of databases or underlying data volumes at either asoftware or hardware level, and either synchronously (typically shorter distances, no data loss) or asynchronously (typicallylonger distances, but with some data loss of transactions in flight that were not fully replicated). A more basic approach mayinvolve taking regular backups and transferring copies of those backups to a remote site for recovery.

More information on disaster recovery and general practices can be found here:https://en.wikipedia.org/wiki/Disaster_recovery

Replicating CAM data to a backup site is one part of the DR story, but in the event of a disaster (or testing a disaster) a newCAM environment must also be installed that uses the replicated data. There are two key considerations with regards to abackup CAM environment:

https://www.ibm.com/support/knowledgecenter/SS2L37_2.1.0.2/cam_using_external_mongodb.html

https://www.ibm.com/support/knowledgecenter/SS2L37_2.1.0.2/cam_using_external_redis.html

https://en.wikipedia.org/wiki/Disaster_recovery



The workloads deployed and managed by CAM in target cloud environments need to be replicated and/or backed upalong side CAM to the backup site. For example, if you have a target VMware environment, that environment and itsworkloads must be replicated or backed up to the backup site outside of CAM. CAM itself does not contain the deployedworkloads or data inside deployed workloads, only the data about how to manage the deployed workloads resides withinCAM.

The backup CAM instance must have equivalent network access to target cloud environments. For example if you have aVMware vSphere target at the primary datacenter that has deployments, the VMware vSphere environment anddeployments must be network accessible in the backup datacenter as they were in the primary datacenter. In this wayexisting cloud connections in CAM will still be valid and deployments from CAM can be managed by the backup site'sCAM. As another example if a public cloud environment like IBM Cloud is used, equivalent network access to IBM Cloudmust also be in place for the backup CAM instance to continue to deploy and manage existing IBM Cloud deployments.

As was discussed in the earlier backup/restore section, CAM stores its data in four persistent volumes. These volumes holdthe CAM data and logs that need to be replicated to a backup site. These four volumes should also be placed in a consistencygroup so that data is transferred across all four volumes in the order it is being modified.

Replicated data can then be used with a new install of CAM at the backup site during an unplanned even (e.g. naturaldisaster), planned event (e.g. planned shutdown of a datacenter) or for a test of disaster recovery practices.

In any of these DR scenarios, the restore procedure documented above can be used to restore the replicated data into anewly deployed instance of CAM.

For an unplanned event (or test of an unplanned event):

Stop replication and take a consistent snapshot of all four replicated CAM volumes at the backup site

Create four new volumes and populate them with the data from the consistent snapshot

Create four PVs that reference the volumes with restored CAM data

Install CAM and reference these four PVs during the CAM chart install

1.

2.

1.

2.

3.

4.



For a planned event (or test of a planned event)

Stop CAM first to prevent any transactions in flight from being lost.

Take a consistent snapshot of all four replcating CAM volumes at the backup site

Create four new volumes and populate them with the data from the consistent snapshot

Create four PVs that reference the volumes with restored CAM data

Install CAM and reference these four PVs during the CAM chart install

Final ThoughtsThis article provides you with strategies and procedures for implementing Backup/Restore, High Availability and DisasterRecovery solutions of CAM instances. Implementations may vary depending on your specific requirements and availabletechnologies but many of the underlying principles around CAM availability and data protection remain the same.Understanding these concepts with CAM will allow you to build a robust instance of CAM for deploying and managingproduction cloud applications.

Special thanks to Jeffrey Luo and Partha Kaushik for their assistance verifying the content of this blog

TAGS BACKUP, CAM, CLOUD AUTOMATION MANAGER, DISASTER RECOVERY, DR, FEATURED, HA, HIGH AVAILABILITY, RESTORE

1.

2.

3.

4.

5.

https://developer.ibm.com/cloudautomation/tag/backup/

https://developer.ibm.com/cloudautomation/tag/cam/

https://developer.ibm.com/cloudautomation/tag/cloud-automation-manager/

https://developer.ibm.com/cloudautomation/tag/disaster-recovery/

https://developer.ibm.com/cloudautomation/tag/dr/

https://developer.ibm.com/cloudautomation/tag/featured/

https://developer.ibm.com/cloudautomation/tag/ha/

https://developer.ibm.com/cloudautomation/tag/high-availability/

https://developer.ibm.com/cloudautomation/tag/restore/



Contact Privacy Terms of use Accessibility Report Abuse Feedback Cookie preferences

Join The Discussion

Enter your comments...

Post Comment

United States - English

https://ibm.com/community/feedback/

https://www.ibm.com/privacy/us/en/

https://developer.ibm.com/terms/ibm-developer-terms-of-use/

https://www.ibm.com/accessibility/us/en/

https://www.ibm.com/developerworks/secure/report/

automa tion manager backup /restore, ha and dr for ibm cloud · 9/24/2018 backup/restore, ha and dr...

Documents