top 10 data migration best practices

© 2020, Amazon Web Services, Inc. or its Affiliates.

Chris Rogers

Global Storage Bus. Development

Top 10 Data Migration Best Practices

Jeff Bartley

Storage Specialist SA

© 2020, Amazon Web Services, Inc. or its Affiliates. © 2020, Amazon Web Services, Inc. or its Affiliates.

Agenda

• Migration basics

• Top 10 Best Practices

• The right tool for the job

• Planning your migration

• Transferring data

• Wrap up / Q&A


Stages of Cloud Adoption for Enterprise Transformation

Value

Time

Cloud native

Legacy applications and data

Project Foundation

Migration

Continuous Reinvention

Modernization


Common Migration Drivers

Data Center Consolidation

DigitalTransformation

Cost Reduction

Large-Scale Compute-Intensive Workloads

Facility or Real Estate Decisions

Colocation or Outsourcing Contract Changes

Agility/Dev Productivity

Acquisitions or Divestitures


Migration Business Outcomes

Agility

Build and operate your foundation for innovation

Operational EfficiencyObtain substantial cost savings, freeing up resources to focus on what differentiates your business

Reduced Risk

Migrate through a secure and proven approach that reduces IT risks by moving to a more resilient IT model


Top 10 Data Migration Best Practices


Section 1: The right tool for the job


Best practice #1: Know your data

Choose the right tool for the job

• Virtual machines (VMs) → CloudEndure for AWS

• Databases→ AWS Database Migration Service

• Unstructured data/file data → AWS DataSync, AWS Snow* Family etc.


Best practice #2: Migrate virtual machines with CloudEndure

CloudEndure continuously replicates any application or database from any source into AWS

Business outcome: Allow self-service, rapid, reliable migrations with minimal business disruption

CloudEndure user consoleCloudEndure agent handshake APIs to create staging area and launch target machines

Replication server configurationand ongoing monitoring

Orchestration and system conversion

ready to run workloads in minutes

Continuous, real-time data

replication traffic (compressed and

encrypted)

Corporate data center/any cloud

Disks

Oracle database, CloudEndure agent

SQL Server, CloudEndure agent

Target region

Staging area subnet

Staging area replication servers

(lightweight Linux EC2 instances)

Staging EBS volumes

Target subnet

Launched target EC2 instance

Launched target EC2 instance

Target EBS volumes

Oracle database

SQL Server


Best practice #3: Migrate databases with AWS Database Migration Service (DMS)

M i g r a t i n g d a t a b a s e s t o A W S

Migrate between on-premises and AWS

Migrate between databases

Automated schema conversion

Data replication for migration with zero downtime

100,000+databases migrated


Section 2: Planning your migration


Best practice #4: Understand available bandwidth

Assumes ~25% network overhead

Usable network bandwidth

100 Mbps 1 Gbps 10 Gbps

1 TB 30 hours 3 hours 18 minutes

10 TB 12 days 30 hours 3 hours

100 TB 124 days 12 days 30 hours

1 PB 3 years 124 days 12 days

10 PB 34 years 3 years 124 days


Best practice #5: Assess operational impact of migration

Run a proof of concept (POC)

• Early discovery and remediation of environmental issues

• Sets more realistic migration and edge compute timelines

• Deploy staging workstations

• Ensure low network latencies (<1ms)

• Ensure larger files (>5MB)

• Benchmark & optimize data transfer (target 300-500 MBps)

Plan devices and scheduling with your account team/TAM before ordering jobsResources

• White paper: AWS Snowball Edge data migration guide

• Blog: Data migration best practices with Snowball Edge

AWS Snowball

https://d1.awsstatic.com/whitepapers/snowball-edge-data-migration-guide.pdf?did=wp_card&trk=wp_card

https://aws.amazon.com/blogs/storage/data-migration-best-practices-with-snowball-edge/


• Every part of the network is critical

• Bottlenecks are a moving target

• The WAN might not be the biggest bottleneck

• Source system configuration dictates read performance

Best practice #5: Assess operational impact of migration

AWS DataSync


Best practice #6: Know your data profile

How much data? How many files?

Files

Batch BatchAWS Snowball

EdgeWorkstationData source

AWS Snowball



How much data? How many files?

Files

DataSync Agent

Data source

AWS DataSync

aws



Partitioning large data sources

Snowball Edgedevice

~80 TiBusable capacity

DataSynctask

50 million filesper task

1 mount pointper task


Best practice #7: Scale out as needed

Scale resources to handle large datasets

Snowball Edge devices

Device availability

Operational overhead

Infrastructure requirements


Best practice #7: Scale out as needed

Scale resources to handle large datasets

One task per agent

DataSync agents

Source storage impact

Bandwidth throttling


Best practice #8: Consider the data source

How will a data transfer impact source storage?

• Is the storage system healthy?

• Permissions available to access all data?

• Can the storage support scale-out access?

• What is the rate of change of the data?

• Sufficient source resources to maintain production workloads?


Section 3: Transferring data (finally)


Best practice #9: Preserving metadata

What is metadata?

• File ownership

• Permissions

• Time stamps

• File system attributes

Workloads that need metadata• Data protection

• Migration to cloud file systems

• On-premises access with File Gateway

• Access from FSx for Lustre


Best practice #10: Validate your assumptions

• Verify you can read and batch data (if needed)

• Verify source performance

• Verify your network works as expected

• Verify service configuration and other settings

• Validate timeframe expectations

Test run to validate your plan


Best practice #11: Verify data transfer

• Verification ensures migrated data matches the source

• Critical for medical records, financial transactions, analytics datasets, etc.

• Make sure to plan time for verifying data


Wrap Up


Best Practices for data migration

The right tool for the job

✓ #1: Know your data

✓ #2: Migrate VMs with CloudEndure

✓ #3: Migrate databases with DMS

Planning your migration

✓ #4: Understand available bandwidth

✓ #5: Assess operational impact of migration

✓ #6: Know your data profile

✓ #7: Scale out as needed

✓ #8: Consider the data source

Transferring data

✓ #9: Preserving metadata

✓ #10: Validate your assumptions

✓ Bonus #11: Verify data transfer


Migrate live applications and databases to AWSLearn more: https://aws.amazon.com/cloudendure-migration

CloudEndure

AWS DataSync

Move data over the network between on-premises storage and AWSLearn more: https://aws.amazon.com/datasync

AWS Snow Family

Offline transfer of large amounts of data into and out of AWSLearn more: https://aws.amazon.com/snow

AWS services covered

AWS Database Migration Service

Migrate your databases to AWS with minimal downtimeLearn more: https://aws.amazon.com/dms

https://aws.amazon.com/cloudendure-migration

https://aws.amazon.com/datasync/?whats-new-cards.sort-by=item.additionalFields.postDateTime&whats-new-cards.sort-order=desc

https://aws.amazon.com/snow

https://aws.amazon.com/dms


Q&A

Chris Rogers

Global Storage Bus. Development

Jeff Bartley

Storage Specialist SA


Thank you!

top 10 data migration best practices

Documents