yinlin chen, jim tuttle, william a. ingram information ...aws lambda amazon dynamodb amazon...
TRANSCRIPT
![Page 1: Yinlin Chen, Jim Tuttle, William A. Ingram Information ...AWS Lambda Amazon DynamoDB Amazon CloudFront Amazon Route 53 Amazon CloudWatch AWS CloudTrail AWS CloudFormation IAM Amazon](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4d925568593756d475d77a/html5/thumbnails/1.jpg)
A Multi-Tenancy Cloud-Native Digital Library Platform
Yinlin Chen, Jim Tuttle, William A. Ingram
{ylchen, jim.tuttle, waingram}@vt.edu
Information Technologies and ServicesVirginia Tech Libraries
![Page 2: Yinlin Chen, Jim Tuttle, William A. Ingram Information ...AWS Lambda Amazon DynamoDB Amazon CloudFront Amazon Route 53 Amazon CloudWatch AWS CloudTrail AWS CloudFormation IAM Amazon](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4d925568593756d475d77a/html5/thumbnails/2.jpg)
Agenda
• Cloud-native concept• Virginia Tech Digital Library Platform (VTDLP)• Design strategy• Architecture overview• Implementation overview• VTL experiences
![Page 3: Yinlin Chen, Jim Tuttle, William A. Ingram Information ...AWS Lambda Amazon DynamoDB Amazon CloudFront Amazon Route 53 Amazon CloudWatch AWS CloudTrail AWS CloudFormation IAM Amazon](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4d925568593756d475d77a/html5/thumbnails/3.jpg)
Cloud-native Concept
• Entire infrastructure is deployed in the Cloud (AWS)• Platform is composed of a suite of microservices and
managed services• Focus on the business logic and workflow• Utilize the advantages provided by the Cloud
![Page 4: Yinlin Chen, Jim Tuttle, William A. Ingram Information ...AWS Lambda Amazon DynamoDB Amazon CloudFront Amazon Route 53 Amazon CloudWatch AWS CloudTrail AWS CloudFormation IAM Amazon](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4d925568593756d475d77a/html5/thumbnails/4.jpg)
Virginia Tech Digital Library Platform (VTDLP)
• New services to Digital Library Platform– ID Minting service, Access Service, Metadata service, …
• Migrating legacy services to Digital Library Platform– IAWA, VTechWork, …
Preservation Data Modeling Presentation
![Page 5: Yinlin Chen, Jim Tuttle, William A. Ingram Information ...AWS Lambda Amazon DynamoDB Amazon CloudFront Amazon Route 53 Amazon CloudWatch AWS CloudTrail AWS CloudFormation IAM Amazon](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4d925568593756d475d77a/html5/thumbnails/5.jpg)
VTDLP Overview
Preservation staging
SW Virginia
IAWA Images
VtechWork ETDs
StorageAPTrustAmazon S3
Presentation
Others
IAWA
Others
BeyondVT
.
.
.
Batch Metadata Service
Metadata ServiceID Minting Service
Resolution Service
Serialization Service
Other Services
![Page 6: Yinlin Chen, Jim Tuttle, William A. Ingram Information ...AWS Lambda Amazon DynamoDB Amazon CloudFront Amazon Route 53 Amazon CloudWatch AWS CloudTrail AWS CloudFormation IAM Amazon](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4d925568593756d475d77a/html5/thumbnails/6.jpg)
Design Strategy
• Cloud native (AWS ecosystem)• Microservice/SOA (AWS lambda)• Serverless (AWS managed services)• CI/CD Pipeline • Caching as much as possible
– Static files– Lambda functions
• Automation as much as possible– Infrastructure as code– No manual provisioning or managing servers
![Page 7: Yinlin Chen, Jim Tuttle, William A. Ingram Information ...AWS Lambda Amazon DynamoDB Amazon CloudFront Amazon Route 53 Amazon CloudWatch AWS CloudTrail AWS CloudFormation IAM Amazon](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4d925568593756d475d77a/html5/thumbnails/7.jpg)
AWS Ecosystem
AmazonS3
Amazon Glacier
AWSLambda
AmazonDynamoDB
Amazon CloudFront
AmazonRoute 53
Amazon CloudWatch
AWSCloudTrail
AWSCloudFormation
IAM
Amazon API Gateway
AmazonSQS
AmazonSNS AWS CLI
Network & Content Delivery Compute & Database
Management
AWSOrganizations
Messaging Security & Identity Storage
Services
Amazon Pinpoint
AmazonCognito
Amazon EC2
Amazon ES
AWS Certificate Manager
AWSAmplify
![Page 8: Yinlin Chen, Jim Tuttle, William A. Ingram Information ...AWS Lambda Amazon DynamoDB Amazon CloudFront Amazon Route 53 Amazon CloudWatch AWS CloudTrail AWS CloudFormation IAM Amazon](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4d925568593756d475d77a/html5/thumbnails/8.jpg)
Software stacks
React AWS Amplify Node.js Python
Web App Microservice(AWS Lambda)
AWS AppSync
![Page 9: Yinlin Chen, Jim Tuttle, William A. Ingram Information ...AWS Lambda Amazon DynamoDB Amazon CloudFront Amazon Route 53 Amazon CloudWatch AWS CloudTrail AWS CloudFormation IAM Amazon](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4d925568593756d475d77a/html5/thumbnails/9.jpg)
Preservation PipelineChecksums
Fixity
Virus Scan
AWS S3
APTrust
PREMIS
Apache Airflow
![Page 10: Yinlin Chen, Jim Tuttle, William A. Ingram Information ...AWS Lambda Amazon DynamoDB Amazon CloudFront Amazon Route 53 Amazon CloudWatch AWS CloudTrail AWS CloudFormation IAM Amazon](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4d925568593756d475d77a/html5/thumbnails/10.jpg)
Lambda Example – Metadata file
1. File upload to S32. S3 triggers a Lambda function3. Lambda function parses file content and inserts/updates record in
the DynamoDB
![Page 11: Yinlin Chen, Jim Tuttle, William A. Ingram Information ...AWS Lambda Amazon DynamoDB Amazon CloudFront Amazon Route 53 Amazon CloudWatch AWS CloudTrail AWS CloudFormation IAM Amazon](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4d925568593756d475d77a/html5/thumbnails/11.jpg)
Lambda Example – DynamoDB / ES
1. Data modifications in DynamoDB will trigger a Lambda function2. Lambda function captures changes and updates Amazon ES
![Page 12: Yinlin Chen, Jim Tuttle, William A. Ingram Information ...AWS Lambda Amazon DynamoDB Amazon CloudFront Amazon Route 53 Amazon CloudWatch AWS CloudTrail AWS CloudFormation IAM Amazon](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4d925568593756d475d77a/html5/thumbnails/12.jpg)
Presentation - Multi-Tenant Architecture
DB
Application Hub
Search
App1 App2 AppN
![Page 13: Yinlin Chen, Jim Tuttle, William A. Ingram Information ...AWS Lambda Amazon DynamoDB Amazon CloudFront Amazon Route 53 Amazon CloudWatch AWS CloudTrail AWS CloudFormation IAM Amazon](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4d925568593756d475d77a/html5/thumbnails/13.jpg)
Web AppAmazon Route 53
Amazon API Gateway
Amazon CloudFront
AWS Certificate Manager
Amazon Cognito
Amazon S3
AWS Lambda
Amazon DynamoDB
Amazon Elasticsearch
Service
AWS Cloud
![Page 14: Yinlin Chen, Jim Tuttle, William A. Ingram Information ...AWS Lambda Amazon DynamoDB Amazon CloudFront Amazon Route 53 Amazon CloudWatch AWS CloudTrail AWS CloudFormation IAM Amazon](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4d925568593756d475d77a/html5/thumbnails/14.jpg)
The International Archive of Women in Architecture
• A level 0 compliant image server using Amazon S3 and Amazon CloudFront
• Tiles images, manifest JSON files, and etc.• Terabytes of scan images to be processed
![Page 15: Yinlin Chen, Jim Tuttle, William A. Ingram Information ...AWS Lambda Amazon DynamoDB Amazon CloudFront Amazon Route 53 Amazon CloudWatch AWS CloudTrail AWS CloudFormation IAM Amazon](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4d925568593756d475d77a/html5/thumbnails/15.jpg)
Image processing workflow
AWS Lambda
Amazon S3
Amazon S3
AWS Batch
Batch Job – image set 1
Batch Job – image set 2
Batch Job – image set 3
Batch Job – image set N
Tiles &Manifest
Raw images
Amazon CloudWatch
Rule
Amazon Elastic File System
Amazon EC2
![Page 16: Yinlin Chen, Jim Tuttle, William A. Ingram Information ...AWS Lambda Amazon DynamoDB Amazon CloudFront Amazon Route 53 Amazon CloudWatch AWS CloudTrail AWS CloudFormation IAM Amazon](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4d925568593756d475d77a/html5/thumbnails/16.jpg)
Batch job - IIIF_S3 Docker
Amazon S3
• Command• Parameters• Environment
variables• vCPUs• Memory
Tiles &Manifest
IIIF
Amazon Elastic File System
AWS Batch
![Page 17: Yinlin Chen, Jim Tuttle, William A. Ingram Information ...AWS Lambda Amazon DynamoDB Amazon CloudFront Amazon Route 53 Amazon CloudWatch AWS CloudTrail AWS CloudFormation IAM Amazon](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4d925568593756d475d77a/html5/thumbnails/17.jpg)
CI/CD with AWS
Developers AWS CodePipeline
AWS CodeBuild
AWS CloudFormation
Amazon S3
AWS Lambda
Amazon API Gateway
(1) (2)
(3)
(4)
(5)
(6)
(7)
![Page 18: Yinlin Chen, Jim Tuttle, William A. Ingram Information ...AWS Lambda Amazon DynamoDB Amazon CloudFront Amazon Route 53 Amazon CloudWatch AWS CloudTrail AWS CloudFormation IAM Amazon](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4d925568593756d475d77a/html5/thumbnails/18.jpg)
Cloud benefit - Backup examples • S3
– Amazon S3 is 99.999999999% durability and 99.99% availability. – On average, may lose one of 10,000 objects every 10 million years or
so. – Cross-region replication
• DynamoDB– Point-in-time recovery (Last 35 days)– On-Demand Backup (Stored in S3)
• ElasticSearch– Daily snapshots (Last 14 days)– On-Demand Backup (Stored in S3)
![Page 19: Yinlin Chen, Jim Tuttle, William A. Ingram Information ...AWS Lambda Amazon DynamoDB Amazon CloudFront Amazon Route 53 Amazon CloudWatch AWS CloudTrail AWS CloudFormation IAM Amazon](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4d925568593756d475d77a/html5/thumbnails/19.jpg)
VTL Experiences
• Entire development team is AWS certified
• One AWS Certification Subject Matter Expert (SME)• AWS trainings and conferences• Thinking and implementing new ideas the Cloud way
![Page 20: Yinlin Chen, Jim Tuttle, William A. Ingram Information ...AWS Lambda Amazon DynamoDB Amazon CloudFront Amazon Route 53 Amazon CloudWatch AWS CloudTrail AWS CloudFormation IAM Amazon](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4d925568593756d475d77a/html5/thumbnails/20.jpg)
Q & A
Thank You!