cloudsolutionday 2016: docker & faas at getvero.com
TRANSCRIPT
TU HOANGSoftware Engineer / @rebyn / [email protected]
Docker @ VeroData Migration with Docker @ Vero
CLOUD FUNCTIONS @ VERO
@getveroAutomate customer interactions based on what your customers do, ie: fire welcome coupon campaign when● User signs up; and● Has at least an
outstanding item in cart for a week; and
● Has never received a coupon before.
+250 large consumer businesses use Vero daily:
#CONTAINERS- Opinionated: EC2-Spot,
ECS, ELB, Logentries- 20+ kinds of workers- Memory spec ranging
from 302MB to 2048MB memory
- Anytime: 148 containers (across 80 instances)
- Autoscaled: +350
- Peak: 16 million emails/day
- Weekly volume: 32.2M 75.2M
- Track 20.000 46.296 customer actions/min (+1B +2B/mth)
- Software engineering: 6- DevOps-y engineers: 3
Scale @getvero (with changes since Nov 2015)
Cloud Functions- Stateless compute containers- Ephemeral- Event-triggered- (AND/OR) Significantly depend on third-party
services
Cloud Function 1/7: autoscaling
Cloud Function 2/7: deliverability
Cloud Function 2/7: deliverability
Cloud Function 2/7: deliverability
Cloud Function 2/7: deliverability
Cloud Function 3/7: newsletter diagnosis
Cloud Function 3/7: deployment
@dustay ecs:deploy_by_image mailship:master
1. Check latest SHA’s pending status2. Check ongoing deployments (if any)
2a. YES: cancel | set a reminder2b. NO: DEPLOY AWAY
Cloud Function 3/7: deployment
Cloud Function 3/7: deployment
@dustay stats:low --threshold=50 | stats:refetch $newsletter_id
Triggershttps://bot.getvero.com:1234 /v1/triggers/UUID
HTTP
Chat commands
- Amazon S3
- Amazon DynamoDB
- Amazon Kinesis Streams
- Amazon Simple Notification Service
- Amazon CloudWatch Logs
- Scheduled Events (powered by Amazon CloudWatch Events)
- Amazon API Gateway
- Other Event Sources: Invoking a Lambda Function On Demand
Trigger Options
- Cloud agnostic (vanilla Docker)
- Navigate our setup of security firewalls (though not much desired since most services are public)
- Language agnostic (unified input/output)
Why we rolled out our own
cloud function service
- Of course, it’s always better if someone else manages it
- Startup latency = used to be an issue (then we learnt how to autoscale and/or prewarm infra)
- Horizontal scaling = rough around edges (also applies to our services (micro/nano), not just functions)
- Testing was a pain
Why we ate up along the
way
- Development friction = extremely minimal now
- Services are being created with confidence and scalability in mind despite small team
- 100% automation = within reach now
- Testing = microservice testing. Should bring about no changes.
- Matured autoscaling agent + spot fleet management = #BIGWIN for both functions and services
- See 75% cost reduction upon spot adoption
#WINS
- Stack decomposing = trend forward. Start first with parts of a monolith that interact with external services significantly.
- We control scaling-up. This is equally crucial.
#WINS
- Functions are being gradually treated as first class citizens
- We usually sit down and approach an issue re: can we narrow this business down to nano services / functions
Misc
- Credentials should be loaded upon instance provisioning or container provisioning, not hardcoded into scripts
- Vendor lockin = a concern, though not pressing. IAM- and security group-based services = kept to minimum and has failover.
- Instrumentation = crucial.
- Request/response mapping templates = must. Shared library.
Gotchas
Infrastructure Setup
Infrastructure Setup
Development Productivity
To quote Amazon:Focus on the code that mattersInnovate rapidlyReduce time to market/product
Continuous scaling
DEPLOYMENT(I run out of time)
https://www.getvero.com/careers(Senior Backend Engineers / Site Reliability Engineers
much needed!)
Thanks for tuning in.