aws re:invent 2016: amazon ecr deep dive on image optimization (con401)
TRANSCRIPT
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Scott Windsor, Sr. Software Development Engineer, AWS
Mao Geng, Site Reliability Engineer, Pinterest
December 1, 2016
CON401
Amazon ECR Deep Dive
on Image Optimization
What to Expect from the Session
• Anatomy of Docker images
• Optimizing image builds
• Overview of Docker Registry V2 API
• CI/CD best practices
• ECR learnings from Pinterest
Dockerfiles: Hello, world
$ docker build -t hello-reinvent -f Dockerfile.hello .
Sending build context to Docker daemon 699.1 MB
Step 1 : FROM alpine
---> baa5d63471ea
Step 2 : MAINTAINER "Scott Windsor"
---> Running in 06bdf87b81ea
---> 4f73f4a232f9
Removing intermediate container 06bdf87b81ea
Step 3 : CMD echo Hello, re:Invent!
---> Running in cf349feca920
---> fa069f014b6a
Removing intermediate container cf349feca920
Successfully built fa069f014b6a
Dockerfiles: Hello, world
$ docker history hello-reinvent
IMAGE CREATED CREATED BY SIZE
fa069f014b6a 3 minutes ago /bin/sh -c #(nop) CMD ["echo" "Hello, re:Inv 0 B
4f73f4a232f9 3 minutes ago /bin/sh -c #(nop) MAINTAINER "Scott Windsor" 0 B
baa5d63471ea 3 weeks ago /bin/sh -c #(nop) ADD file:7afbc23fda8b0b3872 4.803 MB
Dockerfiles: Hello, world
$ docker history alpine
IMAGE CREATED CREATED BY SIZE
baa5d63471ea 3 weeks ago /bin/sh -c #(nop) ADD file:7afbc23fda8b0b3872 4.803 MB
Dockerfiles: Hello, world
$ docker build -t hello-reinvent -f Dockerfile.hello .
Sending build context to Docker daemon 699.1 MB
Step 1 : FROM alpine
---> baa5d63471ea
Step 2 : MAINTAINER "Scott Windsor"
---> Using cache
---> 4f73f4a232f9
Step 3 : CMD echo Hello, re:Invent <3 <3 <3!
---> Running in 51080848d506
---> 19c7fb45a397
Removing intermediate container 51080848d506
Successfully built 19c7fb45a397
Optimizing Image Builds: Hello, ruby
FROM ruby:2.3
ADD Gemfile Gemfile.lock /
RUN bundle install
ADD hello.rb .
CMD ["ruby", "hello.rb"]
Optimizing Image Builds: Hello, ruby
source 'https://rubygems.org'
gem "colorize"
Optimizing Image Builds: Hello, ruby
#!/bin/env ruby
require "colorize"
color_wheel = [:light_magenta, :light_blue,
:light_green, :light_red, :light_yellow].shuffle.cycle
greeting = "Hello, re:Invent!"
s = greeting.each_char.map do |c|
c.colorize color_wheel.next.to_sym
end.join ''
puts s
Optimizing Image Builds: Hello, ruby
Sending build context to Docker daemon 5.111 MB
Step 1 : FROM ruby:2.3
---> 45766fabe805
Step 2 : ADD Gemfile Gemfile.lock /
---> f30a7987497f
Removing intermediate container d9828a354cab
Step 3 : RUN bundle install
---> Running in 3e501ce02c9c
Fetching gem metadata from https://rubygems.org/.............
Fetching version metadata from https://rubygems.org/.
Installing colorize 0.8.1
Using bundler 1.13.5
Bundle complete! 1 Gemfile dependency, 2 gems now installed.
Bundled gems are installed into /usr/local/bundle.
---> 0e217ba0ced7
Removing intermediate container 3e501ce02c9c
Optimizing Image Builds: Hello, ruby
Step 4 : ADD hello.rb .
---> b41a47dceb98
Removing intermediate container 486b3b403224
Step 5 : CMD ruby hello.rb
---> Running in 8e1e1428d56c
---> 2cb069515f81
Removing intermediate container 8e1e1428d56c
Successfully built 2cb069515f81
Optimizing Image Builds: Hello, ruby
#!/bin/env ruby
require "colorize"
color_wheel = [:light_magenta, :light_blue, :light_green,
:light_red, :light_yellow].shuffle.cycle
greeting = "Hello, re:Invent "
s = greeting.each_char.map do |c|
c.colorize color_wheel.next.to_sym
end.join ''
puts s + "<3 <3 <3".colorize(:light_magenta) + "!"
Optimizing Image Builds: Hello, ruby
Sending build context to Docker daemon 5.124 MB
Step 1 : FROM ruby:2.2
---> 8fecf438974e
Step 2 : ADD Gemfile Gemfile.lock /
---> Using cache
---> 3b8bc7055d6f
Step 3 : RUN bundle install
---> Using cache
---> 9a8fe024a6c1
Step 4 : ADD hello.rb .
---> d1bd2fd4e2d2
Removing intermediate container b323bf7041a4
Step 5 : CMD ruby hello.rb
---> Running in 55b4ec70135f
---> 706d48ba1af3
Removing intermediate container 55b4ec70135f
Successfully built 706d48ba1af3
Optimizing Image Builds: Hello, golang
FROM golang:1.7
ADD main.go .
RUN go build -o hello
CMD ["./hello"]
Optimizing Image Builds: Hello, golang
package main
import "fmt"
func main() {
fmt.Println("你好, re:Invent!")
}
Optimizing Image Builds: Hello, golang
$ docker build -t hello-golang -f Dockerfile.golang-combined .
Sending build context to Docker daemon 5.226 MB
Step 1 : FROM golang:1.7
---> 47734a1408b7
Step 2 : ADD main.go .
---> 85d0f410b40d
Removing intermediate container ae79fe0c7846
Step 3 : RUN go build -o hello
---> Running in bc6e71b350ae
---> 5b59c94af88f
Removing intermediate container bc6e71b350ae
Step 4 : CMD ./hello
---> Running in 252c9b2ef359
---> 3f982ea19213
Removing intermediate container 252c9b2ef359
Successfully built 3f982ea19213
Optimizing Image Builds: Hello, golang
$ docker images hello-golang
REPOSITORY TAG IMAGE ID CREATED SIZE
hello-golang latest 3f982ea19213 4 days ago 674 MB
Optimizing Image Builds: Hello, golang
$ docker history hello-golang
IMAGE CREATED CREATED BY SIZE
COMMENT
3f982ea19213 4 days ago /bin/sh -c #(nop) CMD ["./hello"] 0 B
5b59c94af88f 4 days ago /bin/sh -c go build -o hello 1.634 MB
85d0f410b40d 4 days ago /bin/sh -c #(nop) ADD file:22b3017a338f88f902 78 B
47734a1408b7 2 weeks ago /bin/sh -c #(nop) COPY file:f6191f2c86edc9343 2.478 kB
<missing> 2 weeks ago /bin/sh -c #(nop) WORKDIR /go 0 B
<missing> 2 weeks ago /bin/sh -c mkdir -p "$GOPATH/src" "$GOPATH/bi 0 B
<missing> 2 weeks ago /bin/sh -c #(nop) ENV PATH=/go/bin:/usr/loca 0 B
<missing> 2 weeks ago /bin/sh -c #(nop) ENV GOPATH=/go 0 B
<missing> 2 weeks ago /bin/sh -c curl -fsSL "$GOLANG_DOWNLOAD_URL" 243.6 MB
<missing> 2 weeks ago /bin/sh -c #(nop) ENV GOLANG_DOWNLOAD_SHA256 0 B
<missing> 2 weeks ago /bin/sh -c #(nop) ENV GOLANG_DOWNLOAD_URL=ht 0 B
<missing> 2 weeks ago /bin/sh -c #(nop) ENV GOLANG_VERSION=1.7.1 0 B
<missing> 2 weeks ago /bin/sh -c apt-get update && apt-get install 138.8 MB
<missing> 3 weeks ago /bin/sh -c apt-get update && apt-get install 122.6 MB
<missing> 3 weeks ago /bin/sh -c apt-get update && apt-get install 44.3 MB
<missing> 3 weeks ago /bin/sh -c #(nop) CMD ["/bin/bash"] 0 B
<missing> 3 weeks ago /bin/sh -c #(nop) ADD file:c6c23585ab140b0b32 123 MB
Optimizing Image Builds: Hello, golang
FROM golang:1.7
VOLUME /build # Add volume for copying out
ADD main.go .
RUN go build -o hello
CMD ["cp", "hello", "/build/hello"]
# Copy binary out of container into volume
Optimizing Image Builds: Hello, golang
FROM scratch # Use scratch (empty)
ADD hello . # Add binary
CMD ["hello"] # Run!
Optimizing Image Builds: Hello, golang$ docker build -t hello-golang-build -f Dockerfile.build .
Sending build context to Docker daemon 1.639 MB
Step 1 : FROM golang:1.7
---> 47734a1408b7
Step 2 : VOLUME /build
---> Running in 53ba8193299a
---> b86a8ebd4f66
Removing intermediate container 53ba8193299a
Step 3 : ADD main.go .
---> b197080c46cd
Removing intermediate container 8df45d03b69d
Step 4 : RUN go build -o hello
---> Running in af46ee1da8bb
---> 219aacd4c756
Removing intermediate container af46ee1da8bb
Step 5 : CMD cp hello /build/hello
---> Running in f43137a7aafe
---> c27e7cb97e2d
Removing intermediate container f43137a7aafe
Successfully built c27e7cb97e2d
Optimizing Image Builds: Hello, golang$ mkdir build
$ docker run -v ${PWD}/build:/build hello-golang-build
$ docker build -t hello-golang -f Dockerfile.run .
Sending build context to Docker daemon 1.644 MB
Step 1 : FROM scratch
--->
Step 2 : ADD ./build/hello .
---> 0b84a2ca1810
Removing intermediate container 21467dda26ab
Step 3 : CMD ./hello
---> Running in 6baa329f27b4
---> 316b3116d7ef
Removing intermediate container 6baa329f27b4
Successfully built 316b3116d7ef
Optimizing Image Builds: Hello, golang$ docker images hello-golang
REPOSITORY TAG IMAGE ID CREATED SIZE
hello-golang latest 316b3116d7ef 50 seconds ago 1.634 MB
$ docker history hello-golang
IMAGE CREATED CREATED BY SIZE
316b3116d7ef About a minute ago /bin/sh -c #(nop) CMD ["./hello"] 0 B
0b84a2ca1810 About a minute ago /bin/sh -c #(nop) ADD file:82ff6b7578af24a8f0 1.634 MB
Tips
• Bring only what you need at runtime
• Separate build vs. runtime containers
• Reducing layers and sizes will improve pull times
Advanced Tips
• Building your own base images:
https://docs.docker.com/engine/userguide/eng-
image/baseimages/
• Squashing layers (in Docker1.13.0-rc2)
https://github.com/docker/docker/commit/362369b4bbea
38881402d281ee2015d16e8b10ce
Pulling an Image
$ docker pull <registry-uri>/<image-name>:<tag>
Docker daemon:1. Fetches image manifest at tag
2. For each layer that it doesn’t have:
1. Fetch layer
Pulling an Image
$ docker pull <registry-uri>/<image-name>:<tag>
Docker daemon:1. GET /v2/<image-name>/manifests/<tag>
2. For each layer it doesn’t have:
1. GET /v2/<image-name>/blobs/<digest>
Pushing an Image
$ docker push <registry-uri>/<image-name>:<tag>
Docker daemon:1. For each layer:
1. Check if layer exists
2. Initiate layer upload
3. Send layer data
4. Completes the upload
2. Pushes manifest when all layers complete
Pushing an Image
$ docker push <registry-uri>/<image-name>:<tag>
Docker daemon:1. For each layer:
1. HEAD /v2/<image-name>/blobs/<digest>
2. POST /v2/<image-name>/blobs/uploads/
3. PATCH /v2/<image-name>/uploads/<uuid>
4. PUT /v2/<image-name>/blobs/<uuid>?digest=<digest>
2. PUT /v2/<image-name>/manifests/<tag>
Docker V2 Registry
Fetching auth token and url:
$ TOKEN_RESULT=`aws ecr get-authorization-token`
$ ECR_TOKEN=`echo $TOKEN_RESULT \
| jq -r ".authorizationData[].authorizationToken"`
$ ECR_URL=`echo $TOKEN_RESULT \
| jq -r ".authorizationData[].proxyEndpoint”`
Docker V2 Registry
Getting username / password:
$ ECR_USERNAME=`echo $ECR_TOKEN | base64 -D | cut -d: -f1`
$ ECR_PASSWORD=`echo $ECR_TOKEN | base64 -D | cut -d: -f2`
Logging in:
$ docker login –u $ECR_USERNAME –p $ECR_PASSWORD $ECR_URL
Docker V2 Registry
Create a repository:$ IMAGE_URI=`aws ecr create-repository --repository-name hello \
| jq -r ".repository.repositoryUri"`
Deleting (when we’re done):$ aws ecr delete-repository --repository-name hello --force
Docker V2 Registry
Fetching an image manifest:$ curl –u ${ECR_USERNAME}:${ECR_PASSWORD} \
${ECR_URL}/v2/hello/manifests/latest -o hello-manifest.json
$ cat hello-manifest.json
{
"schemaVersion": 1,
"name": "hello",
"tag": "latest",
"architecture": "amd64",
...
Docker V2 Registry
...
"fsLayers": [
{
"blobSum":
"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
},
{
"blobSum":
"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
},
{
"blobSum":
"sha256:3690ec4760f95690944da86dc4496148a63d85c9e3100669a318110092f6862f"
}
],
...
Docker V2 Registry...
"history": [
{
"v1Compatibility": "{\"architecture\":\"amd64\",\"author\":\"\\\"Scott
Windsor\\\"\",\"config\":{\"Hostname\":\"1d811a9194c4\",\"Domainname\":\"\",\"User\":\"\",\"Att
achStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"Tty\":false,\"OpenStdin\":fals
e,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/
bin\"],\"Cmd\":[\"echo\",\"Hello, re:Invent \\u003c3 \\u003c3
\\u003c3!\"],\"Image\":\"sha256:4f73f4a232f9809fe21ef1a8fe0e2ef3140c38341d6590ad4e97a4f973b6360
a\",\"Volumes\":null,\"WorkingDir\":\"\",\"Entrypoint\":null,\"OnBuild\":[],\"Labels\":{}},\"co
ntainer\":\"51080848d50609793dd2c9821873254c5f96d0b7f5b9642749d80a7029f6c90c\",\"container_conf
ig\":{\"Hostname\":\"1d811a9194c4\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"A
ttachStdout\":false,\"AttachStderr\":false,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":fals
e,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[\"/b
in/sh\",\"-c\",\"#(nop) \",\"CMD [\\\"echo\\\" \\\"Hello, re:Invent \\u003c3 \\u003c3
\\u003c3!\\\"]\"],\"Image\":\"sha256:4f73f4a232f9809fe21ef1a8fe0e2ef3140c38341d6590ad4e97a4f973
b6360a\",\"Volumes\":null,\"WorkingDir\":\"\",\"Entrypoint\":null,\"OnBuild\":[],\"Labels\":{}}
,\"created\":\"2016-11-14T01:21:09.976223694Z\",\"docker_version\":\"1.12.0-
rc3\",\"id\":\"07c891b928a4818a46a4e9a6c5bbbdec4be0d201290056ae593ada8a2c8dbf50\",\"os\":\"linu
x\",\"parent\":\"db1d48cc60e1d888c9c57d75e0bc64f6a7f70d6e3bd8af91f106d60ac65cb37e\",\"throwaway
\":true}"
},
...
Docker V2 Registry
...
{
"v1Compatibility":
"{\"id\":\"db1d48cc60e1d888c9c57d75e0bc64f6a7f70d6e3bd8af91f106d60ac65cb37e\",\"parent\
":\"4b59778f82f9d17a484a278bd23d2d0b3c7ddcf022ab5250cf4f59308b3bc3f5\",\"created\":\"20
16-11-14T01:08:23.946015875Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c
#(nop) MAINTAINER \\\"Scott Windsor\\\"\"]},\"author\":\"\\\"Scott
Windsor\\\"\",\"throwaway\":true}"
},
{
"v1Compatibility":
"{\"id\":\"4b59778f82f9d17a484a278bd23d2d0b3c7ddcf022ab5250cf4f59308b3bc3f5\",\"created
\":\"2016-10-18T20:31:22.321427771Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c
#(nop) ADD file:7afbc23fda8b0b3872623c16af8e3490b2cee951aed14b3794389c2f946cc8c7 in /
\"]}}"
}
],
...
Docker V2 Registry
...
"signatures": [
{
"header": {
"jwk": {
"crv": "P-256",
"kid": "R3RU:PVFU:SPAV:GX42:JCTL:NTFT:W2BC:6S6N:IAU2:YGVJ:URDN:MS5G",
"kty": "EC",
"x": "HowD731zWxKgRhk5r2mywedUJGV6thJru8YqMSVOb9M",
"y": "G8Zi951QEjSLHcQeX1H9Bv9hAYFsvVG-mOK8rbRnrKg"
},
"alg": "ES256"
},
"signature":
"j0GQtDPSPMCrNUPBnZllOhltEaCkAfrHdpdLZGyuFLOE7qQvgf1wqeeHRZSY1mIoqSx3jiDPlEWPVlnrA2lACQ",
"protected":
"eyJmb3JtYXRMZW5ndGgiOjI3MzcsImZvcm1hdFRhaWwiOiJDbjAiLCJ0aW1lIjoiMjAxNi0xMS0yMlQxODoxNzowNVoifQ
"
}
]
}
Docker V2 Registry
Fetching layers
$ cat hello.manifest.json | \
jq ".fsLayers[].blobSum" -r \
> hello.layers.txt
$ <hello.layers.txt xargs -I {} \
curl -L –u ${ECR_USERNAME}:${ECR_PASSWORD} \
${ECR_URL}/v2/hello/blobs/{} \
-o {}.tar.gz
Docker V2 Registry
Inspecting layers
$ du -hs sha256:*.tar.gz
du -hs sha*
2.2M sha256:3690ec4760f95690944da86dc4496148a63d85c9e31
00669a318110092f6862f.tar.gz
4.0K sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16
422d00e8a7c22955b46d4.tar.gz
Docker V2 Registry
Inspecting layers
$tar tfvz
sha256:3690ec4760f95690944da86dc4496148a63d85c9e3100669a318110092
f6862f.tar.gz
drwxr-xr-x 0 0 0 0 Oct 18 11:58 bin/
lrwxrwxrwx 0 0 0 0 Oct 18 11:58 bin/ash ->
/bin/busybox
lrwxrwxrwx 0 0 0 0 Oct 18 11:58 bin/base64 ->
/bin/busybox
lrwxrwxrwx 0 0 0 0 Oct 18 11:58 bin/bbconfig ->
/bin/busybox
-rwxr-xr-x 0 0 0 805032 Aug 12 07:38 bin/busybox
...
Docker V2 Registry
Inspecting layers
$ tar tfvz
sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8
a7c22955b46d4.tar.gz
$
Empty layer!
Tips
• Reducing number of layers reduces upload/downloads
needed
• Reducing layer contents reduces uploads/download time
• Minimizing layer changes results in faster deployments
CI/CD Best Practices
• Always tag each build with any of these schemes:
• Semantic version (i.e. “1.3.2-9”)
• Git SHA (i.e. “aebcbca”)
• Build Number (i.e., “127”)
• Build Id (i.e. “511d5e51-b415-4cb2-b229-b3c8a46b7a2f”)
• Avoid ”latest” or ”stable”
• This allows for rolling deployments (and rollback)
CI/CD Tips
• Build caches can help improve build times
• Reduce build image sizes & number of layers
• Build and push in same AWS region if possible
About Me
● Enterprise Software Developer
● Performance Engineer
● Site Reliability Engineer
gengmao
Mao Geng
64
Catalog of Ideas
75 billion pins
1.5 billion boards
150 million monthly active users
100 million weekly active users
80% of users are via mobile app
> 50% of users are from outside the U.S.
Issues
● #22648 Unstoppable Zombie
Processes - AUFS
● #12327 pip install fails -
overlay
● #12080 Overlayfs
does not work with
unix domain
sockets
Tips
● Use the storage driver works for you
● Use newer kernel• “WE DO NOT BREAK USERSPACE!” - Linus Torvalds
● Bake Docker Engine in AMI
● Less is more• --net=host
• --volume
● Use new version Docker Engine
Build Images of Mono Repos
● Python, Java, C++, Golang
● Build vs Runtime images
● One build image per mono repo could be very large and
time-consuming to build
● Separate dependencies into multiple build images• Organize a hierarchy of build images and use Makefile to
manage their build order
• Use shell script to calculate checksum of a build image’s
dependencies, and use the checksum to tag build image
● --cache-from probably is better solution
ECR login token
● Deal with TOOMANYREQUESTS: Rate exceeded
● Share docker config cross users
• $(aws ecr get-login) in cron
• DOCKER_CONFIG=/etc/.docker/config.json
docker-credential-ecr-login
● https://github.com/awslabs/amazon-
ecr-credential-helper
•/usr/bin/docker-credential-ecr-
login
•cache token for ½ refresh
window•Authenticating Amazon ECR
Repositories for Docker CLI with
Credential Helper
ECR with Mesos
● URIS: [ “file:///etc/docker.tar.gz” ]
• Marathon: Using a Private Docker
Registry
● --docker-
config=/etc/.docker/config.json
• Mesos >= 1.0
• Mesos Configuration
● #4278 Marathon Web UI not
support mesos containerizer