tracking huge files with git lfs - linuxcon 2016

@kannonboyPhoto: Le Monde en Vidéo

@kannonboy

@kannonboy@kannonboyPhoto: Le Monde en Vidéo

@kannonboy

Git LFS!

Git LOB!


@kannonboy

ok cool


Git LFS!

TIM PETTERSEN • SENIOR DEVELOPER • ATLASSIAN • @KANNONBOY

Tracking huge files with Git LFS

@kannonboy

G I T L F S

T H E P R O B L E M W I T H B I G F I L E S

Agenda

C O N V E R T I N G Y O U R R E P O

T I P S F O R T E A M S

@kannonboy

data model

@kannonboy

master

feature/JIRA-123

@kannonboy

434bb..tree

bab1e..parent

Tim P <kannonboy@…> 1455209277 -0800committer

Tim P <kannonboy@…> 1455209277 -0800author

My life is my commit message.

98ca9..

bab1e..

fad3d..git cat-file -p 98ca9$

@kannonboy

git cat-file -p 434bb$

434bb..

98ca9..

bab1e..

fad3d..

ace23..100644 blob .gitignoredbdbd..100644 blob README.mda0bc3..040000 tree app33d33..040000 tree configb1de7..100755 blob deploy-prod.sh7011e..100755 blob deploy-staging.sh

typefilemode SHA-1

@kannonboy

434bb..

98ca9..

bab1e..

fad3d..

ace23.. 1010101

dbdbd..

a0bc3..

33d33..

b1de7..

7011e..

1010101

1010101

1010101

master

1010101

1010101

1010101

@kannonboy

98ca9..

bab1e..

fad3d..

@kannonboy

dabad..

98ca9..

bab1e..

fad3d..

86753..

434bb..

50mb100mb150mb

@kannonboy

@kannonboy

(Large File Storage)

Git LFS

@kannonboy

dabad..

98ca9..

bab1e..

fad3d..

86753..

434bb..

$

LFS store

Git host

@kannonboy

☞

☞

☞

dabad..

98ca9..

bab1e..

fad3d..

86753..

434bb..

Git host

LFS store

$

@kannonboy

☞

☞

☞

dabad..

98ca9..

bab1e..

fad3d..

86753..

434bb..

Git host

LFS store

git push$

@kannonboy

☞

☞

☞

dabad..

98ca9..

bab1e..

fad3d..

86753..

434bb..

git pull$

LFS store

Git host

@kannonboy

☞

☞

☞

dabad..

98ca9..

bab1e..

fad3d..

86753..

434bb..☞

☞

☞

dabad..

98ca9..

bab1e..

fad3d..

86753..

434bb..

https://git-lfs.github.com/spec/v1version

sha256:325ddfb…oid

29342295size

git cat-file -p 4749d$

4749d..

bdd12..

778aa..

@kannonboy

$ brew install git-lfs

$ git lfs install

@kannonboy

$ cat ~/.gitconfig

[filter "lfs"] clean = git-lfs clean %f smudge = git-lfs smudge %f required = true

@kannonboy

$ git lfs track “*.mp4”

$ cat .gitattributes

*.mp4 filter=lfs diff=lfs merge=lfs -text

@kannonboy

massive_video.mp4

Work tree

dev

.git/lfs/objects

Clean filter(git-lfs clean)

☞

Index

massive_video.mp4

$

.git/objects

git add

@kannonboy

$

dev

.git/lfs/objects

Smudge filter(git-lfs smudge)

Work tree

massive_video.mp4

☞

Commit tree

massive_video.mp4.git/objects

LFS Store

git checkout

@kannonboy

.git/lfs/objects

.git/objectsHosted repo

LFS store

git push / pull

@kannonboy

$ ls .git/hooks/

prepare-commit-msg.sample post-update.sample pre-commit.sample pre-push ...

@kannonboy

$ git push

Git LFS: (12 of 13 files, 1 skipped) 168.75 MB / 180.87 MB, 12.12 skipped

Counting objects: 22, done. ...

@kannonboy

$ git pull

remote: Counting objects: 3, done. ... Downloading massive_video.mp4 (38.79 MB) ... 1 file changed, 2 insertions(+)

@kannonboy

POST .../repo.git

I need object cafebabe

@kannonboy

/info/lfs/objects/batch{“objects”:[

{“oid”: “cafebabe...”,“size”: 40689401}, ...

],“operation”: “download”}

@kannonboy

200 OK

{“objects”:[ {“oid”: “cafebabe…”, “size”: 40689401, “actions”: {

“download”: { “href”: “https://…/lfs/cafebabe…”,

@kannonboy

cafebabe is over there

} }

...

@kannonboy

200 OK

{“objects”:[ {“oid”: “cafebabe…”, “size”: 40689401, “actions”: {

“download”: { “href”: “https://…/lfs/cafebabe…”,

@kannonboy

cafebabe is over there

“header”: {“Authorization”: “JWT eyJ0eXA…”,

} } }

...

@kannonboy

Converting to Git LFS

☞

@kannonboy

434bb..

fad3d..

98ca9..

41222..

dabad.. 100mb150mb

ace34..☞ 150mb!?!?

@kannonboy

git filter-branch

$ git filter-branch --force --index-filter \ 'git rm --cached --ignore-unmatch big_video.mp4’ \ --prune-empty --tag-name-filter cat -- --all

DON’T DO

THIS!

@kannonboy

$ git filter—branch --prune-empty --tree-filter ' git config -f .gitconfig lfs.url “https://bitbucket.example.com/team/repo.git” git lfs track "*.mp4" git add .gitattributes .gitconfig

for file in $(git ls-files | xargs git check-attr filter | grep "filter: lfs" | sed -r "s/(.*): filter: lfs/\1/"); do git rm -f --cached ${file} git add ${file} done' --tag-name-filter cat -- --all

@kannonboy

DON’T DO

THIS

EITHER!

@kannonboy

BFG Repo-Cleaner

@kannonboy

by @rtyley

@kannonboy

BFG Repo-Cleaner

@kannonboy

10-720x faster than filter-branch

built to kill history

Git LFS support

by @rtyley

@kannonboy

$ git filter—branch --prune-empty --tree-filter ' git config -f .gitconfig lfs.url “https://bitbucket.example.com/team/repo.git” git lfs track "*.mp4" git add .gitattributes .gitconfig

for file in $(git ls-files | xargs git check-attr filter | grep "filter: lfs" | sed -r "s/(.*): filter: lfs/\1/"); do git rm -f --cached ${file} git add ${file} done' --tag-name-filter cat -- --all

@kannonboy

DON’T DO

THIS

EITHER!

@kannonboy

$ brew install bfg

$ bfg —-convert-to-git-lfs ‘*.{zip,mp4}’ --no-blob-protection

@kannonboy

Repofactoring

@kannonboy

@kannonboy

Identifying large objects

github.com/bloomberg/repofactor

by @hashpling

@kannonboy

$

a295ef4… 102437 953722cc7063… 152171 140443

blob SHA size on disk average blob size

generate-larger-than 50000


@kannonboy

a295ef4… 102437 95372 2cc7063… 152171 140443

, PNG, 1148 x 482, PNG, 1101 x 800

$ generate-larger-than 50000 \| add-file-info


@kannonboy

$ generate-larger-than 50000 \ | add-file-info

2cc7063… 152171 140443, PNG, 1101 x 800 a295ef4… 102437 95372, PNG, 1148 x 482

order by average blob size

\| sort -k3nr


@kannonboy

$ generate-larger-than 50000 \ | add-file-info \ | sort -k3nr

$ report-on-large-objects big-stuff.txt

logo-hdpi.png 2cc7063… 152171 140443, PNG…logo-mdpi.png a295ef4… 102437 95372, PNG…

paths

> big-stuff.txt


@kannonboy

$ bfg —-convert-to-git-lfs ‘logo-*.png’ --no-blob-protection


@kannonboy

Enable in Bitbucket

@kannonboy

@kannonboy

Tips for teams

@kannonboyBeware merge conflicts @kannonboy

@kannonboy

…meanwhile in

@kannonboy

Locking

@kannonboy

Source: etonline.com/music

@kannonboy

master

feature0

feature1

File locking (single branch model)

@kannonboy

master

feature0

feature1

File locking (single branch model)

LAME

@kannonboy

master

feature0

feature1

File locking (multi branch model)

@kannonboy

$ git lfs lock <path>

$ git lfs unlock <path>

$ git lfs locks [-i id] [-p path]

@kannonboy

Until then…

@kannonboyTeamwork @kannonboyTeamwork

@kannonboy

$ git lfs fetch

master

feature0

feature1

@kannonboy

$ git lfs fetch

$ git config lfs.fetchrecentalways “true”

master

feature0

feature1

--recent

T-minus 7 days

@kannonboy

$ git config lfs.fetchrecentalways “true”

lfs.fetchrecentrefsdays

lfs.fetchrecentremoterefs

lfs.fetchrecentcommitsdays

(default = 7)

(default = 0)

$ git lfs fetch --recent

@kannonboy

$

master

feature0

feature1

git lfs prune

@kannonboy

$

master

feature0

feature1

--recent(7 days)

prune(10 days)

git lfs prune

@kannonboy

lfs.pruneoffsetdays

lfs.pruneverifyremotealways

(default = 3)

Set this!

$ git lfs prune

@kannonboy

# for a build that just runs the unit tests $ git lfs fetch --exclude Assets/**

# for an audio engineer $ git lfs fetch --include Assets/Audio/**

@kannonboy

# for a build that just runs the unit tests $ git lfs fetch --exclude Assets/**

# for an audio engineer $ git lfs fetch --include Assets/Audio/**

$ git config lfs.fetchexclude Assets/**

$ git config lfs.fetchinclude Assets/Audio/**

@kannonboy

coming very soon!Bitbucket Cloud

git-lfs.github.comdocs

github.com/github/git-lfssource

atlassian.com/bitbucketBitbucket Server

Lookingfor

more?

@kannonboy

coming very soon!Bitbucket Cloud

git-lfs.github.comdocs

github.com/github/git-lfssource

atlassian.com/bitbucketBitbucket Server

Lookingfor

more?

Follow me for occasional Git,Bitbucket & JIRA trivia

tracking huge files with git lfs - linuxcon 2016

Software