Using a Private Git Server for Packaging Software

Git for fun and profit at iThemes

Collaboration Problems

● Everyone had to coordinate in order to make code changes

● Only one of us could modify a specific file at any time

● If we ever made a mistake, we could lose code and possibly lose hours or days of effort

Collaboration Solution: Git

● Developers can work on the same projects without having to coordinate each change

● Work cannot be accidentally lost or overwritten

● A full history of all the changes, including who made the last change to individual file lines in case there are any questions or concerns, is built into every repository

Git Workflow Example

$ git initInitialized empty Git repository in git-test/.git/$ echo "Testing" > new-file$ git add new-file$ git commit -m 'Added new-file'[master (root-commit) c6ecaa7] Added new-file 1 file changed, 1 insertion(+) create mode 100644 new-file

Hosting Problems

Git may be decentralized, but we still needed a central authority to send code to. This meant that we needed a server solution.● Needed to scale to an unknown number of

collaborators● Needed to scale to an unknown number of

repositories● Needed control in order to add advanced

“hooks” (discussed later)● Needed to do so in a cost-effective manner

Hosting Solution: Gitosis

● Self-managed centralized server that all code changes are sent to

● Unlimited repositories and unlimited users● Large number of hooks that can be used to

run code when changes are made to repositories

● Per-user permissions for repository access

Since Gitosis is no longer actively maintained, I would use Gitolite if I were starting over today.

Remote and Local Repositories

When using a Git server such as Gitosis, a repository lives on the server. This is the remote repository.

Our developers clone (make a copy) of the remote repository. This is the local repository. A developer makes modifications to their local repository and sends the changes back to the remote repository.

Remote and Local Repositories Workflow

$ git clone git@example.com:example-repo...$ cd example-repo/$ echo "Testing" > new-file$ git add new-file$ git commit -m 'Added new-file'...$ git pushCounting objects: 4, done.Delta compression using up to 8 threads....

As developers of WordPress plugins and themes, our products are given to customers as zip files.

We used to create these manually.

There had to be a better way.

Packaging Problems

Packaging Solution: Git Hooks

Git provides a large number of hooks that can execute code when a repository changes.

The post-receive hook runs after changes have been pushed to a remote repository (the repository hosted by the Git server, Gitosis). By modifying this hook, we can perform actions after changes are received by the server.

post-receive Hook Example

#!/bin/sh

# Run packager/usr/local/bin/git-post-receive-hook "" "$1" "$2" "$3"

git-post-receive-hook Script Packaging Example

#!/bin/bashrepo=$(basename `pwd`|sed 's/\.git$//')temp_dir=`mktemp -d`cd $temp_dirgit clone git@git.example.com:$repocd $repogit submodule update --init --recursivefind . -name ".git*" -exec rm -rf {} \;cd ..zip -r $repo.zip $repomv $repo.zip /home/site/html/releases

Deployment Problems

Just as we used to generate zip files manually, we used to upload files for our customers manually as well.

Since we now had an automated system to generate zip files, it only made sense to also automate the distribution of these files.

Deployment Solution: Git Hooks

With some modifications to our git-post-receive-hook script, we could easily send the files anywhere we needed them.

git-post-receive-hook Script Deployment Example

# Code from packaging example

zip=/home/site/html/releases/$repo.zip

# Send to membership serverscp $zip user@server:html/member/downloads

# Send to S3 storages3cmd -c /root/.s3cfg --no-progress put $zip s3://downloads.example.com/products

Other Features: Easily Share Code Across Projects

Git supports nesting a repository inside a directory of another repository. These nested repositories are called submodules.

We broke out repeated code libraries into their own submodules. This prevents fixing a bug or adding a feature in only some of the projects that use the library. We’ve even automated updates to all projects that use the library each time the library is updated.

Other Features: Simplified WordPress Localization Domains

WordPress has a series of localization functions, for example:

__( ‘Translate string’, ‘domain’ );

The domain keeps the translations for each plugin and theme separate from all the other plugins and themes. This means that it needs to be unique for each.

Other Features: Simplified WordPress Localization Domains

We’ve simplified this with our packager. All of our projects use the domain of ‘LION’:

__( ‘Translate string’, ‘LION’ );

The packager then takes care of changing each instance of this domain to a generated, unique domain that is specific to the project.

Future Feature: Unit Testing

The next feature to be integrated into our packing system is an automated unit testing platform. This will feature global unit tests that apply to all projects and project-specific unit tests.

In order for a commit to be accepted by the remote repository, it must pass all the unit tests.

Hosted Git Server Solutions

If you prefer to have someone else host the Git server, you have options. Both GitHub and Bitbucket offer great hosted solutions and provide access to hooks for adding automated features to your remote repositories.

Thanks

Find this presentation at chrisjean.com/200ok

Chris Jeanchrisjean.com@chrisjean