how to host an app for $20 in 20min using buildout and hostout
DESCRIPTION
an introduction zc.buildout and how it can help manage complex python installations and reproducible production environments. Dylan will also introduce a new open source project: collective.hostout, a technology for rapidly provisioning and deploying buildout environments on many hosts at once.TRANSCRIPT
[email protected] June 2009 Dylan Jay
Buildout and Hostoutor
“How to host an app for $20 in 20min”
Dylan [email protected]
Technical Solutions Manager
PretaWeb
(Thanks to Darryl Cousins for slides 1-10 from NZPUG presentation)
[email protected] June 2009 Dylan Jay
A python module: hello.py
We can import the method from the module:
def helloworld(): print u"Hello World
from hello import helloworld
[email protected] June 2009 Dylan Jay
A Python Package: hello
A package is a module that contains other modules: hello/
__init__.py hello.py
Now we must import the method from the module within the module.
from hello.hello import helloworld
[email protected] June 2009 Dylan Jay
Disutils
Distutils was written so we have a unified way to install python modules.
python setup.py install
[email protected] June 2009 Dylan Jay
Disutils – creating a distribution
To distribute the hello module we need have it in a directory with a setup.py file. workingdir/
setup.py hello/
__init__.py hello.py
The setup.py file needs at the least the following.
from distutils.core import setup setup(name="hello",
)
[email protected] June 2009 Dylan Jay
Disutils creating a distribution continued
Now we can create a distribution tarball with disutils. python setup.py sdist
Our directory now looks like this workingdir/
setup.py hello/
__init__.py hello.py
dist/ hello-1.0.tar.gz
If we unpack the source distribution it looks like this:
Hello-1.0/ PKG-INFO setup.py hello/
__init__.py hello.py
[email protected] June 2009 Dylan Jay
setuptools
• Setuptools is built on top of distutils • uses the setup.py • uses eggs for distribution • allows us to save our modules as eggs to pypi
Installing setuptools Installing setuptools wget http://peak.telecommunity.com/dist/ez_setup.py wget http://peak.telecommunity.com/dist/ez_setup.py python ez_setup.pypython ez_setup.py
[email protected] June 2009 Dylan Jay
Eggs
To create an egg change the import line in setup.py
from setuptools import setup setup(name="hello",
version="1.0", )
We can call that with: python setup.py bdist_egg
Which creates a binary egg in our dist directory dist/
hello-1.0-py2.4.egg
[email protected] June 2009 Dylan Jay
pypi
If we want that egg available on pypi and we have an account we can do that with a single command. python setup.py sdist upload
Which all the world can use easy_install hello
[email protected] June 2009 Dylan Jay
virtualenv
If we want to install without affecting whole system$ easy_install virtualenv$ virtualenv myenv$ source myenv/bin/activate(myenv)$ easy_install hello. . . (myenv)$ deactivate
[email protected] June 2009 Dylan Jay
zc.buildout – what is it?“Buildout is a system of configuring repeatable steps for
assembling complicated systems (applications) from multiple parts.”
Configure different parts using one config fileGlue the parts togeatherSpecify versionsIsolated in one directoryRepeat that configuration over and over
[email protected] June 2009 Dylan Jay
Installing buildout
$ easy_install zc.buildout$ cd myproj$ buildout init
This creates the following directory structure
myproj/bin/
buildoutparts/develop-eggs/buildout.cfg
[email protected] June 2009 Dylan Jay
buildout.cfg
Buildout does very little by itself[buildout] parts =
A buildout is made of parts. Parts are instances of recipes. Recipes do the work.
[buildout] parts = py
[py] recipe = zc.recipe.egg interpreter = py eggs = hello
[email protected] June 2009 Dylan Jay
Recipes – 185+ on pypi
[email protected] June 2009 Dylan Jay
Running buildout
$ bin/buildoutInstalling py.Generated interpreter 'myproj/bin/py'.
$ bin/py>>> from hello import helloworld
>>> helloworld() Hello World
[email protected] June 2009 Dylan Jay
Buildout - Scripts
You can generate scripts
[buildout]parts = buildout1 buildout2
[buildout1] recipe = zc.recipe.egg eggs = zc.buildout==1.2.1scripts = buildout=buildout1
[buildout2]recipe = zc.recipe.eggeggs = zc.buildout<=1.2.0scripts = buildout=buildout2
[email protected] June 2009 Dylan Jay
Buildout - Versions
You can pin versions
[buildout]parts = buildout1 buildout2versions = versions
[buildout1] recipe = zc.recipe.egg eggs = zc.buildoutscripts = buildout=buildout1
[versions]zc.buildout = 1.2.1
[email protected] June 2009 Dylan Jay
Buildout - Scripts are wrappers$ cat bin/buildout1
#!/bin/python
import syssys.path[0:0] = [ '/download-cache/eggs/zc.buildout-1.2.1-py2.4.egg', '/download-cache/eggs/setuptools-0.6c9-py2.4.egg', ]
import zc.buildout.buildout
if __name__ == '__main__': zc.buildout.buildout.main()
[email protected] June 2009 Dylan Jay
Buildout – manages installation$ bin/buildout -vUninstalling py.Installing buildout1.Installing 'zc.buildout==1.2.1'.We have the distribution that satisfies 'zc.buildout==1.2.1'.Adding required 'setuptools' required by zc.buildout 1.2.1.Picked: setuptools = 0.6c9Generated script 'myproj/bin/buildout1'.Installing buildout2.Installing 'zc.buildout==1.2.0'.We have the distribution that satisfies 'zc.buildout==1.2.0'.Adding required 'setuptools' required by zc.buildout 1.2.0.Picked: setuptools = 0.6c9Generated script 'myproj/bin/buildout2'.
[email protected] June 2009 Dylan Jay
Buildout – variables and dependencies and distutils
[buildout]parts = py
[py] recipe = zc.recipe.egg Interpreter = pyextra-paths=
${reportlab:output}
[reportlab]recipe = collective.recipe.distutilsurl = http://www.reportlab.org/ftp/ReportLab_2_3.tar.gzoutput = ${buildout:directory}/parts/site-packages/
[email protected] June 2009 Dylan Jay
Helloworld in Plonesrc/hello/hello/hello.py
from Products.Five.browser import BrowserView
class HelloWorld(BrowserView):
def __call__(self): return "Hello World"
src/hello/hello/configure.zcml
<configurexmlns:browser="http://namespaces.zope.org/browser"> <browser:page name="hello" class=".hello.HelloWorld" permission="zope2.Public" /></configure>
[email protected] June 2009 Dylan Jay
Helloworld in Plone buildoutbase.cfg
[buildout]parts = instancedevelop = src/hello
[zope2]recipe = plone.recipe.zope2installurl = http://www.zope.org/Products/Zope/2.10.8/Zope-2.10.8-final.tgz
[instance]recipe = plone.recipe.zope2instancezope2-location = ${zope2:location}user = dylan:jayeggs = hello Plonezcml = hello
[email protected] June 2009 Dylan Jay
Run Helloworld
$ bin/buildout -c base.cfgDevelop: 'myproj/src/hello'Updating zope2.Updating fake eggsInstalling instance.
$ bin/instance fg2009-06-08 20:24:54 INFO ZServer HTTP server started at Mon Jun 8 20:24:54 2009
Hostname: localhostPort: 8080
...2009-06-08 20:25:44 INFO Zope Ready to handle requests
[email protected] June 2009 Dylan Jay
Production buildout extends baseprod.cfg
[buildout]extends = base.cfgparts += supervisor
[instance]zeo-client = Truezeo-address = ${zeo:zeo-address}
[zeo]recipe = plone.recipe.zope2zeoserverzope2-location = ${zope2:location}Zeo-address = 127.0.0.1:9000
[varnish]recipe = plone.recipe.varnish:instancebind = 127.0.0.1:80mode = foregroundbackends = ${instance:http-address}daemon = ${varnish-build:location}/sbin/varnishd
[varnish-build]recipe = zc.recipe.cmmiurl = http://waix.dl.sourceforge.net/sourceforge/varnish/varnish-2.0.3.tar.gz
[supervisor]recipe = collective.recipe.supervisorprograms = 10 zeo ${zeo:location}/bin/runzeo 20 instance1 ${instance:location}/bin/runzope 70 cache ${buildout:bin-directory}/varnish ${varnish:location} true
[email protected] June 2009 Dylan Jay
Hosting a buildout
First get a server with root access
~$20USD p/m for 256mb
Slicehost, Amazon ec2, etc
[email protected] June 2009 Dylan Jay
collective.hostout
Aims to to make a hosted python app :- as easy to install as a cPanel app as easy to deploy as google app engine keep control of code
[email protected] June 2009 Dylan Jay
Put collective.hostout in buildout.cfg
[buildout]extends = base.cfgparts += hostout
[hostout]recipe = collective.hostouthost=myproj.slicehost.combuildout = prod.cfgstart_cmd = ${buildout:bin-directory}/supervisordstop_cmd = ${buildout:bin-directory}/supervisorctl shutdown
[email protected] June 2009 Dylan Jay
Run hostout$ bin/buildout Develop: 'myproj/src/hello'Updating zope2.Updating fake eggsUpdating instance.Installing hostout.
$ bin/hostout running setup script 'src/hello/setup.py'.…creating deployment file: versions.cfg prod.cfg base.cfg hello1.0dev_12...6-py2.4.egglogging into the following hosts as root: myproj.slicehost.comPassword:Installing pythonBootstrapping buildout in /var/local/buildoutUnpacking deployment fileRunning buildoutRunning start command
[email protected] June 2009 Dylan Jay
Hostout – with source control
Doesn't need access to svn or git on the server
[email protected] June 2009 Dylan Jay
Hostout TODO
Multi serverTransfer databasesIntegrate with source controlRollback
http://plone.org/products/collective.hostout