arecibo documentation - read the docs · the variables arecibo_public_account_number and...

Arecibo DocumentationRelease 2

Andy McKay and Contributors

April 16, 2012

CONTENTS

1 Introduction to Arecibo 3

2 Installation (Linux) 52.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.2 Installation steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

3 Installation (App Engine) 93.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93.2 Installation steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

4 User Access (Linux) 134.1 Anonymous access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134.2 Full access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

5 User Access (App Engine) 155.1 The first time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155.2 Every other time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155.3 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

6 Remote access (App Engine) 17

7 Customising the server 197.1 Signals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197.2 Default methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197.3 Customising templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

8 Server side models 218.1 Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218.2 Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218.3 Project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

9 Arecibo clients 23

10 Posting errors 2510.1 HTTP Post . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

11 Variable documentation 2711.1 Required (sort of) variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2711.2 Optional variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2711.3 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

i

12 Sample Javascript (Browser) Client 3112.1 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3112.2 Advantages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3112.3 Disadvantages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

13 Sample Python Client 3313.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3313.2 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

14 Sample Ruby Client 3514.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3514.2 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

15 Sample PHP Client 3715.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3715.2 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

16 Sample Plone client 3916.1 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3916.2 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3916.3 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

17 Sample Django Client 4117.1 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4117.2 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4117.3 Adding to your middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4117.4 Adding to your custom views . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4217.5 Using celery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4217.6 Further configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4317.7 Other times . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4317.8 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

18 Sample Rails client 4518.1 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4518.2 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4518.3 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

19 Release Notes 47

20 Indices and tables 49

ii

Arecibo Documentation, Release 2

Contents:

CONTENTS 1


2 CONTENTS

CHAPTER

ONE

INTRODUCTION TO ARECIBO

Arecibo is an error tracker. It allows you to log errors from web sites and collect them all in one location. It thenallows you to prioritise and group them. Notifications can then be sent to developers of the errors.

There are two main components to Arecibo:

• client, this is normally a web application on residing on a server that sends an error into Arecibo. If there isn’tan existing API for your application, then there’s an API you can use to build your own.

• server, this is your instance of Arecibo that you can use to receive errors.

The server was written to run on App Engine in 2010. In 2011 a none App Engine port was made that should run in a“standard” environment. The App Engine port is essentially frozen now with no new updates. It works, but isn’t realydeveloped. This situation may change, but if we can recommend a version to use, don’t use the App Engine one.

There is also a bit of feature fork between the two. Primarily the App Engine one has an Issue tracker. The none AppEngine one does not. Because of it’s use at Mozilla the none App Engine one has no need of an Issue tracker.

The best way to get a feel for what Arecibo does, is to give the demo server a try. The details of the demo server areat http://www.areciboapp.com/demo

3

http://www.areciboapp.com/demo


4 Chapter 1. Introduction to Arecibo

CHAPTER

TWO

INSTALLATION (LINUX)

2.1 Requirements

The installation has been tested on Ubuntu Linux. It should work on any Linux installation, it might even work onWindows.

You will require:

• Connection to a database either locally or remotely. We recommend Postgresql 1.

• Access to a Celery queue and a corresponding backend 2.

• A working Django 1.3 will be installed, or use an existing one 3. Packages are installed via pip 4.

• A working Python 2.4 or greater installation 5.

• You will need git to check out Arecibo from github 6.

• You will need a web server, this documentation covers Apache 7 and mod_wsgi 8, but others will work.

2.2 Installation steps

The following steps are done on Ubuntu Linux. Other operating systems may vary.

2.2.1 1. Download Arecibo

Create a directory to install Arecibo too. Run the following command:

~ $ git clone git://github.com/andymckay/arecibo.gitCloning into arecibo...... done.

1 http://www.postgresql.org/2 http://ask.github.com/celery/3 http://www.djangoproject.com/4 http://pypi.python.org/pypi/pip5 http://www.python.org/6 http://git-scm.com/7 http://httpd.apache.org/8 http://code.google.com/p/modwsgi/

5

http://www.postgresql.org/

http://ask.github.com/celery/

http://www.djangoproject.com/

http://pypi.python.org/pypi/pip

http://www.python.org/

http://git-scm.com/

http://httpd.apache.org/

http://code.google.com/p/modwsgi/


2.2.2 2. Install dependencies

You must have at least Python 2.4 as noted above and pip so that you can install the Python dependencies. Virtualenvironment setup is recommended, but not required. Run the following:

~ $ pip install -r arecibo/listener/normal/requirements.txtDownloading/unpacking Django==1.3...

The other dependencies: database, web server, celery backend (eg Rabbit MQ) should be installed now.

2.2.3 3. Configure Arecibo

Run:

~ $ cd arecibo/listener/normal~/arecibo/listener/normal $ cp local_settings.py.dist local_settings.py

Then configure local_settings.py with at least:

• the Django database configuration

• the Django secret key

• any other Django configuration that needs overriding, such as celery.

Run:

~/arecibo/listener/normal $ python manage.py syncdb

Create your super user. You should now be able to run:

~/arecibo/listener/normal $ python manage.py runserver

And see it all working.

2.2.4 4. Configure Apache

Run:

~/arecibo/listener/normal $ cd config~/arecibo/listener/normal/config $ cp arecibo.wsgi.sample arecibo.wsgi

Now link that up in Apache:

WSGIScriptAlias / /path/to/arecibo/listener/normal/config/arecibo.wsgi

Further configuration options would include mapping media to the static media files inside Arecibo listener/media.

Restart Apache and you should be able to see Arecibo.

2.2.5 5. Final test

Change local_settings.py:

ANONYMOUS_POSTING = True

Then hit:

6 Chapter 2. Installation (Linux)


http://path.to.arecibo/v/1/

And you should get a response: Error recorded.

Anything else means you need to check your database connection and that Django can speak to the celery queue.

2.2. Installation steps 7


8 Chapter 2. Installation (Linux)

CHAPTER

THREE

INSTALLATION (APP ENGINE)

The App Engine installation of Arecibo is no longer recommended and will not be updated. It still works just fine, butI’m not maintaining it. Hopefully someone else will.

3.1 Requirements

You will require a App Engine application. These are free to create, or you can run one locally to test. To deploy aninstance of Arecibo to App Engine you will need a copy of the App Engine SDK. You will need the Python version ofthe App Engine SDK 1.

You will need git to check out Arecibo from github 2.

You will need the current version of Django. At the time of writing, we are supporting Django 1.2.3 3.

3.2 Installation steps

The following steps are done on Mac OS X. Other operating systems may vary.

3.2.1 1 a. Create an App Engine instance

At http://appengine.google.com/ create a new App Engine application. Give your new App Engine installation aunique name. This is the app name that you will be using in later steps.

3.2.2 1 b. Install the Google App Engine SDK

Install the Google App Engine SDK. If you already have it installed, you can skip this step.

3.2.3 2. Download Arecibo

Run the following command:

1 http://code.google.com/appengine/downloads.html#Google_App_Engine_SDK_for_Python2 http://git-scm.com/3 http://www.djangoproject.com/download/1.2.1/tarball/

9

http://appengine.google.com/

http://code.google.com/appengine/downloads.html#Google_App_Engine_SDK_for_Python

http://git-scm.com/

http://www.djangoproject.com/download/1.2.1/tarball/


~ $ git clone git://github.com/andymckay/arecibo.gitInitialized empty Git repository in /Users/andy/arecibo/.git/remote: Counting objects: 601, done.remote: Compressing objects: 100% (511/511), done.remote: Total 601 (delta 233), reused 209 (delta 40)Receiving objects: 100% (601/601), 260.74 KiB | 275 KiB/s, done.Resolving deltas: 100% (233/233), done.

3.2.4 3 a. Configure Arecibo (easy way)

Run create.py and follow the prompts:

~ $ cd arecibo~/arecibo $ python configure.py

3.2.5 3 b. Configure Arecibo (manual way)

There are at least two things you will need to change: the app.yaml file and local_settings.py file. There are examplesof each of these files in the app_engine, directory. The backup version of these files need to be copied and changed.

First app.yaml:

~ $ cd arecibo/listener/app_engine~/arecibo/listener/app_engine $ cp app.yaml.example app.yaml

Alter the first line of app.yaml:

application: your_application_error

Replacing your_application_error with the app name of your application from step 1.

Second local_settings.py:

~/arecibo/listener/app_engine $ cp local_settings.py.example local_settings.py

Then alter the file as detailed:

ARECIBO_PUBLIC_ACCOUNT_NUMBER = "your_public_account_number_here"ARECIBO_PRIVATE_ACCOUNT_NUMBER = "your_private_account_number_here"

DEFAULT_FROM_EMAIL = "[email protected]_engine"SITE_URL = "http://theurl.to.your.arecibo.instance.com"

The variables ARECIBO_PUBLIC_ACCOUNT_NUMBER and ARECIBO_PRIVATE_ACCOUNT_NUMBERshould be unique id that you’ll be using to post to your site. This is used in URLs, so try to avoid / and unicode,any combination of 32 letters and numbers normally works.

DEFAULT_FROM_EMAIL is the Google email address you used to setup your App Engine site. This has to be anemail that is authorized by App Engine, the simplest is to use the one you created your App Engine site with. Fordocumentation on how to send email from App Engine, see here: http://code.google.com/appengine/docs/python/mail/

SITE_URL the full URL (including protocol) that your site is at.

An example file might be:

ARECIBO_PUBLIC_ACCOUNT_NUMBER = "wetlwk36524352345y.rutywr5hs.tgywq43w5jy2w35"ARECIBO_PRIVATE_ACCOUNT_NUMBER = "swtqak365qkt6qo45tyh45tq3k5w345qhtr2q75y2"

10 Chapter 3. Installation (App Engine)

http://code.google.com/appengine/docs/python/mail/


DEFAULT_FROM_EMAIL = "[email protected]"SITE_URL = "http://my-arecibo-site.appspot.com"

Download and copy over Django (this is not included for licensing reasons). Then we can be sure we use the versionof Django that is compatible, not the the one that comes by default with App Engine:

~/arecibo/listener/app_engine $ wget http://media.djangoproject.com/releases/1.2/Django-1.2.1.tar.gz..~/arecibo/listener/app_engine $ tar zxf Django-1.2.1.tar.gz~/arecibo/listener/app_engine $ mv Django-1.2.1/django .~/arecibo/listener/app_engine $ rm -rf Django-1.2.1*

3.2.6 4. Upload to App Engine

If you don’t have the Python App Engine SDK you will need to install it at this point.

Uploading to App Engine is a simple as:

~/arecibo/listener/app_engine $ appcfg.py update .

Follow the prompts for your email and password. You should see quite a few messages scroll past. If you get thesemessages at the end, then it’s worked and you should be able to visit the site in your browser:

Checking if new version is ready to serve.Closing update: new version is ready to start serving.Uploading index definitions.Uploading cron entries

3.2. Installation steps 11


12 Chapter 3. Installation (App Engine)

CHAPTER

FOUR

USER ACCESS (LINUX)

User access is managed inside the database and can be managed using the Django admin interface. You will need tomake sure you have a super user defined 1.

4.1 Anonymous access

Users aren’t able to access errors by default. If you set:

ANONYMOUS_ACCESS = True

Users will then be able to access, but not modify anything.

4.2 Full access

To create a new account go to: http://path.to.your.arecibo/users/create/ and create a new account. The user won’t beable to do anything until the admin goes to http://path.to.your.arecibo/users/ and allows the user.

Or you can go to the good old Django admin at admin/auth/user. Users will need the staff flag set.

1 https://docs.djangoproject.com/en/dev/ref/django-admin/?from=olddocs#createsuperuser

13

http://path.to.your.arecibo/users/create/

http://path.to.your.arecibo/users/

https://docs.djangoproject.com/en/dev/ref/django-admin/?from=olddocs#createsuperuser


14 Chapter 4. User Access (Linux)

CHAPTER

FIVE

USER ACCESS (APP ENGINE)

Any user with a Google account can login to your Arecibo account, however they cannot view anything at all until yougive them access.

5.1 The first time

For the first time you log in to your site, you won’t be able to access because you have not been given access.Administering users to give them access requires admin rights, something you don’t have yet. To gain admin rights,you log in through the Google App Engine console and provide admin rights to the user you are going to be using.

To do this:

• Go to your App Engine admin console http://code.google.com/appengine/docs/theadminconsole.html

• Login, if needed as the Google user who has access to administer the site.

• Go to your app > datastore viewer > user

• Find the user you’d like to give access to, click on them

• Set “is_staff” to True

5.2 Every other time

Once you’ve used the admin console, you can give any other user rights to administer the site using the admin console.

• In Arecibo go to Users, this will give you a list of users.

• Click on a user to toggle their access.

If by doing this you accidentally lock yourself out, or want to delete users from showing up there at all, you can usethe App Engine admin console as noted above to approve yourself or delete users.

5.3 Notes

If you got Google Apps already for your domain, then you can restrict the login just to your domain in the App Engineconsole.

15

http://code.google.com/appengine/docs/theadminconsole.html


16 Chapter 5. User Access (App Engine)

CHAPTER

SIX

REMOTE ACCESS (APP ENGINE)

The instance on Google App Engine can be accessed by using the *remote_api*[1]_. A management command hasbeen added to allow you to do this:

python manage.py remoteApp Engine interactive console for test-areciboapp>>>

You can then execute Python commands. To view the timestamp of the first error:

>>> Error.all()[0].timestampUsername:***********@googlemail.comPassword:datetime.datetime(2010, 6, 14, 16, 25, 49, 835365)

Note that the console is not appropriate for tasks that do a large number of datastore requests, since each is a HTTPrequest. For example: batch altering of records - for that you’d be better off uploading a specific script.

17


18 Chapter 6. Remote access (App Engine)

CHAPTER

SEVEN

CUSTOMISING THE SERVER

The server is not designed to be completely customisable, but rather to have some key customisation points you canalter. The main target for customisation is in the workflow - that is when and how errors are processed. Rather thangive a lot of large forms full of options, we’d rather focus on some scripts to allow this to happen.

The default area for customisation is within the custom folder. Being open source, you can of course change as muchas you wish, however you then have more patches to maintain as Arecibo progresses.

7.1 Signals

Arecibo uses Django’s signal handling mechanism to deal with customizing. Whenever an item is created, signals aresent and those can be caught, default behaviour changed and so on as required. The signals are:

• user_created: sent when a user is created

• error_created: sent when an error is created

• group_created: sent when an error grouping is created

• group_assigned: sent when a group is created

• error_assigned: sent when an error is assigned to group

• notification_created: sent when a notification is created

You can connect your own methods to these signals, or disconnect the default methods, as you would like. By defaultthe Python script at custom/listeners.py is imported, so this a good place to do this customisation.

7.2 Default methods

The following are the default methods that are run on the signals. Feel free to turn these off or replace with whatevermethod you’d like.

7.2.1 default_grouping

Connected to: error_created Location: from error.listeners import default_grouping

This is how errors are grouped together. By default it does it by adding together the fields: type, server, msg and status.If you’d like to remove this, then add the following to custom/listeners.py:

19


from error.signals import error_createdfrom error.listeners import default_grouping

error_created.disconnect(default_grouping, dispatch_uid="default_grouping")

If you’d like to create your own group system you can do this by attaching a new signal:

from error.signals import error_created

def my_grouping(instance, **kw):pass # your code here

error_created.connect(my_grouping, dispatch_uid="my_grouping")

7.2.2 default_browser_parsing

Connected to: error_created Location: from error.listeners import default_browser_parsing

This parses the user agent string sent by the error into something more recognizable, producing the browser andoperating system. However if you’d like something a little more custom you can do that. If you’d like to remove this,then add the following to custom/listeners.py:

7.2.3 default_notification

Connected to: error_created Location: from notifications.listeners import default_notification

This parses the error and figures out if a notification needs to be sent. This is a good opportunity to customize whatnotification is sent to whom. The actual email (or whatever) will be sent by a cron job, this process is just to figure outthe list of notifications that are to be sent.

7.2.4 default_project

Connected to: group_assigned Location: from projects.listeners import default_project

This figures out what project the group gets attached to.

7.3 Customising templates

If you would like to customise a template, then place it in custom templates. The easiest way to do this is to find thetemplate you’d like to customise, copy it into that folder and then make your changes.

20 Chapter 7. Customising the server

CHAPTER

EIGHT

SERVER SIDE MODELS

The following describes the models that exist on the server side and their relationships.

8.1 Error

The error that has been recorded by the incoming receiver.

8.2 Group

A grouping of errors. Rather than dealing with individual errors, you might like to deal with groups. This is probablymore advisable for tickets or tracking. When an error is written a signal is triggered. A default group method listensto this signal to place the error in a group. An error can only be in one group.

8.3 Project

A website or group of websites collected into a project. A project will have one or more domains. When an error iswritten a signal is triggered. A default project method listens to this signal to place the error within the appropriateproject. An error can only be in one project

21


22 Chapter 8. Server side models

CHAPTER

NINE

ARECIBO CLIENTS

Clients are the parts that send information to Arecibo using one of its API’s.

All clients can be found in the clients directory of the Arecibo source.

23


24 Chapter 9. Arecibo clients

CHAPTER

TEN

POSTING ERRORS

This covers the process of posting an error from your website into Arecibo. There main method of sending an error toArecibo is by a HTTP post. It’s easy to add in your own and hopefully more will be made open source.

10.1 HTTP Post

The URL for version 1 of the API is:

http://yoursite.com/v/1/

When the next major release of Arecibo API happens, we will increment the URL to /v/2/, leaving the /v/1/ active.This will allow clients to upgrade as needed.

The method for sending an error is simple, a HTTP POST to the above URL containing the variables set out in thedocumentation.

10.1.1 Notes

• HTTP GET requests will be ignored, only POST

• The only required variable is the Arecibo API key

• If possible set a timeout in your posting client (say 10 seconds)

10.1.2 Example in Python

In this example we’ll use the public test site:

• URL: http://test-areciboapp.appspot.com/

• Account number: “w3;5qwy45qshtqu46tdtgheq47s.ert6ew45e4i2w65”

Sending a HTTP POST in Python is easy using the urllib module. The most minimal request that can be sent justcontains the API key, so let’s send that as a first example:

>>> import urllib>>> data = {"account":"w3;5qwy45qshtqu46tdtgheq47s.ert6ew45e4i2w65"}>>> encoded = urllib.urlencode(data)>>> urllib.urlopen("http://test-areciboapp.appspot.com/v/1/", encoded).read()’Error recorded’

25

http://yoursite.com/v/1/

http://test-areciboapp.appspot.com/


If the request is successful, a HTTP status of 200 will be returned.

The only thing you can really do wrong with this is get the API key wrong, Arecibo will do its best to record the errorno matter what the error. If you’ve got some things in the POST that are wrong besides the API key, then Arecibo willsimply log them on error. Sending a false API key will give you:

>>> data = {"account":"wrongaccountnumber"}>>> encoded = urllib.urlencode(data)>>> urllib.urlopen("http://test-areciboapp.appspot.com/v/1/", encoded).read()’Problem recording the error: Account wrongaccountnumber does not exist.’

The request will also return a HTTP status of 500.

Once you’ve established how to make your error post, you can proceed to add in more variables. Here we are sendingan error including the priority, status and some extra information in the msg:

>>> data = {"account":"w3;5qwy45qshtqu46tdtgheq47s.ert6ew45e4i2w65", "status": 403, "priority": 1,... "msg": "Someone tried to access /secure without the appropriate authorization"}>>> encoded = urllib.urlencode(data)>>> urllib.urlopen("http://test-areciboapp.appspot.com/v/1/", encoded).read()’Error recorded’

For more detailed and full examples on how to do a such a POST, please see the existing sample clients Python, Rubyor JavaScript libraries. For example the Python library does: UTF-8 encoding, checks for required variables and setstime outs to prevent leaving sockets open.

10.1.3 Notes for Authors

• Where possible, set a timeout for your application posting to Arecibo, should Arecibo not be available, you donot want this to affect your site.

• Make sure your client library is robust, so that an error posting to Arecibo - either on your end formatting thedata or on Arecibo’s end - does not cause more errors. In the example Plone integrations when it cannot writeto Arecibo because of some error, it logs to the local log file as a place of last resort.

• The more information you can pass the better.

• Sending every may not be useful we’ve found some errors that get continually reported may not be useful. Sofor example we ignore a 404 that might occur on: favicon.ico or robots.txt in the example integrations. If thereare errors that just going to occur regularly, either don’t send them or set them as very low priority.

26 Chapter 10. Posting errors

CHAPTER

ELEVEN

VARIABLE DOCUMENTATION

This covers all the variables that can be sent in a post to Arecibo. There is only one required variable: your public APIkey. All others are optional, leaving it to you to figure out exactly how much you data you want to send to the server.

11.1 Required (sort of) variables

11.1.1 account

The public API key for your account on Arecibo. Without it we don’t know where to assign the request. This is thepublic API key as defined in the API key page. If you have ANONYMOUS_POSTING set to True in your settings, thisvariable will not be required, it just gets ignored.

account = "123123124124..."

11.2 Optional variables

All the following examples use the JavaScript API as examples. All values are taken to be strings.

11.2.1 status

Limit: length of 3 and a valid status code

The HTTP status code that is returned. This must be a valid status code 1; any other values are ignored.

status = "404"

11.2.2 priority

Limit: empty or between 1 and 10

A number between 1 and 10. The values 1 to 3 are coloured in the web interface, with number 1 being the mostimportant. A user can configure notifications to be sent to them when errors are above a certain priority.

priority = "3"

1 Valid HTTP statuses are: 100, 101, 102, 200, 201, 202, 203, 204, 205, 206, 207, 226, 300, 301, 302, 303, 304, 305, 307, 400, 401, 402, 403,404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 414, 415, 416, 417, 422, 423, 424, 426, 500, 501, 502, 503, 504, 505, 507, 510.

27


A standard a 404 “Page not found” could be priority 5. They could occur regularly and there may not be much youcan do about them. However, if a “Page not found” has a referer from your site, then you could assign priority of 3,since this is something you may be able to fix. A 500 “Server Error” could be priority 1.

11.2.3 ip

Limit: 15 chars

The IP address of the client triggering the error. If you are setting this on your server, be sure to take into account anyproxies that might be in the way.

ip = "192.168.1.53"

11.2.4 user_agent

Limit: 255 chars

The user agent string that the client sends will provide a great deal of information about the browser, operating systemand so on. Please note, however, that this can be easily faked and cannot be relied on 100%.

user_agent = "Mozilla/5.0 (Macintosh; U; Intel Mac OS X..."

11.2.5 url

Limit: 10,000 chars

The full URL making the request at that time. Include any GET parameters in your URL so that these are properlyprocessed.

url = "http://clearwind.ca/this/url/does/not/exist?test=yes"

11.2.6 uid

Limit: 255 chars

A unique id generated for each error. This uid can be included in an error message to the user. The user can theninclude this uid when inquiring about an error. A unique id for a simple site could be the output of a timestamp.However, on a large site this can cause confusion when there are concurrent errors; therefore a more complex uid maybe necessary. Arecibo does not require this value to be unique, but if there are multiple errors with the same id, theadministrator would have to sort out which was which.

uid = "1256235123.1020"

Providing a uid to the user is helpful because you can simply tell the user to contact you with the uid and you can lookit up.

11.2.7 type

Limit: 255 chars

The type of error that has occurred, for example a DatabaseError. This string is completely up to you.

28 Chapter 11. Variable documentation


type = "ZeroDivisionError"

11.2.8 server

Limit: 1,024 chars, UTF-8 encoding

The name of the server the error is occurring on. This allows you to identify the actual server, useful for when youhave a web site balanced across several servers.

server = "serverA"

11.2.9 msg


A message that goes along with this error. This could be a more detailed error returned by your application. It couldalso be your chance to include any other notes you feel relevant to this issue. All HTML is ignored on the server.

msg = "Lorem ipsum dolor sit amet..."

11.2.10 traceback


If your application provides a useful stack trace, then here is the opportunity to include it, this is arguably one of themost important elements, so include it if you can. All HTML is ignored and there is a limit to the amount of text sent.

traceback = "[COMException (0x80040154): Retrieving the COM class factoryfor component with CLSID {4D880EAB-BF35-423A-A859-B1D9F2AC4CC1} faileddue to the following error: 80040154.]"

11.2.11 timestamp

Limit: valid string

The time that the error occurs. The date and time that Arecibo needs is the current time for the GMT time zone. Theformat is as specified by RFC 2822, for example: Fri, 02 Jan 2009 19:19:51 -0000. As convenience, we also accept aprefix of GMT which is interpreted as -0000.

var now = new Date;timestamp = now.toUTCString();

11.2.12 request


Text of all the request variables sent with the request. This is a text area where you can capture any other particularvariables you thing might be relevant.

request = "..."

11.2. Optional variables 29


11.2.13 username

Limit: 255 chars

If your application has a username, this is the user that is currently using the application. If you know specifically thatit’s an Anonymous user, setting this to “Anonymous” will it make clear that you know there was no user logged in.

username = "Bob the Builder"

11.3 Notes

• Text in the following fields: traceback, msg, type and server are assumed to be UTF-8 encoding. We plan onsupporting other encoding later, but at the moment everything is tested with UTF-8 data. All other fields areASCII strings.

• Any text over the limit for that field will be truncated. An error will be written into the error field (visible on aview) so you can spot this and correct.

• We won’t reject any error, unless it has an invalid private key. The error will still be written so one mistake inthe posting of data does not invalidate the whole report.

• All HTML is going to be quoted for display, so feel free to send any HTML without worrying about security.

30 Chapter 11. Variable documentation

CHAPTER

TWELVE

SAMPLE JAVASCRIPT (BROWSER)CLIENT

The JavaScript (Browser) client is probably the easiest of all the possible clients to set up and we recommend trying itfirst. It also provides the greatest flexibility in terms of being platform agnostic.

To install simply add the following to your error page.

<script type="text/javascript" src="http://your-site-name.com/lib/error.js"></script><script type="text/javascript">

arecibo.account = ’yourpublicaccountnumber’;arecibo.run();

</script>

12.1 Notes

The following variables are automatically assigned: ip, user_agent, url. However any of them may be overridden, withyour own values should you want to.

An object is created called arecibo. Variables would be assigned in the following manner:

arecibo.variable = name

For example to set a status and priority, you would add in more values:

<script type="text/javascript" src="http://your-site-name.com/lib/error.js"></script><script type="text/javascript">

arecibo.account = ’yourpublicaccountnumber’;arecibo.status = 500;arecibo.priority = 1;arecibo.run();

</script>

12.2 Advantages

• Can be used on any website regardless of programming language or configuration.

• Requires minimal install, we include it by default with Arecibo.

31


12.3 Disadvantages

• Requires client to have JavaScript enabled. Most browsers will have this enabled; however, robots and otherprograms will not. On the other hand, do you really want to get all 404’s from robots?

• Exposes your public key to others who may also use this key.

• Exposes variables and data in the source. You may not want this information to be known for security purposes.

32 Chapter 12. Sample Javascript (Browser) Client

CHAPTER

THIRTEEN

SAMPLE PYTHON CLIENT

The Python client allows you to easily send errors either via HTTP. The Python client can be used independently, oras part of a greater implementation. First get the arecibo library:

pip install arecibo

Quick example:

from arecibo import postarecibo = post()arecibo.server(url="http://yoursite/")arecibo.set("account", "yourpublicaccountnumber")arecibo.set("status", "403")arecibo.set("url", "http://badapp.org")arecibo.send()

This will do a HTTP POST to the server. If you’d like to do an email:

from arecibo import postarecibo = post()arecibo.server(email="[email protected]")arecibo.transport = "smtp" # not necessary for httparecibo.set("account", "yourpublicaccountnumber")arecibo.set("status", "403")arecibo.set("url", "http://badapp.org")arecibo.send()

13.1 Requirements

• Tested on Python 2.4, 2.5 and 2.6, however any 2.x version of Python is likely to be sufficient. Permission tomake a HTTP post to the Arecibo server is needed.

• The simplejson library is included for compatibility with versions of Python before 2.6.

13.2 Notes

• The only value set automatically is server, which we set to the output of socket.gethostname() (if that’s available)

• To prevent timeouts, the library does change the socket timeout value, sends the HTTP request and then instantlysets it back to the original value. This should minimize the impact on any other socket connections.

33


34 Chapter 13. Sample Python Client

CHAPTER

FOURTEEN

SAMPLE RUBY CLIENT

The Ruby client allows you to easily send errors via HTTP. The Ruby client can be used independently, or as part of agreater implementation. Quick example:

require ’arecibo’dict = {

:account => ’yourpublicaccountnumber’,:priority => 1,:url => "http://badapp.org",:uid => "123124123123",:ip => "127.0.0.1",:type => "An error",:server => "Test Script"

}p = Arecibo.new("http://yoursite/v/1/", dict)p.send

14.1 Requirements

• If using the http client, permission to make a HTTP post to the Arecibo server is needed.

14.2 Notes

• No values are automatically set.

• There is 10 second timeout set on the HTTP post to prevent hangs.

35


36 Chapter 14. Sample Ruby Client

CHAPTER

FIFTEEN

SAMPLE PHP CLIENT

The PHP client allows you to easily send errors via HTTP. The PHP client can be used independently, or as part of agreater implementation. Quick example.

<?phpinclude("arecibo.php");$fields = array(

"account" => "yourpublicaccountnumber","status" => "403","url" => "http://badphpapp.org"

);post("http://yoursevers/v/1/", $fields);?>

15.1 Requirements

• Tested on PHP 5.2.6.

• If using the http client, permission to make a HTTP post to the Arecibo server is needed.

15.2 Notes

• No values are automatically set.

37


38 Chapter 15. Sample PHP Client

CHAPTER

SIXTEEN

SAMPLE PLONE CLIENT

Note: this has not been updated to the newest version of Arecibo which means it will try and contact the main site.You’ll need to alter it to point your server.

This is a specific Arecibo implementation for Plone.

16.1 Installation

Add in the following into your buildout.cfg. Under eggs add:

clearwind.arecibo

Under zcml add:

clearwind.arecibo

Re-run buildout and restart Plone. Then

• Go to your Add/Remove Products and install Arecibo.

• Then enter your Arecibo public API key.

Installation is now complete.

16.2 Requirements

• Tested on Plone 3.1.5.1, it uses eggs and would be unlikely to work on anything earlier than this. However thecore part of the code to send to Arecibo could be easily refactored into Plone 3 or 2 if needed.

• If you wish to use the HTTP transport, then the Plone process will need to be allowed to make HTTP posts tothe Arecibo server.

16.3 Notes

• By default we set all 404 as priority 5, 500 as priority 1 and the rest as priority 3. This can be altered in thewrapper.py module.

• The UID that Plone generates is passed on to Arecibo, meaning that by default: UID (for 500 errors only), IP,server, type, traceback, status, user_agent, url, server, priority and notes are set.

39


• To prevent timeouts, the library does change the socket timeout value, sends the HTTP request and then sets itback to the original value. This should minimize the impact on any other socket connections.

• Redirect errors are ignored.

40 Chapter 16. Sample Plone client

CHAPTER

SEVENTEEN

SAMPLE DJANGO CLIENT

This is a specific Arecibo installation for Django.

17.1 Installation

Install django_arecibo from pypi:

pip install django_arecibo

This will pull down the arecibo library as well.

17.2 Configuration

Add the following to settings.py:

ARECIBO_PUBLIC_ACCOUNT_NUMBER = "yourpublicaccountnumber"ARECIBO_SERVER_URL = "http://url.to.your.server"

Optionally if you want to use email:

ARECIBO_SERVER_EMAIL = "[email protected]"

Emailing in errors is currently only supported by the App Engine server.

There are two ways to send errors to Arecibo. You can either add in some middleware or add to your custom errorhandlers. If you do both you’ll likely end up posting everything twice.

17.3 Adding to your middleware

Django has a middleware layer to handle errors from views. This means that url routing errors 404’s (for example)that do not reach a view, do not get processed. Add in the following line to the middleware in Django:

django_arecibo.middleware.AreciboMiddleware

So that it would look something like this:

MIDDLEWARE_CLASSES = (’django.middleware.common.CommonMiddleware’,’django.contrib.sessions.middleware.SessionMiddleware’,

41


’django.contrib.auth.middleware.AuthenticationMiddleware’,’django_arecibo.middleware.AreciboMiddleware’,

)

One disadvantage of the middleware is that it means your unit tests will raise errors. It’s also slightly harder tocustomise, unless you write your own middleware.

17.4 Adding to your custom views

Django has a 404 or 500 error handler that can be overridden. We can use this to post to Arecibo, for example here’sa standard 500 error handler:

from django.template import RequestContext, loaderfrom django.http import HttpResponse

def application_error(request):t = loader.get_template(’500.html’)c = RequestContext(request)return HttpResponse(t.render(c), status=500)

Add into this your post to Arecibo so it reads:

from django.template import RequestContext, loaderfrom django.http import HttpResponsefrom django_arecibo.wrapper import post

def application_error(request):t = loader.get_template(’500.html’)uid = post(request, 500)c = RequestContext(request, {"uid": uid})return HttpResponse(t.render(c), status=500)

This is relatively easy to do since, you’ll be writing a handler 500 anyway. Feel free to customize your data to sendthrough different priorities etc. In your 500.html template, you’ll now have access to the UID that was posted toArecibo if you’d like to display that to users.

17.5 Using celery

You can use celery to delay the posting of Arecibo so that your error page is not dependent upon the Arecibo request.To use a celery version of post, change the following:

from django_arecibo.wrapper import post

To:

from django_arecibo.tasks import post

You must add the following to your Django settings.py so that Celery will correctly import the task:

CELERY_IMPORTS = (’django_arecibo.tasks’,)

If you are using middleware, you can do this by using instead:

’django_arecibo.middleware.AreciboMiddlewareCelery’

42 Chapter 17. Sample Django Client


17.6 Further configuration

You can pass through an extra configuration dictionary for filtering or excluding variables posted to Arecibo. Forexample, if you have a field “password” in your site and you don’t want this ever posted to Arecibo, you can excludethis:

ARECIBO_SETTINGS = {’EXCLUDED_POST_VARS’: [’password’,],

}

The options are:

• EXCLUDED_POST_VARS - a list of the fields you’d like not to post to Arecibo.

• EXCLUDED_FILES - a list of the files you’d like not to send information about to Arecibo.

• FILTERED_POST_VARS - instead of sending the value of the field, sends * instead.

• FILTERED_FILES - instead of sending information about the file, sends * instead.

• CALLBACKS - a list of Python methods to be called before processing the error so that you could filter outerrors that you don’t want to send, by whatever logic you’d like. Any method that returns False (or None) stopprocessing that error. For example let’s exclude all 404’s from the GoogleBot:

def stop_google(request, status, **kw):if (’Googlebot’ in request.META.get(’HTTP_USER_AGENT’) and status == 404):

return Falsereturn True

ARECIBO_SETTINGS = {’CALLBACKS’: [stop_google,]# These can be any importable function for example:# ’myapp.utils.arecibo_callback’

}

• GROUP_POSTS - boolean, True or False to activate grouping of posts. If you do this, then all errors get passedto celery and wait for GROUP_WAIT seconds before sending it. It will then count the number of times this errorhas occurred in those GROUP_WAIT seconds. It will then send only ONE error, with the count of the numberof times it occurred. This is to prevent one problem on the site that causes lots of errors filling up your site withjunk. For example if the database goes down, one error will suffice on that subject, not 6,000 per second. Note:this requires celery and memcache to be functioning.

• GROUP_WAIT - period to wait for GROUP_POSTS.

17.7 Other times

You can send an error to Arecibo at pretty much any time, for example if you need to capture an error in a view. Youcan just call the post method and pass through the request.

17.8 Notes


17.6. Further configuration 43


• The Django documentation does suggest that a 500 may not be rendered. It is conceivable that your error couldbe so bad that Arecibo never gets called. For example you could be losing network connections or somethingworse like a syntax error. In this case not much can save you.

• The following errors will be set automatically: url, ip, traceback, type, msg, status, uid and user_agent.

44 Chapter 17. Sample Django Client

CHAPTER

EIGHTEEN

SAMPLE RAILS CLIENT

Note: this has not been updated to the newest version of Arecibo which means it will try and contact the main site.You’ll need to alter it to point your server.

This is a specific Arecibo implementation for Ruby on Rails.

18.1 Installation

Check out the Rails client from git.

Place clients/trunk/rails/ into your vendor/plugins directory.

Add in the following two lines into your Rails environment settings, normally this would be config/environment.rb:

ARECIBO_ACCOUNT_NUMBER = ’yourpublicaccount’ARECIBO_RESULT_TEMPLATE = "layouts/arecibo"

Add the following into your application controller (controllers/application.rb):

def rescue_action(exception)report_to_arecibo(exception)

end

And finally copy the file from arecibo/assets/arecibo.rhtml into app/views/layouts. This is so that once the error hasbeen sent to Arecibo, an error page is rendered. You can change the ARECIBO_RESULT_TEMPLATE to point to anerror file of your liking or simply change the one now in place.

Installation is now complete.

18.2 Requirements

• If you wish to use the HTTP transport, then the Rails process will need to be allowed to make HTTP posts tothe Arecibo server.

18.3 Notes


45


• A UID is generated by Rails is passed on to Arecibo, meaning that by default: UID, IP, server, type, traceback,url, status, user_agent, server and priority are set.

• This uses the Ruby library which sets a 10 second time out.

46 Chapter 18. Sample Rails client

CHAPTER

NINETEEN

RELEASE NOTES

Version 3.0:

• June 2010: merging of the none App Engine port in. You can now run Arecibo without App Engine. Followingis a summary of the lots of changes made in the none App Engine port.

– Removing of issues.

– New model definitions, with some db indexes and denormalisation.

– Ability to name and alter groups of errors.

– Updated stats, new charts which are much simpler.

– Ability for anonymous views and posting of errors, since we can now run the server behind a firewall.

• June 2010: freezing of the App Engine port. There should be no changes to this.

Version 2.0:

• Nov 2010: cleaning up CSS and the filter form on the project page.

• Nov 2010: posting errors from django now can remove sensitive data (thanks offbyone).

• Sept 7th 2010: test for readthedocs.org

• July 21th 2010: Adding in markdown and a few ui tweaks. Also only show approved users in issues.

• July 16th 2010: Apologies for not keeping this up to date.

• July 16th 2010: Ability to make errors public. Helps with debugging.

• July 16th 2010: Issue signals added: issue_status_changed, issue_priority_changed, issue_assigned_changed,issue_changed

• July 16th 2010: Prettier buttons thanks to http://www.zurb.com/article/266/super-awesome-buttons-with-css3-and-rgba

• Big change, beginning of July, Issues, you can now create issues from errors, assign them to people andall that sort of good stuff. This is quite a few changes and at this time they are pretty alpha, theremight be multiple changes on them. Issues are a big change and need lots and lots more work see:http://www.agmweb.ca/blog/andy/2275/

• June 20th 2010: alright, not many people are using it, let’s just keep going anyway

• will be frozen and released any day now.

• June 14th 2010: added in notes about remote access.

Version 1.0:

• this was the privately hosted version by Clearwind and was not open sourced.

47

http://www.zurb.com/article/266/super-awesome-buttons-with-css3-and-rgba

http://www.zurb.com/article/266/super-awesome-buttons-with-css3-and-rgba

http://www.agmweb.ca/blog/andy/2275/


48 Chapter 19. Release Notes

CHAPTER

TWENTY

INDICES AND TABLES

• genindex

• modindex

• search

49

arecibo documentation - read the docs · the variables arecibo_public_account_number and...

Documents