2

Has this happened to you?• Email to users results in

50+ undeliverable

• Need to verify the users in Active Directory

• Then “deactivate” former employees in Crowd

• 750 mouse clicks later, you’re done! http://www.flickr.com/photos/left-hand/4231405740/

Scripting Atlassian applications with Python

Dave ThomasConfiguration Management Architect, FIS

3

Automate That!

4

Agenda

• Use cases for scripting

• Atlassian APIs available for scripting

• The awesome power and simplicity of python

• Examples

5

When is scripting useful?

• Automate time consuming tasks

• Perform data analysis

• Cross-reference data from multiple systems

6

Some specific use cases

• Crowd – Deactivate Users and remove from all groups

• Bamboo – Disable all plans in a project

• JIRA – Release Notes

• Subversion – custom commit acceptance

• Custom build processes – pull code linked to a specific issue into a patch archive

Why Scripts?Why Not Plugins?

7

• I’m not a Java Developer

• Installing new plugins can require a restart

• Prefer to minimize ad hoc changes on the server

• Need to correlate information from several systems

• Need an agile process to accommodate changing requirements

8

APIs for scripting(that we avoid if possible)

• The user interface• Can do anything a user can do• Reporting tasks are relatively easy (particularly when xml is available)• Actions are relatively hard (and prone to breakage)• Capture browser traffic with livehttpheaders, firebug, etc• Form token checking can be an obstacle

• XML-RPC and SOAP• Relatively low-level interface• Many actions available• Relatively complex to use

9

More APIs for scripting(the ones we prefer to use)

• RESTful Remote APIs (now deprecated)• High level interface• Supports a handful of actions

• Now emerging: “real” REST interfaces• High level interface• Supports a handful of actions• http://confluence.atlassian.com/display/REST/Guidelines+for+Atlassi

an+REST+API+Design

10

Why Python?• Powerful standard libraries

• Http(s) with cookie handling• XML and JSON• Unicode

• Third Party Libraries• SOAP• REST• Templates• Subversion

• Portable, cross-platform

11

Python Versions

• 2.x• Ships with most linux distributions• Lots of third-party packages available

• 3.x• Latest version• Deliberately incompatible with 2.x• Not as many third-party libraries

12

HTTP(s) with Python• Python 2

• httplib – low level, all HTTP verbs• urllib – GET and POST, utilities• urllib2 – GET and POST using Request class, easier manipulation

of headers, handlers for cookies, proxies, etc.• Python 3

• http.client – low level, all HTTP verbs• http.parse - utilities• urllib.request – similar to urllib2

• Third-Party• httplib2 – high-level interface with all HTTP verbs, plus caching,

compression, etc.

13

Example 1JIRA Issue Query & Retrieval

14

Discovering URLs for XML

15

Simple Issue Retrievalimport urllib, httplibimport xml.etree.ElementTree as etree

jira_serverurl = 'http://jira.atlassian.com'jira_userid = 'myuserid'jira_password = 'mypassword'

detailsURL = jira_serverurl + \"/si/jira.issueviews:issue-xml/JRA-9/JRA-9.xml" + \"?os_username=" + jira_userid + "&os_password=" + jira_password

f = urllib.urlopen(detailsURL)tree=etree.parse(f)f.close()

Construct a URL that looks like the one in the UI, with extra parms for

our user auth

Open the URL with one line!Parse the XML with one

line!

16

Find details in XML

details = tree.getroot()print "Issue: " + details.find("channel/item/key").textprint "Status: " + details.find("channel/item/status").textprint "Summary: " + details.find("channel/item/summary").textprint "Description: " + details.find("channel/item/description").text

Issue: JRA-9Status: OpenSummary: User Preference: User Time ZonesDescription: <p>Add time zones to user profile. That way the dates displayed to a user are always contiguous with their local time zone, rather than the server's time zone.</p>

Find based on tag name or path to

element

17

Behind the scenes…cookies!httplib.HTTPConnection.debuglevel = 1f = urllib.urlopen(detailsURL)

send: 'GET /si/jira.issueviews:issue-xml/JRA-9/JRA-9.xml?os_username=myuserid&os_password=mypassword HTTP/1.0\r\nHost: jira.atlassian.com\r\nUser-Agent: Python-urllib/1.17\r\n\r\n'reply: 'HTTP/1.1 200 OK\r\n'header: Date: Wed, 20 Apr 2011 12:04:37 GMTheader: Server: Apache-Coyote/1.1header: X-AREQUESTID: 424x2804517x1header: X-Seraph-LoginReason: OKheader: X-AUSERNAME: myuseridheader: X-ASESSIONID: 19b3b8oheader: Content-Type: text/xml;charset=UTF-8header: Set-Cookie: JSESSIONID=A1357C4805B1345356404A65333436D3; Path=/header: Set-Cookie: atlassian.xsrf.token=AKVY-YUFR-9LM7-97AB|e5545d754a98ea0e54f8434fde36326fb340e8b7|lin; Path=/header: Connection: close

Turn on debugging and see exactly what’s

happening

JSESSIONID cookie sent from JIRA

18

Authentication• User credentials determine:

• The data returned• The operations allowed

• Methods Available:• Basic Authentication• JSESSIONID Cookie• Token Method

19

Basic Authentication

• Authentication credentials passed with each request

• Can be used with REST API

20

JSESSIONID Cookie• Authentication credentials passed once;

then cookie is used

• Used when scripting the user interface

• Can be used with REST API for JIRA, Confluence, and Bamboo

21

Token Method• Authentication credentials passed once;

then token is used

• Used with Fisheye/Crucible REST

• Used with Deprecated Bamboo Remote API

22

Obtaining a cookie

• Scripting the user interface login page

• Adding parameters to the user interface URL: “?os_username=myUserID&os_password=myPassword”

• Using the JIRA REST API

23

JIRA REST Authenticationimport urllib, urllib2, cookielib, json

# set up cookiejar for handling URLscookiejar = cookielib.CookieJar()myopener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookiejar))

creds = { "username" : jira_userid, "password" : jira_password }queryurl = jira_serverurl + "/rest/auth/latest/session"req = urllib2.Request(queryurl)req.add_data(json.dumps(creds))req.add_header("Content-type", "application/json")req.add_header("Accept", "application/json")fp = myopener.open(req) fp.close()

urllib2 handles cookies automatically. We just need to

give it a CookieJar

Request and response are both

JSONWe don’t care about response, just the

cookie

24

Submitting a JIRA Querywith the user interface

# Search using JQLqueryJQL = urllib.quote("key in watchedIssues()")queryURL = jira_serverurl + \ "/sr/jira.issueviews:searchrequest-xml/temp/SearchRequest.xml" + \ "?tempMax=1000&jqlQuery=" + queryJQLfp = myopener.open(queryURL)

# Search using an existing filterfilterId = "20124"queryURL = jira_serverurl + \ "/sr/jira.issueviews:searchrequest-xml/" + \ "{0}/SearchRequest-{0}.xml?tempMax=1000".format(filterId)fp = myopener.open(queryURL)

Pass any JQL Query

Or Pass the ID of an existing shared filter

25

A JQL Query using REST# Search using JQLqueryJQL = "key in watchedIssues()"IssuesQuery = { "jql" : queryJQL, "startAt" : 0, "maxResults" : 1000 }queryURL = jira_serverurl + "/rest/api/latest/search"req = urllib2.Request(queryURL)req.add_data(json.dumps(IssuesQuery))req.add_header("Content-type", "application/json")req.add_header("Accept", "application/json")fp = myopener.open(req)data = json.load(fp)fp.close()

Pass any JQL Query

Request and response are both

JSON

26

XML returned from user interface query

An RSS Feed with all issues and requested

fields that have values

27

JSON returnedfrom a REST query

{u'total': 83, u'startAt': 0, u'issues': [{u'self': u'http://jira.atlassian.com/rest/api/latest/issue/JRA-23969',

u'key': u'JRA-23969'}, {u'self': u'http://jira.atlassian.com/rest/api/latest/issue/JRA-23138',

u'key': u'JRA-23138'}, {u'self': u'http://jira.atlassian.com/rest/api/latest/issue/BAM-2770',

u'key': u'BAM-2770'}, {u'self': u'http://jira.atlassian.com/rest/api/latest/issue/BAM-2489',

u'key': u'BAM-2489'}, {u'self': u'http://jira.atlassian.com/rest/api/latest/issue/BAM-1410',

u'key': u'BAM-1410'}, {u'self': u'http://jira.atlassian.com/rest/api/latest/issue/BAM-1143',

u'key': u'BAM-1143'}], u'maxResults': 200}

A list of the issues found, with links to

retrieve more information

28

JSON issue details

All applicable fields are returned, even if

there’s no value

Expand the html property to get rendered html for description,

comments

29

What’s the difference?<reporter username="mlassau">Mark Lassau [Atlassian]</reporter><customfield id="customfield_10160" key="com.atlassian.jira.toolkit:dayslastcommented">

<customfieldname>Last commented</customfieldname><customfieldvalues>

1 week ago</customfieldvalues>

</customfield>

u'reporter': {u'type': u'com.opensymphony.user.User', u'name': u'reporter', u'value': {

u'self': u'http://jira.atlassian.com/rest/api/latest/user?username=mlassau', u'displayName': u'Mark Lassau [Atlassian]', u'name': u'mlassau'}},

u'customfield_10160': {u'type': u'com.atlassian.jira.toolkit:dayslastcommented', u'name': u'Last commented', u'value': 604800},

XML values are display strings

REST values are type-dependent

REST vs. non-REST

30

REST

• More roundtrips to query JIRA and get issue details

• Returns all fields

• Values require type-specific interpretation

• Easier to transition issues

• Easier to get info for projects, components

Non-REST

• Can query based on existing filter

• XML returns only fields that contain values

• Values always one or more display strings

• Can do anything a user can do (with a little work)

31

Example 2Cross-referencing JIRA, Fisheye, and

Bamboo build results

32

Which build resolved my issue?• Bamboo keeps track of “related issues” (based on issue IDs

included in commit comments), but doesn’t know when issues are resolved.

• If we know the issue is resolved in JIRA, we can look to see the latest build that lists our ID as a “related issue”

• Not a continuous integration build? We’ll need to look in fisheye to determine the highest revision related to this issue and then look in bamboo to see if a build using this revision has completed successfully.

33

To Fisheye for related commits!

queryURL = FisheyeServer + "/rest-service-fe/changeset-v1/listChangesets" + \ "?rep={0}&comment={1}&expand=changesets".format(FisheyeRepo, myissue)

req = urllib2.Request(queryURL)auth_string = '{0}:{1}'.format(fisheye_userid,fisheye_password)base64string = base64.encodestring(auth_string)[:-1]req.add_header("Authorization", "Basic {0}".format(base64string))response = myopener.open(req)issuecommits=etree.parse(response).getroot()response.close()

Query a specific fisheye repository for a commit with our JIRA issue ID in the

comments

Use basic auth headers to

authenticate

34

Fisheye changesets returned<results expand="changesets"> <changesets> <changeset>

<csid>130948</csid> <date>2011-04-29T12:35:56.150-04:00</date> <author>lc6081</author> <branch>trunk</branch> <comment>MYJIRAPROJECT-2823 Modified to add parameters</comment> <revisions size="1" />

</changeset> </changesets></results>

35

Parsing the changesetscommits = []for changeset in issuecommits.findall("changesets/changeset"): commits.append(changeset.findtext("csid"))

commits.sort()print "Highest commit is: " + commits[-1]

Highest commit is: 130948

36

Logging into Bamboo

urllib2.HTTPCookieProcessor(cookiejar))

cookiejar = cookielib.CookieJar()myopener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookiejar))

queryURL = bambooServer + "/userlogin!default.action“

params = urllib.urlencode({ "os_username" : bambooUserid, "os_password" : bambooPassword})response = myopener.open(queryURL, params) response.close()

Using a POST to the user interface login screen to

retrieve a JSESSIONID cookie

37

Querying for build results

# Warning: This is a very resource-intensive operation. # You should consider limiting the number of builds returnedqueryURL = bambooServer + "/rest/api/latest/result/MYPROJECT-MYPLAN" + \ "?expand=results[-10:-1].result.jiraIssues"

req = urllib2.Request(queryURL)req.add_header("Accept", "application/xml")response = myopener.open(req)results=etree.parse(response).getroot()response.close()

Use negative indexes to return the last entries in build list, e.g. [-10:-1] returns last ten builds in list

Request the related issues

Ask for XML(JSON also available)

38

Example (partial) build results

<results expand="results"><link href="http://mybamboo.domain.com:8080/rest/api/latest/result/MYPROJECT-MYPLAN" rel="self" /><results expand="result" max-result="25" size="46" start-index="0"><result expand="comments,labels,jiraIssues,stages" id="3146125" key="MYPROJECT-MYPLAN-26" lifeCycleState="Finished" number="26" state="Successful">

<link href="http://mybamboo.domain.com:8080/rest/api/latest/result/MYPROJECT-MYPLAN-26" rel="self" /><buildStartedTime>2011-04-29T05:04:14.460-05:00</buildStartedTime><buildCompletedTime>2011-04-29T05:34:35.687-05:00</buildCompletedTime><buildRelativeTime>4 days ago</buildRelativeTime><vcsRevisionKey>4483</vcsRevisionKey><buildReason>Code has changed</buildReason><comments max-result="0" size="0" start-index="0" /><labels max-result="0" size="0" start-index="0" /><jiraIssues max-result="1" size="1" start-index="0">

<issue iconUrl="http://myjira.domain.com/images/icons/bug.gif" issueType="Defect" key="MYJIRAPROJECT-1629" summary="Need to display an error message when balance is zero."><url href="http://myjira.domain.com/browse/MYJIRAPROJECT-1629" rel="self" />

</issue></jiraIssues><stages max-result="1" size="1" start-index="0" />

</result></results></results>

Can also expand comments, labels, and

stages

jiraIssues property has been expanded here

39

Walking through build resultsfor result in results.findall("results/result"): print result.get("key") + ":" print "\tRevision: " + result.findtext("vcsRevisionKey") issues = [issue.get("key") for issue in result.findall("jiraIssues/issue")] print "\tIssues: " + ", ".join(issues)

MYPROJECT-MYPLAN-31: Revision: 4489 Issues: MYJIRAPROJECT-1658MYPROJECT-MYPLAN-30: Revision: 4486 Issues: MYJIRAPROJECT-1630MYPROJECT-MYPLAN-29: Revision: 4485 Issues: MYJIRAPROJECT-1616, MYJIRAPROJECT-1663

40

Example 3Removing a user from a Crowd group

41

Beyond GET and POST

connection = httplib.HTTPConnection('myCrowdServer.mydomain.com:8080')operation = 'DELETE'urlpath = "/rest/usermanagement/latest/user/group/direct" + \ "?username={0}&groupname={1}".format(userToRemove, fromGroup)body = Noneauth_string = '{0}:{1}'.format(crowdAppName, crowdAppPassword)base64string = base64.encodestring(auth_string)[:-1]headers = {'Authorization' : "Basic {0}".format(base64string)}connection.request(operation, urlpath, body, headers)response = connection.getresponse()print response.status, response.reasonconnection.close()

204 - group membership is successfully deleted 403 - not allowed to delete the group membership 404 - the user or group or membership could not be found

Lower-level HTTPConnection needed

Authenticate as a Crowd

Application

42

A few loose ends

• Be prepared to handle Unicode strings

• Error handling – not shown here, but important!

• Formatting output – several python libraries for handling templates are available

• REST Interfaces – you can write your own!http://confluence.atlassian.com/display/DEVNET/Plugin+Tutorial+-+Writing+REST+Services

43

Links for more information

• http://confluence.atlassian.com/display/JIRA/Displaying+Search+Results+in+XML

• http://confluence.atlassian.com/display/JIRA/JIRA+REST+API+(Alpha)+Tutorial

• http://confluence.atlassian.com/display/CONFDEV/Confluence+REST+APIs

• http://confluence.atlassian.com/display/FECRUDEV/REST+API+Guide

• http://confluence.atlassian.com/display/BAMBOO/Bamboo+REST+APIs

• http://confluence.atlassian.com/display/CROWDDEV/Crowd+REST+APIs