django in the office: get your admin for nothing and your sql for free

Django in the Office:Get Your Admin for Nothing

and Your SQL for Free(Department of Molecular & Cellular Biology)

11.8.2011

“I wish I had a database”

source: Bureaucratics by Jan Banning

http://www.janbanning.com/gallery/bureaucratics/

http://www.janbanning.com/gallery/bureaucratics/

custom websites/databases

pre-2004/05 2005+

Django: python-based web framework

source: http://xkcd.com/353/

http://xkcd.com/353/

http://xkcd.com/353/

Python and Django: Under-Hyped, Widely Used

Django: Scalable (without magic and superclouds)

Firefox’s Add-Ons Site Commenting System

250k add-ons 17,000 requests/second 150 million views/month 40 million users/month 500+ million api hits/day 450,000 websites (rewritten from CakePHP) 15 million profiles

75 million comments

(stats from June 2011) (stats from 10/2010)

can start small: instrument scheduler

(stats from July 2011. initial launch in jan/feb 2009)

Django: Maintenance/Security

Django- 2011 – 2 updates- 2010 – 2 updates- 2009 – 2 updates

- Only update is the core Django code (python files) - No changes to your code

- No changes to the database- No changes to HTML templates- No changes to static or uploaded files

Python and Django: Online and Local Support

Python: “For Dummies” - list

>>> l = ['cat', 'fish', 7, 'eats’, 108]>>> l.sort()>>> for x in l:... print x 7108cateatsfish

>>> len(x)5

Python: “For Dummies” - dict

>>> d = {1:'Bob', 2:'Susan', ‘buckle’:’shoe'}>>> d.keys()[1, 2, ‘buckle’]

>>> for k,v in d.iteritems():... print k, v.capitalize()1 Bob 2 Susan buckle Shoe

>>> print d.get(4, ‘not found’)not found

what about in the office?

story of a 1-day, limited-use project(“raw example/bad naming, etc”)

The Request: Wednesday

Wednesday: The Request

• 200+ trainees

• 40 different labs or offices

• 8 training dates

• 2 types of training

• 5 trainee classifications

Wednesday: The Response

Let’s make a database!



You can manage it through a website admin!




Sortable column headers!





You can have searching and filters!






We can send personalized emails with RSVP links!!






We can send personalized emails with RSVP links!!

Just in case, you can download it all back to Excel!

Wednesday: “The Vision”

Wednesday: “Over Promising”

Thursday: Resource Constraints

1 FTE dependent on the 77 bus

Thursday: Response -> Task List (9:30am)

1- Let’s make a database!

2 - You can manage it through a website admin!

3 - Sortable column headers!

4 - You can have searching and filters!

5 - We can send personalized emails with RSVP links!!

6 - Just in case, you can download it all back to Excel!

a bit of setup: choose database (9:15am)

In the settings.py file:

DATABASES = { 'default': { 'ENGINE': 'django.db.backends.sqlite3',

# Add 'postgresql_psycopg2', 'postgresql', 'mysql', 'sqlite3' or 'oracle'. 'NAME': '/db/mcb_hcom.db3', 'USER': '', 'PASSWORD': '', 'HOST': '', 'PORT': '', }}

a bit of setup: start an “app” (9:20am)

>python manage.py startapp hcom_training

Data models: models.py (still 9:20am)

/hcom_training/models.py -> SQL tables!admin.py

a bit of setup: choose database (9:15am)

In the settings.py file:

INSTALLED_APPS = ( #... 'hcom_training', #...

)

data models: think (9:30 to 9:40)

What would C.J. Date do?

data models: 1st think (9:30 to 9:45)

data models

class Trainee(models.Model): """ Trainee Information """

fname = models.CharField(‘First Name’, max_length=50) lname = models.CharField(‘Last Name’, max_length=50)

email= models.EmailField(blank=True) confirmed_training_date = models.BooleanField(default=False)

location = models.ForeignKey(LocationInfo) # training session (+ 13 more fields)

data models

class LocationInfo(models.Model): """ Training Session Information """

name = models.CharField(max_length=255)

training_datetime = models.DateTimeField()

room = models.CharField(max_length=100)

(+2 more fields)

models.py: create your tables (10:15am)

> python manage.py validate0 errors found

> python manage.py syncdbCreating tables ...Creating table hcom_training_specialCreating table hcom_training_labofficeCreating table hcom_training_locationinfoCreating table hcom_training_traineeCreating table hcom_training_traineemessageInstalling custom SQL ...Installing indexes ...No fixtures found

models.py

>python manage.py sqlall hcom_training

CREATE TABLE `hcom_training_locationinfo` ( ìd` integer AUTO_INCREMENT NOT NULL PRIMARY KEY, `name` varchar(255) NOT NULL, `training_type` varchar(100) NOT NULL, `training_datetime` datetime NOT NULL, ...

CREATE TABLE `hcom_training_trainee` ( ìd` integer AUTO_INCREMENT NOT NULL PRIMARY KEY, `fname` varchar(50) NOT NULL, `lname` varchar(50) NOT NULL, èmail` varchar(75) NOT NULL, `confirmed_training_date` bool NOT NULL, `location_id` integer, `lab_or_office_id` integer NOT NULL, ...

2 files: data models + admin

/hcom_training/models.py -> SQL tables!admin.py -> web-based admin,

sortable columns, searching and filters

admin.py: 1 line of code (“You can manage it through a website admin!”)

admin.site.register(Trainee)

admin.py: 3 lines of code (“Sortable column headers!”)

class TraineeAdmin(admin.ModelAdmin):

list_display = ('lname', ’fname', 'email', 'confirmed_training_date’, 'completed_training’)admin.site.register(Trainee, TraineeAdmin)

admin.py: 4 lines of code (“You can have searching and filters!)

class TraineeAdmin(admin.ModelAdmin): list_display = ('lname', ’fname', 'email', 'confirmed_training_date’, 'completed_training’) search_fields = ('lname', 'fname', 'email', ‘location_room’)admin.site.register(Trainee, TraineeAdmin)

admin.py: 5 lines of code

class TraineeAdmin(admin.ModelAdmin): list_display = ('lname', ’fname', 'email', 'confirmed_training_date’, 'completed_training’) search_fields = ('lname', 'fname', 'email', ‘location_room’) list_filter = ('confirmed_training_date', 'completed_training', 'location')

admin.site.register(Trainee, TraineeAdmin)

admin.py: 5 lines of code

class TraineeAdmin(admin.ModelAdmin):

list_display = ('lname', 'fname', 'email', 'confirmed_training_date', 'completed_training')

search_fields = ('lname', 'fname', 'email', 'location__room')

list_filter = ('confirmed_training_date', 'completed_training', 'location')

admin.site.register(Trainee, TraineeAdmin)

2 Files: Models & Admin

../hcom_training/models.py -> SQL tables!admin.py -> web-based admin,

sortable columns, searching and filters

admin.py -> just scratching the surface

class TraineeAdmin(admin.ModelAdmin): list_display = ('lname','fname', 'email', 'active', 'confirmed_training_date','completed_training', 'special', 'approver_or_shopper', 'lab_or_office', 'hands_on_training_date', 'demo_training_date',) list_editable = ('completed_training', ) search_fields = ('lname', 'fname', 'email',) list_filter = ('active', 'confirmed_training_date', 'completed_training','has_hands_on_training', 'has_demo_training','special','approver_or_shopper', 'location',) readonly_fields = ( 'has_hands_on_training', 'has_demo_training', 'last_update', 'id_md5', 'num_emails_sent','confirmation_link' ) inlines = [ TraineeMessageInline, ] fieldsets = [ ('name / email', {'fields': [( 'fname','lname',), 'active', 'email', 'special', 'lab_or_office','last_update', 'num_emails_sent',]}) ,('training', {'fields': ['confirmed_training_date', 'completed_training', 'training_order', 'approver_or_shopper', 'location', ('has_hands_on_training', 'hands_on_training_date',)\ , ('has_demo_training', 'demo_training_date'), ]})\ ,('Notes', {'fields': [ 'notes', 'id_md5' ,'confirmation_link' ]})\ ] admin.site.register(Trainee, TraineeAdmin)

models.py -> shell -> ADD

>python manage.py shell

>>> t = Trainee(fname='Raman', lname='Prasad' , email='[email protected]')

>>> t.save()

>>> print t.id, t.lname, t.fname, t.email1457 Prasad Raman [email protected]

>>> t.delete()

mailto:[email protected]




models.py -> shell -> QUERY

>>> l = Trainee.objects.filter(completed_training=False, special__name='Faculty')

>>> l.count() 23

>>> for t in l:... print t.fname, t.lname

(publicly listing tardy faculty members = bad idea!)

models.py -> shell -> EDIT

>>> new_location = LocationInfo( name='special faculty training' , training_type='Hands-On Shopper' , training_date='2011-11-15'

, training_time='10:00'

, room='NW 404')

>>> new_location.save()

>>> for t in l:... t.location = new_location... t.save()

models.py -> direct database access

>python manage.py dbshell

SQLite version 3.7.6Enter ".help" for instructionsEnter SQL statements terminated with a ";"sqlite>.tableshcom_training_laboffice hcom_training_locationinfo hcom_training_special hcom_training_trainee hcom_training_traineemessage

models.py -> dumpdata

>python manage.py dumpdata hcom_training --indent=4{ "pk": 8, "model": "hcom_training.locationinfo", "fields": { "training_type": "Shopper Demo", "room": "Biolabs room 1080 (Lecture Hall)", "training_date": "2011-06-02", "training_time": "10:00:00", "week_num": "3", "name": "Shopper Demo: Thursday June 2nd 10am -12pm @ Biolabs room 1080 (Lecture Hall)" } }, { "pk": 1277, "model": "hcom_training.trainee", "fields": { "demo_training_date": "2011-05-20", "notes": "", "completed_training": true, "lab_or_office": 279, "confirmed_training_date": true, "has_demo_training": true, "last_update": "2011-05-23 10:39:14", "lname": "Akhmetova", "approver_or_shopper": "SHOPPER", "location": 3, "fname": "Laila", "active": true, "training_order": 14, "has_hands_on_training": false, "id_md5": "d759175de8ea5b1d9a2660e45554894f", "email": "[email protected]", "special": 356 } },

> JSON, XML data dump

> JSON, XML data load

models.py -> but there’s more!

>python manage.py dumpdata hcom_training –-format=xml --indent=4

{<object pk="9" model="hcom_training.locationinfo"> <field type="CharField" name="name">special faculty training</field> <field type="CharField" name="training_type">shopper</field> <field type="DateField" name="training_date">2011-11-15</field> <field type="TimeField" name="training_time">10:00:00</field> <field type="CharField" name="week_num"></field> <field type="CharField" name="room"></field> </object> <object pk="1277" model="hcom_training.trainee"> <field type="CharField" name="fname">Laila</field> <field type="CharField" name="lname">Akhmetova</field> <field type="BooleanField" name="active">True</field> <field to="hcom_training.special" name="special" rel="ManyToOneRel">356</field> <field type="CharField" name="email">[email protected]</field> <field type="DateField" name="hands_on_training_date"><None></None></field> <field type="BooleanField" name="has_hands_on_training">False</field> <field type="DateField" name="demo_training_date">2011-05-20</field> <field type="BooleanField" name="has_demo_training">True</field> <field type="BooleanField" name="confirmed_training_date">True</field> <field type="BooleanField" name="completed_training">True</field> <field to="hcom_training.locationinfo" name="location" rel="ManyToOneRel">3</field> <field type="CharField" name="approver_or_shopper">SHOPPER</field> <field to="hcom_training.laboffice" name="lab_or_office" rel="ManyToOneRel">279</field> <field type="IntegerField" name="training_order">14</field> <field type="TextField" name="notes"></field> <field type="DateTimeField" name="last_update">2011-05-23 10:39:14</field> <field type="CharField" name="id_md5">d759175de8ea5b1d9a2660e45554894f</field> </object>

models.py -> loaddata

>python manage.py dumpdata hcom_training > hcom_2011_1106.json

>python manage.py loaddata hcom_2011_1106.json

Installed 750 object(s) from 1 fixture(s)

Data In/Out

- Web-based Admin

-Python Shell or Python Scripts/Programs

- dumpdata / loaddata (json, xml, yaml)

- database shell / db specific commands (MySQL, sqlite, etc)

All for 2 files!!







personalized emails: message

Dear Raman Prasad, You have been selected as an individual who will be placing orders via the Harvard Crimson Online Marketplace or "HCOM" procurement system on behalf of your lab or group. For REQUIRED Hands-on Shopper training, please save the SAVE THE DATE:

Time: May 18th, 2011 at 10am Location: Science Center, Rm. 226.

The training will last 2 hours and lunch will be provided. To RSVP, please click on the following link: http://mcb.harvard.edu/hcom/confirm/2812e5cf6d8f21d69c91dddeefb792a7/

If for any reason, you are not able to attend this training, please contact me immediately to make alternate arrangements. Thank you.

personalized emails: message as a template

Dear {{ trainee.fname }} {{ trainee.lname }}, You have been selected as an individual who will be placing orders via the Harvard Crimson Online Marketplace or "HCOM" procurement system on behalf of your lab or group. For REQUIRED Hands-on {{ trainee.training_type }} training, please save the SAVE THE DATE:

Time: {{ trainee.location.training_date|date:"F jS, Y at P" }} Location: {{ trainee.location.room }}

The training will last 2 hours and lunch will be provided. To RSVP, please click on the following link: {% url view_hcom_confirmation trainee.id_md5 %}

If for any reason, you are not able to attend this training, please contact me immediately to make alternate arrangements. Thank you.

personalized emails: use the template

for trainee in Trainee.objects.all(): msg = render_to_string('confirmation_email.txt’

, { 'trainee': trainee } )

personalized emails

for trainee in Trainee.objects.all(): msg = render_to_string('confirmation_email.txt', \ { 'trainee': trainee } ) subject = 'Required HCOM Training at %s' % trainee.location.name

from_email ='[email protected]’ to_addresses = [ trainee.email ] bcc = ['[email protected]'] send_mail(subject, msg, from_email, to_addresses, bcc=bcc)

Source: https://docs.djangoproject.com/en/dev/topics/email/

personalized emails: the link

http://mcb.harvard.edu/hcom/confirm/2812e5cf6d8f21d69c91dddeefb792a7/

urlpatterns = patterns( 'hcom_training.views'

, url(r'^hcom/confirm/(?P<trainee_id>(\w){32})/$', 'view_hcom_confirmation'

, name='view_hcom_confirmation'))

personalized emails: simplified view

def view_hcom_confirmation(request, trainee_id): """Simple view showing trainee confirmation page""" trainee = Trainee.objects.get(id_md5=trainee_id) # get trainee trainee.confirmed_training_date = True # set confirmed to true trainee.save() # save the information # display web page return render_to_response('hcom_templates/hcom_confirm_page.html'

, { ‘trainee’: trainee}, context_instance=RequestContext(request))

personalized emails: the view

def view_hcom_confirmation(request, trainee_id): """Simple view showing trainee confirmation page""" lu = {} # Find the trainee; make sure he/she exists try: trainee = Trainee.objects.get(id_md5=trainee_id) lu.update({ 'trainee' : trainee }) except Trainee.DoesNotExist: lu.update({ 'ERR_MSG' : True, 'ERR_trainee_not_found' : True }) return render_to_response('hcom_templates/hcom_confirm_page.html', lu, context_instance=RequestContext(request)) # Check if the person has already confirmed if trainee.confirmed_training_date: # Yes, err message lu.update({ 'ERR_MSG' : True , 'ERR_already_confirmed' : True }) else: # New confirmation, update database trainee.confirmed_training_date = True trainee.save() # show confirmation page to use return render_to_response('hcom_templates/hcom_confirm_page.html', lu, context_instance=RequestContext(request))

All for 2 files!!







export as Excel

used python xlwt library

style_info_cell = easyxf('pattern: pattern solid, fore_colour white; align: wrap on;')

book = xlwt.Workbook(encoding="utf-8") # make Excel workbooksheet1 = book.add_sheet('hcom training') # add a sheet row_num = 2for trainee in Trainee.objects.all(): sheet1.write(row_num, 1, trainee.lname, style_info_cell) # lname sheet1.write(row_num, 2, trainee.fname, style_info_cell) # fname sheet1.write(row_num, 3, trainee.email, style_info_cell ) # email (…) row_num += 1

Example: https://webapps.sciences.fas.harvard.edu/mcb/mcb-control-panel/hcom_training/

https://webapps.sciences.fas.harvard.edu/mcb/mcb-control-panel/hcom_training

All for 2 files!!







Project Results

- Emails sent out by 3:30/4pm after an hour of Q/A

Time savings

- RSVPs started coming. Database updated as email recipients clicked link

- 35 people re-scheduled. No need to update shared spreadsheet. Staff used dropdown box in admin to change dates and add notes.

- Able to send reminder emails before each training date

Project Results

- Filters used to generate attendance spreadsheets

- Tracked “completed training” via the admin

- System used as new employees come on board

- Home on time

- BUT: Should have been a 1.5 to 2 day project

Spreadsheets -> Databases

- Genome Modification Center

- 46 tables

- 2,389 objects for 500+ projects

- Life Sciences Course Tracker

- 37 Tables

- 1,293 objects including 798 semesters of course data

Spreadsheets -> Databases

Conclusion: Django Models + Admin can take you far….

Spreadsheets -> Databases -> Models

But there’s more!!

>python manage.py inspectdb <your existing db>

Learning more .. .

Django official site – Work through the Tutorial!!https://www.djangoproject.com/

Python http://python.org

Dive into Pythonhttp://www.diveintopython.net/toc/index.html

[email protected]

https://www.djangoproject.com/

http://python.org/

http://www.diveintopython.net/toc/index.html

thank you

django in the office: get your admin for nothing and your sql for free

Technology