Python II: Applied Python
Table of ContentsEVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 TOCAbout this Course
Chapter 1 – Python Refresher
Chapter 1 Objectives...................................................................................................1-1
Variables......................................................................................................................1-2
Python data types........................................................................................................1-3
Sequences....................................................................................................................1-4
Mapping types.............................................................................................................1-7
Program structure......................................................................................................1-10
Files and console I/O.................................................................................................1-12
Conditionals..............................................................................................................1-14
Loops.........................................................................................................................1-15
Defning a function....................................................................................................1-16
Function parameters..................................................................................................1-17
Builtins......................................................................................................................1-19
Chapter 2 – OS Services
Chapter 2 Objectives...................................................................................................2-1
The os module.............................................................................................................2-2
Environment variables................................................................................................2-6
Launching external programs......................................................................................2-8
Paths, directories, and flenames...............................................................................2-11
Walking directory trees.............................................................................................2-16
Dates and times.........................................................................................................2-18
The time module.......................................................................................................2-21
The calendar module.................................................................................................2-27
Python II: Applied Python TOC-1© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
TOC rev. 3.0
Chapter 3 – Pythonic Programming
Chapter 3 Objectives...................................................................................................3-1
Pythonic programming................................................................................................3-2
Common Python idioms.............................................................................................3-3
Slicing and dicing.......................................................................................................3-4
Unpacking function arguments...................................................................................3-7
Lambda functions........................................................................................................3-8
Nested functions........................................................................................................3-10
List comprehensions..................................................................................................3-11
Iterables.....................................................................................................................3-13
Generator expressions...............................................................................................3-14
Writing Generators....................................................................................................3-16
Python Time Travel...................................................................................................3-18
Three Python Easter Eggs.........................................................................................3-19
A String Trick............................................................................................................3-20
String formatting.......................................................................................................3-21
Chapter 4 – Modules and Packages
Chapter 4 Objectives...................................................................................................4-1
Modules.......................................................................................................................4-2
Using import..............................................................................................................4-4
Initialization code........................................................................................................4-5
Namespaces.................................................................................................................4-6
Executing modules as scripts......................................................................................4-7
Packages......................................................................................................................4-9
Confguring import with __init__.py........................................................................4-11
Name resolution (AKA scope)..................................................................................4-13
Nested functions........................................................................................................4-11
Python style...............................................................................................................4-15
TOC-2 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 TOCChapter 5 – Classes
Chapter 5 Objectives...................................................................................................5-1
About OO Programming.............................................................................................5-2
Defning classes..........................................................................................................5-3
Constructors................................................................................................................5-4
Instance methods.........................................................................................................5-7
Properties....................................................................................................................5-8
Class methods and data.............................................................................................5-10
Static methods...........................................................................................................5-11
"Private" methods.....................................................................................................5-12
Inheritance.................................................................................................................5-13
Untangling the nomenclature....................................................................................5-16
Chapter 6 – Metaprogramming
Chapter 6 Objectives...................................................................................................6-1
Metaprogramming.......................................................................................................6-2
Special attributes.........................................................................................................6-3
globals() and locals()...................................................................................................6-4
Working with attributes...............................................................................................6-5
The inspect module.....................................................................................................6-7
Decorators...................................................................................................................6-9
Decorator functions...................................................................................................6-10
Decorator classes.......................................................................................................6-12
Decorating classes.....................................................................................................6-15
Creating classes at runtime.......................................................................................6-16
Monkey patching.......................................................................................................6-17
Python II: Applied Python TOC-3© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
TOC rev. 3.0
Chapter 7 – Developer Tools
Chapter 7 Objectives...................................................................................................7-1
Program development.................................................................................................7-2
Comments...................................................................................................................7-3
pylint...........................................................................................................................7-4
Customizing pylint.....................................................................................................7-5
Using pyreverse.........................................................................................................7-6
The unittest module....................................................................................................7-7
Skipping tests............................................................................................................7-10
Making a suite of tests...............................................................................................7-11
Automated test discovery..........................................................................................7-13
Using Nose................................................................................................................7-14
The Python debugger................................................................................................7-15
Starting debug mode.................................................................................................7-16
Stepping through a program......................................................................................7-17
Setting breakpoints....................................................................................................7-18
Profling.....................................................................................................................7-19
Benchmarking...........................................................................................................7-20
Chapter 8 – Distributing Modules
Chapter 8 Objectives...................................................................................................8-1
Installing packages......................................................................................................8-2
Ways to distribute code...............................................................................................8-3
Overview of distutils..................................................................................................8-4
Preparing for distribution............................................................................................8-5
Creating a source distribution.....................................................................................8-7
Creating built distributions.........................................................................................8-8
setup.py options.........................................................................................................8-9
setup.py commands..................................................................................................8-10
Code portability.........................................................................................................8-11
TOC-4 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 TOCChapter 9 – Database access
Chapter 9 Objectives...................................................................................................9-1
The DB API.................................................................................................................9-2
Available Interfaces.....................................................................................................9-3
Connecting to a server.................................................................................................9-4
connect() examples.....................................................................................................9-5
Creating a cursor.........................................................................................................9-6
Executing a statement.................................................................................................9-7
Fetching data...............................................................................................................9-8
Tip: Making an iterator for large queries..................................................................9-10
Parameterized statements..........................................................................................9-11
Dictionary cursors.....................................................................................................9-12
Metadata....................................................................................................................9-15
Transactions..............................................................................................................9-17
Object-relational mappers.........................................................................................9-18
Chapter 10qt – GUI Programming with PyQt4
Chapter 10qt Objectives.........................................................................................10qt-1
What is PyQt4?......................................................................................................10qt-2
Event Driven Applications.....................................................................................10qt-3
GUI Application Flow Chart..................................................................................10qt-4
External Anatomy of a PyQt4 Application............................................................10qt-5
Internal Anatomy of a PyQt4 Application.............................................................10qt-6
Using designer......................................................................................................10qt-7
Anatomy of a designer-based application..............................................................10qt-9
Naming Conventions...........................................................................................10qt-10
Common Widgets.................................................................................................10qt-11
Layouts.................................................................................................................10qt-13
Selectable Buttons................................................................................................10qt-14
Making your application stretch..........................................................................10qt-15
Actions and events...............................................................................................10qt-16
Menu bar..............................................................................................................10qt-19
Status bar..............................................................................................................10qt-20
Using predefned dialogs.....................................................................................10qt-22
Creating custom dialogs.......................................................................................10qt-25
Tabs......................................................................................................................10qt-27
Niceties................................................................................................................10qt-29
Working with images...........................................................................................10qt-30
Complete example...............................................................................................10qt-32
Python II: Applied Python TOC-5© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
TOC rev. 3.0
Chapter 10tk – GUI Programming with Tkinter
Chapter 10tk Objectives.........................................................................................10tk-1
Tkinter Overview...................................................................................................10tk-2
Basic Tkinter Programming...................................................................................10tk-4
Object-oriented Tkinter..........................................................................................10tk-5
Widgets..................................................................................................................10tk-6
Labels.....................................................................................................................10tk-8
Buttons...................................................................................................................10tk-9
Setting fonts.........................................................................................................10tk-10
Colors...................................................................................................................10tk-11
Standard Colors....................................................................................................10tk-13
Variable wrappers.................................................................................................10tk-15
Selectable buttons................................................................................................10tk-16
Text entry blanks..................................................................................................10tk-18
Multiline text entry/display..................................................................................10tk-19
Listbox.................................................................................................................10tk-21
Arranging widgets................................................................................................10tk-23
Using pack()........................................................................................................10tk-24
Tweaking the layout.............................................................................................10tk-26
Frames..................................................................................................................10tk-29
Adding scrollbars.................................................................................................10tk-31
Callbacks..............................................................................................................10tk-33
Callback parameters.............................................................................................10tk-34
Binding events.....................................................................................................10tk-36
Event specifcations.............................................................................................10tk-37
Creating menus....................................................................................................10tk-40
TOC-6 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 TOCChapter 11 – Network Programming
Chapter 11 Objectives...............................................................................................11-1
Sockets......................................................................................................................11-2
Socket options...........................................................................................................11-3
Client concepts..........................................................................................................11-4
Server concepts.........................................................................................................11-5
Application protocols................................................................................................11-6
Forking servers..........................................................................................................11-7
Grabbing HTML from the Web.................................................................................11-8
Consuming Web services........................................................................................11-10
Web data the easier way..........................................................................................11-11
Sending email..........................................................................................................11-14
Binary data..............................................................................................................11-16
The struct module...................................................................................................11-17
Chapter 12 – Multiprogramming
Chapter 12 Objectives...............................................................................................12-1
Multiprogramming....................................................................................................12-2
What are threads?......................................................................................................12-3
The Python thread manager......................................................................................12-4
The threading module..............................................................................................12-5
Threads for the impatient..........................................................................................12-6
Creating a thread class..............................................................................................12-7
Variable sharing.........................................................................................................12-4
Using queues...........................................................................................................12-12
Debugging threaded programs................................................................................12-15
The multiprocessing module...................................................................................12-16
Alternatives to multiprogramming..........................................................................12-19
Python II: Applied Python TOC-7© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
TOC rev. 3.0
Chapter 13 – System Administration
Chapter 13 Objectives...............................................................................................13-1
The subprocess module.............................................................................................13-2
subprocess convenience functions............................................................................13-4
Using the sh module.................................................................................................13-7
Remote access.........................................................................................................13-10
Other useful modules..............................................................................................13-13
Permissions.............................................................................................................13-14
Saving information..................................................................................................13-16
Creating a useful command line script....................................................................13-17
Creating flters.........................................................................................................13-18
Parsing the command line.......................................................................................13-21
Simple logging........................................................................................................13-25
Logging levels.........................................................................................................13-26
Formatting log entries.............................................................................................13-28
Logging to other destinations..................................................................................13-30
Chapter 14 – XML and JSON
Chapter 14 Objectives...............................................................................................14-1
About XML...............................................................................................................14-2
Normal approaches to XML.....................................................................................14-3
Which module to use?...............................................................................................14-4
Getting started with ElementTree.............................................................................14-5
How ElementTree works...........................................................................................14-6
Creating a new XML Document...............................................................................14-8
Parsing an XML Document.....................................................................................14-11
Navigating the XML Document..............................................................................14-12
Using XPath............................................................................................................14-15
About JSON............................................................................................................14-19
Reading JSON.........................................................................................................14-20
Writing JSON..........................................................................................................14-21
TOC-8 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 TOCChapter 15 – Extending Python
Chapter 15 Objectives...............................................................................................15-1
Why extend Python?.................................................................................................15-2
Ways to extend Python with C..................................................................................15-3
Hand-coded C...........................................................................................................15-4
Overview...................................................................................................................15-5
The C program..........................................................................................................15-6
Methods.....................................................................................................................15-7
The method table.......................................................................................................15-8
The init function........................................................................................................15-9
Handling errors.......................................................................................................15-10
Custom exception objects.......................................................................................15-11
Putting it all together...............................................................................................15-13
Using SWIG............................................................................................................15-16
The interface fle.....................................................................................................15-17
Generating the wrappers.........................................................................................15-20
Building and installing the extension......................................................................15-21
ctypes......................................................................................................................15-22
For more information..............................................................................................15-17
Appendix A: Books
Appendix B: String Formatting
Python II: Applied Python TOC-9© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
TOC rev. 3.0
TOC-10 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
Python II: Applied Python
About this Course
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 About this Course
Welcome!
● We're glad you're here
● You should be a Python programmer at a basic level
● Class has hands-on labs for nearly every chapter
● Please make a name tent
● Instructor name: ____________________________
● Instructor e-mail: ___________________________
Have Fun!
Python II: Applied Python ATC-1© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
About this course rev. 3.0
Classroom etiquette
● Noisemakers off
● No phone conversations
● Come and go quietly during class.
Please turn off boom boxes, cell phone ringers and other noisemakers.
If you need to have a phone conversation, please leave the classroom.
We're adults here; feel free to leave the classroom if you need to use the restroom, make aphone call, etc. You don't have to wait for a lab or break, but please try not to disturb yourfellow students.
If you have an accordion with you, please leave it in the case. Some students may be frightened by it.
ATC-2 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 About this Course
Course Outline
Day 1Chapter 1 Python Refresher
Chapter 2 OS Services
Chapter 3 Pythonic Programming
Chapter 4 Modules and Packages
Day 2Chapter 5 Classes
Chapter 6 Metaprogramming
Chapter 7 Developer Tools
Chapter 8 Distributing Modules
Day 3Chapter 9 Database Access
Chapter 10 GUI Programming
Chapter 11 Network Programming
Chapter 12 Multiprogramming
Day 4Chapter 13 System Administration
Chapter 14 XML and JSON
Chapter 15 Extending PythonOptional: other topics as suggested by you
The actual schedule varies with circumstances. The last day typically includes ad hoc topics requested by students
Python II: Applied Python ATC-3© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
About this course rev. 3.0
Student flesYou will need to load some student fles onto your computer. These fles include
● data for exercises
● example scripts
● answers to exercises
● resources
The fles may be on a CD, a USB drive, or the instructor will give you a URL from whichthey can be downloaded.
The fles are in a compressed archive. When you extract them onto your computer, they will all be extracted into a directory named python2.
python2/EXAMPLES contains the examples from the course manuals.
python2/ANSWERS contains sample answers to the labs.
python2/DATA contains data fles for examples and labs.
python2/SETUP setup scripts and data
The student files do not contain Python itself. It will need to be installed separately. This has probably already been done for you.
ATC-4 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 About this Course
Extracting the student fles
Unix (includes Unix, Linux, MacOS, etc)
Copy the fle python2.tgz to any your home directory, them from there, type:
tar xzvf python2.tgzNote: if your version of Unix is elderly, its tar command may not support the z option. If this is so, use this command line instead:
gzip -dc python2.tgz | tar xvf -This will create the python2 directory under your home directory.
Windows
Open the fle python2.zip. Extract all fles to desktop or any other convenient location. This will create the folder python2.
Included is a file named python2_links.html, which contains links to many useful sites related to Python programming
Python II: Applied Python ATC-5© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
About this course rev. 3.0
Examples
● Most examples from course manual provided in EXAMPLES subdirectory
● First line of output shows how script was invoked
For example:
external.py
#!/usr/bin/python
import os
os.system("who")
ls = os.popen("ls -l /tmp","r")
for entry in ls: print entry[:-1]print
# backticks equivalenth = os.popen("uname -n").read()[:-1]
print h
external.pyjstrick :0 2008-08-25 11:47 jstrick pts/0 2008-08-25 13:06 (:0.0) jstrick pts/1 2008-08-25 13:13 (:0.0) total 44 drwx------ 3 jstrick jstrick 4096 2008-08-25 11:47 gconfd-jstrickdrwx------ 2 root root 4096 2008-08-25 13:10 gconfd-root drwx------ 2 jstrick jstrick 4096 2008-08-25 11:47 keyring-Ii5L9tsrwxr-xr-x 1 jstrick jstrick 0 2008-08-25 11:47 mapping-jstrick drwx------ 2 jstrick jstrick 4096 2008-08-25 18:14 orbit-jstrick drwx------ 2 root root 4096 2008-08-25 13:10 orbit-root srwxr-xr-x 1 jstrick jstrick 0 2008-08-25 11:48 OSL_PIPE_1000_SingleOfficeIPC_709aa55976c37ee189ec71e931.2573 drwx------ 2 jstrick jstrick 4096 2008-08-25 16:28 plugtmp drwx------ 2 jstrick jstrick 4096 2008-08-25 11:47 ssh-uIikNW5481drwxr-xr-x 3 jstrick jstrick 4096 2008-08-25 17:46 svh07.tmp drwx------ 2 jstrick jstrick 4096 2008-08-25 18:14 v1.2170 drwx------ 2 jstrick jstrick 4096 2008-08-25 17:39 v97670 drwx------ 2 jstrick jstrick 4096 2008-08-25 11:47 virtual-jstrick.Hfflbh
gimli
ATC-6 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 About this Course
Lab Exercises
● Relax – a lab is not a quiz
● Feel free to modify labs
● Ask the instructor for help
● Work on your own scripts or data
Python II: Applied Python ATC-7© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
About this course rev. 3.0
Appendices
● Appendix A: Python Books
● Appendix B: Python Gotchas
ATC-8 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
Python II: Applied Python
Chapter 1
Python Refresher
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 Python Refresher
Chapter 1 Objective
● Review basic Python syntax
Python II: Applied Python 1-1© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
Python Refresher rev. 3.0
Variables
● Variables are declared by assignment
● Type of variable is set dynamically
Variables are declared by assigning to them. Python does not require explicit type specifers, but sets the type implicitly by examining the value that was assigned.
Thus, assigning a number to a variable creates a variable of type int, while assigning text to a variable creates a variable of type string. Once a variable is assigned to, it will cause an error if the variable is used with an operator or function that is inappropriate for the type.
A variable cannot be used before it is assigned to.
Typing DisciplinesDynamic Static
Weak awkcsh, ksh, bash
JavaScriptPerlPHP
Visual Basic
Strong PythonCommon Lisp
Ruby
C, C++C#
COBOLFortran
JavaVisual Basic
1-2 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 Python Refresher
Basic Python Data types
● Python has many data types
● Use type name builtin to convert
Python has many data types. There are builtin functions to convert from one type to another. If the source type cannot be converted to the target type, a TypeError is thrown.
● Numeric types
• bool
• int
• long (merged with int in python 3)
• foat
• complex
● Sequence types
• str (renamed to bytes in Python 3)
• unicode (renamed to str in Python 3)
• list
• tuple
● Mapping types
• dict
• set
• frozenset
Python II: Applied Python 1-3© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
Python Refresher rev. 3.0
Sequences
● Strings are text (arrays of characters)s = "text";
● Lists are ordered sets of valuesmy_list = []
● sequence[start:limit:stride]
● Tuples are immutable listsmy_tuple = ()
Python supports three types of sequences – strings (including Unicode), lists, and tuples. All sequences share a common set of operations, methods, and builtin functions; each type has operations specifc to that type.
A string can be thought of as a list of characters.
All sequences support slicing, which means returning a subset of the sequence using the [start,limit,step] syntax.
Note that the starting value of a slice is inclusive, while the endingvalue is exclusive.
1-4 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 Python RefresherExamplesequences.py
def main(): list_demo() sep() tuple_demo() sep() string_demo()
def list_demo(): colors = ['red', 'blue', 'purple', 'pink'] print colors colors.append('green') print colors colors.extend(['brown', 'black', 'white']) print colors c1 = colors.pop() c2 = colors.pop(0) # AKA shift print c1, c2 print colors
for color in colors: print color
print colors[:4] print colors[-4:] print colors[:-1]
def tuple_demo(): person = 'Eric', 'Idle', 'Robin', 'yellow' print person[:2]
def string_demo(): name = 'Fred E. Flintstone' print len(name) print name[:4] print name[8:13] print name.upper() print name.count('e') print name.lower().count('e')
s = ' foo ' print s.lstrip() + '!' print s.rstrip() + '!' print s.strip() + '!'
def sep(): print '-' * 50
if __name__ == '__main__': main()
Python II: Applied Python 1-5© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
Python Refresher rev. 3.0
sequences.py['red', 'blue', 'purple', 'pink']['red', 'blue', 'purple', 'pink', 'green']['red', 'blue', 'purple', 'pink', 'green', 'brown', 'black', 'white']white red['blue', 'purple', 'pink', 'green', 'brown', 'black']bluepurplepinkgreenbrownblack['blue', 'purple', 'pink', 'green']['pink', 'green', 'brown', 'black']['blue', 'purple', 'pink', 'green', 'brown']--------------------------------------------------('Eric', 'Idle')--------------------------------------------------18FredFlintFRED E. FLINTSTONE23foo ! foo!foo!
1-6 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 Python Refresher
Mapping Types
● Dictionaries are mapped sets of values
● Sets are similar to dictionaries but contain only keys
● Syntaxdict = { }
set = set()
fset = frozenset()
Python also supports unordered sets of values. These non-sequential sets of data are also called mapping types.
A dictionary is a set of values indexed by an immutable keyword. Dictionaries are used for many tasks, including mapping one set of values to another, and counting occurrencesof values.
A set is an unindexed set of values. Sets are used to fnd unique values, and to test for membership. A frozenset is an immutable set, whose contents are determined when it is declared.
Dictionaries are good for counting things. There are several ways to count with dictionaries.
Python II: Applied Python 1-7© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
Python Refresher rev. 3.0
Examplecount_with_dictionaries.py
import collections
fruits = ''' peach apple banana apple peach apple banana apricot peach apple apple peach'''.split()
def main(): d1 = count_with_dict1() display_counts("count_with_dict1", d1)
d2 = count_with_dict2() display_counts("count_with_dict2", d2)
d3 = count_with_dict3() display_counts("count_with_dict3", d3)
def count_with_dict1(): d = {} for fruit in fruits: if fruit in d: d[fruit] = d[fruit] + 1 else: d[fruit] = 1 return d
def count_with_dict2(): d = {} for fruit in fruits: d[fruit] = d.get(fruit,0) + 1 return d
def count_with_dict3(): d = collections.Counter() for fruit in fruits: d[fruit] += 1 return d
def display_counts(label, d): print label print '-' * len(label) for fruit, count in sorted(d.iteritems()): print fruit, count print
# program entry pointif __name__ == '__main__': main()
1-8 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 Python Refresher
count_with_dictionaries.pycount_with_dict1----------------apple 5apricot 1banana 2peach 4
count_with_dict3----------------apple 5apricot 1banana 2peach 4
count_with_dict3----------------apple 5apricot 1banana 2peach 4
Process finished with exit code 0
Python II: Applied Python 1-9© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
Python Refresher rev. 3.0
Program structure
● Variables, functions, and classes must be declared before use
● Normal order is
- Imports
- Global variables
- Main Function
- Other Functions
- Call to Main Function
In Python, modules must be imported before their contents may be accessed. Variables, functions, and classes must be declared before they can be used. Thus most scripts are ordered in this way:
1. import statements
2. global variables (AKA constants)
3. main function (typically named main() although this is not required)
4. other functions
5. call to main()
You may want to make a template for your Python scripts. Most editors and IDEs support code templates or insertable snippets.
1-10 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 Python RefresherExample#!/usr/bin/env python''' description of application by Author (c) YYYY Author or Organization'''#imports – system, then third-party, then localimport sys
# constants (global variables)
# main functiondef main(args): pass
# other functions...
# program entry pointif __name__ == '__main__': main(sys.argv[1:])
Python II: Applied Python 1-11© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
Python Refresher rev. 3.0
Files and console I/O
● open()
● raw_input()
To output to the screen, use the print statement. print normally outputs a newline after its arguments, but a trailing comma suppresses that behavior. print puts spaces between its arguments.
To avoid the spaces, either use string formatting, or join the strings together with an empty string.
To read a fle, open it with the open() function as part of a with statement.
To read it line by line, iterate through the fle with a for loop. To read the entire fle, use fle.read(); to read all the lines into a list, use fle.readlines(). To read a specifed numberof bytes, use fle.read(n).
To navigate within a fle, use fle.seek(offset,whence); to get the current location, use fle.tell().
To get input from the user, use raw_input(). It provides a prompt to the user, and returns a string, with then newline already trimmed.
1-12 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 Python RefresherExamplereading_and_writing_fles.py
def main(): read_file() print write_to_file()
def read_file(): with open('../DATA/mary.txt') as MARY: for line in MARY: print line,
def write_to_file(): colors = ['red','green','purple', 'pink', 'orange'] with open('colors.txt', 'w') as COLORS_OUT: for color in colors: COLORS_OUT.write(color + '\n')
with open('colors.txt') as COLORS_IN: for line in COLORS_IN: print line,
if __name__ == '__main__': main()
reading_and_writing_files.pyMary had a little lamb,Its fleece was white as snow,And everywhere that mary wentThe lamb was sure to go
redgreenpurplepinkorange
Python II: Applied Python 1-13© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
Python Refresher rev. 3.0
Conditionals
● Test a Boolean value
● Variations on if
● Syntax
if EXPR:elif EXPR:else:
The conditional statement in Python, like most languages, is if. There are several variations on how if is used. All depend on testing a value to see whether it is true or false.
The following values are false:
False, None, empty collections, and numeric zero
Everything else is true.
Python has a shortcut if-else that is something like the ?: operator in C, Perl and other curly-brace languages – expression1 if boolean-expression else expression2
Examples
if name == 'root: print "do not run this utility as root"elif name == 'guest': print "sorry – guests are not allowed to run this utilityelse: print "starting processing"
limit = sys.args[1] if len(sys.args) > 1 else 100
1-14 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 Python Refresher
Loops
● Two kinds of loops
● while is used when waiting for a condition (typically waiting for data, as from a fle or database)
● for is used when iterating over a sequence
Python has two kinds of loops.
The while loop is used for reading data, typically from a database or other data source, orwhen waiting for user input to end a loop.
The for loop is used to iterate through a sequence of data. Because Python uses iterators to simplify access to many kinds of data, the for loop is used in places that would use while in most languages. The most frequent example of this is in reading lines from a fle.
Python II: Applied Python 1-15© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
Python Refresher rev. 3.0
Defning a function
● Indent body
● Specify parameters
● Variables are local by default
Functions are one of Python's callable types. Once a function is defned, it can be called from anywhere.
Functions can take fxed or variable parameters and return single or multiple values.
Functions must be defned before they can be called.
Defne a function with the def keyword, the name of the function, a (possibly empty) list of parameters in parentheses, and a colon.
Examplesfunctions.py
def hello(): print "Hello, world"
def sqrt(n): return n ** .5
hello()
n = sqrt(2)m = sqrt(1234)
print "m is ",m," n is",n
functions.pyHello, worldm is 35.1283361405 n is 1.41421356237
1-16 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 Python Refresher
Function parameters
● Three kinds of parametersFixed
Optional
Keyword
● No type checking
When defning a function, you need to specify the parameters that the function expects. There are three ways to do this.
Specify one or more fxed parameters. The interpreter will then expect exactly that many parameters:
def spam(a,b,c): # function bodyThis function expects three parameters, which can be of any Python data type.
Fixed parameters can have default values, in which case the parameters can be omitted when the function is called.
Specify optional arguments. Prefx a parameter with an asterisk:
def eggs(*params): # function body
This function will take any number of arguments, which are then available in the list params.
Specify named parameters. Prefx the parameter with two asterisks.
def spam(**kw): # function bodyThis function takes any number of keyword arguments, which are available in the dictionary kw:
spam(name="bob",grade=10)
Python II: Applied Python 1-17© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
Python Refresher rev. 3.0
Examplesfunction_params.py
spam = "spam"
def fun_one():print "Hello, world"
def fun_two(n):return n ** 2
def fun_three(n,*opt):print "n is ",nprint "opt is",opt
def fun_four(**named_args):for name in named_args:
print name,"==> ",named_args[name]
def fun_five(count=3):for i in range(count):
print spam,print
fun_one()x = fun_two(5)print "fun_two(5) is",xfun_three('apple',"blueberry","peach","cherry")fun_four(name="Lancelot",quest="Grail",color="red")fun_five()fun_five(5)
function_params.pyHello, worldfun_two(5) is 25n is appleopt is ('blueberry', 'peach', 'cherry')color ==> redquest ==> Grailname ==> Lancelotspam spam spamspam spam spam spam spam
1-18 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 Python Refresher
Builtins
● About 60 builtin functions
● Not called from an object or package
● Can work on many different data types
Python has many builtin functions. These provide generic functionality that is not tied to a particular type or package.
They can be applied to many different data types, but not all functions can be applied to all data types.
Python II: Applied Python 1-19© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
Python Refresher rev. 3.0
Chapter 1 Exercises
Exercise 1-1 (pres_states.py)
Using the fle presidents.txt (in the DATA folder), count the number of Presidents who were born in each state. In other words, the output of your script should be a sorted list ofstates, with the number of presidents that were born in that state:
Arkansas 1 California 1 Connecticut 1 Georgia 1 etc
Exercise 1-2 (pres_dates.py)
Write an interactive script that asks for a president's last name. For each president whose last name matches, print out their full name, date of birth and date of death. For presidents who are still alive, print three asterisks for the date.
For the ambitious (pres_dates_amb.py):1. Make the name search case-insensitive2. Change your script to print out matches for partial names – so "Jeff" would fnd "Jefferson", e.g.
1-20 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
Python II: Applied Python
Chapter 2
OS Services
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 OS Services
Chapter 2 Objectives
● Working with the OS
● Running external programs
● Walking through a directory tree
● Working with path names
● Calculating dates and times
Python II: Applied Python 2-1© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
OS Services rev. 3.0
The os module
● Provides OS-specifc services
The os module provides basic services from the operating system. The interface is the same for different operating systems.
2-2 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 OS Services
The os moduleMethod or Data Description
os.path either posixpath or ntpath
os.name either 'posix', 'nt', 'os2' or 'ce'.
os.curdir a string representing the current directory ('.' or ':')
os.pardir a string representing the parent directory ('..' or '::')
os.sep the (or a most common) pathname separator ('/' or ':' or '\\')
os.extsep the extension separator (always '.')
os.altsep the alternate pathname separator (None or '/')
os.pathsep the component separator used in $PATH etc
os.linesep the line separator in text fles ('\r' or '\n' or '\r\n')
os.defpath the default search path for executables
os.devnull the fle path of the null device ('/dev/null', etc.)
abort() Abort the interpreter immediately
access() Use the real uid/gid to test for access to a path
chdir() Change the current working directory to the specifed path.
chmod() Change the access permissions of a fle.
chown() Change owner and group id of path to the numeric uid and gid.
chroot() Change root directory to path.
close() Close a fle descriptor (for low level IO).
closerange() Closes all fle descriptors in list, ignoring errors.
confstr() Return a string-valued system confguration variable.
ctermid() Return name of the controlling terminal
device_encoding() Return string describing the encoding of the device
dup() Return a duplicate of a fle descriptor.
dup2() Duplicate fle descriptor.
fchdir() Change to directory of given fle descriptor.
fchmod() Change permissions of fle given by fle descriptor
fchown() Change owner/group id of the fle given by fle descriptor
fdatasync() force write of fle with fledescriptor to disk.
fork() Fork a child process.
forkpty() Fork a new process with a new pseudo-terminal
fpathconf() Return the confguration limit name for the fle descriptor
fstat() Return stat result for an open fle descriptor.
fsync() force write of fle with fledescriptor to disk.
ftruncate() Truncate a fle to a specifed length.
getcwd() Return unicode string representing current working directory.
getegid() Return the current process's effective group id.
getenv(key, default=None) Get specifed environment variable or None/Default
geteuid() Return the current process's effective user id.
getgid() Return the current process's group id.
getgroups() Return list of supplemental group IDs for the process.
Python II: Applied Python 2-3© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
OS Services rev. 3.0
getloadavg() Return number of processes averaged over 1, 5, and 15 minutes
getlogin() Return the actual login name.
getpgid() Call the system call getpgid().
getpgrp() Return the current process group id.
getpid() Return the current process id
getppid() Return the parent's process id.
getsid() Call the system call getsid().
getuid() Return the current process's user id.
isatty() Return True if fle descriptor is an open fle descriptor
kill() Kill a process with a signal.
killpg() Kill a process group with a signal.
lchown() Change owner/group of path (don't follow symlinks)
link() Create a hard link to a fle.
listdir() Return list of all entries in the directory.
lseek() Set the current position of a fle descriptor.
lstat() Like stat(path), but do not follow symbolic links.
major() Extracts device major number from a raw device number.
makedev() Composes a raw device number from major/minor device numbers.
makedirs() Super-mkdir (like unix mkdir -p)
minor() Extracts device minor number from a raw device number.
mkdir() Create a directory.
mkffo() Create a FIFO (a POSIX named pipe).
mknod() Create a flesystem node
nice() Decrease priority of process by inc and return new priority.
open() Open a fle (for low level IO).
openpty() Open a pseudo-terminal
pathconf() Return confguration limit name for fle or directory path.
pipe() Create a pipe.
putenv() Change or add an environment variable.
read() Read a fle descriptor.
readlink() Return string represention of symlink target
remove() Remove a fle (same as unlink(path)).
removedirs(name) Super-rmdir; remove leaf directory and all empty intermediate ones
rename() Rename a fle or directory.
renames(old, new) Super-rename; create directories as necessary
rmdir() Remove a directory.
setegid() Set the current process's effective group id.
seteuid() Set the current process's effective user id.
setgid() Set the current process's group id.
setgroups() Set the groups of the current process to list.
setpgid() Call the system call setpgid().
setpgrp() Make this process a session leader.
2-4 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 OS Servicessetregid() Set the current process's real and effective group ids.
setreuid() Set the current process's real and effective user ids.
setsid() Call the system call setsid().
setuid() Set the current process's user id.
spawn() Execute fle with arguments from args in a subprocess.
stat() Perform a stat system call on the given path.
stat_foat_times() Determine whether os.[lf]stat represents time stamps as foat objects.
statvfs() Perform a statvfs system call on the given path.
strerror() Translate an error code to a message string.
symlink() Create a symbolic link
sysconf() Return an integer-valued system confguration variable.
system() Execute the command (a string) in a subshell.
tcgetpgrp() Return the process group associated with the terminal given by a fd.
tcsetpgrp() Set the process group associated with the terminal given by a fd.
times() Return tuple of foats indicating process times.
ttyname() Return the name of the terminal device
umask() Set the current numeric umask and return the previous umask.
uname() Return a tuple identifying the current operating system.
unlink() Remove a fle (same as remove(path)).
unsetenv() Delete an environment variable.
utime() Set the access and modifed time of fle
wait(...) Wait for completion of a child process.
walk() Directory tree generator.
write() Write a string to a fle descriptor.
Python II: Applied Python 2-5© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
OS Services rev. 3.0
Environment variables
● Syntaxvalue = os.environ[varname]
value = os.environ.get(varname)
value = os.getenv(varname)
value = os.getenv(varname,default)
str2 = os.path.expandvars(str1)
There are several ways to access environment variables from Python.
The most direct is to use os.environ, which is a dictionary of the current environment. If a non-existent variable name is specifed, a KeyError will be raised, so it is safer to use os.environ.get(varname[,default]) than os.environ[varname].
You can also use the os.getenv(varname[,default]) method. It is a shortcut for os.environ.get().
Another way to use environment variables is to expand a string that contains them, using the expandvars(string) method of the os.path object. This takes a string containing one or more environment variables and returns the strings with environment variables expanded to their values. This is probably only useful to Unix administrators
If the variables do not exist in the environment, they are left unexpanded.
2-6 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 OS ServicesExamplesgetenv_ex.pyimport os
def main(): user = os.getenv("USER") count = os.getenv("COUNT") print "count is",os.environ['COUNT'],"user is",os.environ['USER'] print "count is",os.environ.get('COUNT'),"user is",os.environ.get('USER') print "count is",count,"user is",user print os.path.expandvars("count is $COUNT user is $USER")
if __name__ == '__main__': main()
Unix
COUNT=12 getenv_ex.pycount is 12 user is jstrickcount is 12 user is jstrickcount is 12 user is jstrickcount is 12 user is jstrick
Windows
set count=5 getenv_ex.pycount is 12 user is jstrickcount is 12 user is jstrickcount is 12 user is jstrickcount is 12 user is jstrick
Python II: Applied Python 2-7© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
OS Services rev. 3.0
Launching external programs
● Different ways to launch programs
- Just launch (use system())
- Capture output (use popen())
● import os module
● Use system() or popen() methods
In Python, you can launch an external command using the os module functions os.system() and os.popen().
os.system() launches any external command, as though you had typed it at a command prompt. popen() opens a command, returning a fle-like object. You can read the output of the command with any of the methods used for a fle.
You can open a process for writing as well, by specifying a mode of "w". .
2-8 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 OS ServicesExamplesexternal_unix.py
import os
def main(): os.system("who")
ls = os.popen("ls -l /tmp","r")
for entry in ls: print entry[:-1] print
# backticks equivalent h = os.popen("uname -n").read()[:-1]
print h
if __name__ == '__main__': main()
external_unix.pyjstrick :0 2008-08-25 11:47 jstrick pts/0 2008-08-25 13:06 (:0.0) jstrick pts/1 2008-08-25 13:13 (:0.0) total 44 drwx------ 3 jstrick jstrick 4096 2008-08-25 11:47 gconfd-jstrickdrwx------ 2 root root 4096 2008-08-25 13:10 gconfd-root drwx------ 2 jstrick jstrick 4096 2008-08-25 11:47 keyring-Ii5L9tsrwxr-xr-x 1 jstrick jstrick 0 2008-08-25 11:47 mapping-jstrick drwx------ 2 jstrick jstrick 4096 2008-08-25 18:14 orbit-jstrick drwx------ 2 root root 4096 2008-08-25 13:10 orbit-root srwxr-xr-x 1 jstrick jstrick 0 2008-08-25 11:48 OSL_PIPE_1000_SingleOfficeIPC_709aa55976c37ee189ec71e931.2573 drwx------ 2 jstrick jstrick 4096 2008-08-25 16:28 plugtmp drwx------ 2 jstrick jstrick 4096 2008-08-25 11:47 ssh-uIikNW5481drwxr-xr-x 3 jstrick jstrick 4096 2008-08-25 17:46 svh07.tmp drwx------ 2 jstrick jstrick 4096 2008-08-25 18:14 v1.2170 drwx------ 2 jstrick jstrick 4096 2008-08-25 17:39 v97670 drwx------ 2 jstrick jstrick 4096 2008-08-25 11:47 virtual-jstrick.Hfflbh
JohnLaptopUnix
Python II: Applied Python 2-9© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
OS Services rev. 3.0
external_windows.py
import os
def main(): # just launch a command os.system('ipconfig')
# open a command and read its output d = os.popen(r'dir \temp')
for entry in d: print entry,
# backticks (``) equiv hostname = os.popen('hostname').read()[:-1]
print 'Hostname is',hostname
if __name__ == '__main__': main()
external_windows.py<output will vary>
2-10 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 OS Services
Paths, directories and flenames
● import os.path module
The os.path module provides many functions for working with pathnames and flenames. These are generally platform neutral, with a couple of Windows-specifc functions as well.
Some of the more common methods are
path.os.exists(), path.os.dirname(), path.os.basename, and path.os.join().
Python II: Applied Python 2-11© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
OS Services rev. 3.0
Examplepaths.pyimport sysimport os.path
UNIX_PATH_1 = "/usr/local/bin/foo"UNIX_PATH_2 = "bin/foo.txt"
WIN_PATH_1 = r"\\marmoset\sharing\technology\docs\myproject\intro.doc"WIN_PATH_2 = r"myproject\intro.doc"
def main(): if sys.platform == 'win32': win_examples() else: unix_examples()
def unix_examples(): print "dirname(UNIX_PATH_1): ",os.path.dirname(UNIX_PATH_1) print "dirname(UNIX_PATH_2): ",os.path.dirname(UNIX_PATH_2) print "basename(UNIX_PATH_1): ",os.path.basename(UNIX_PATH_1) print "basename(UNIX_PATH_2): ",os.path.basename(UNIX_PATH_2) print 'expanduser("~"): ',os.path.expanduser("~") print 'expanduser("~jstrick"): ',os.path.expanduser("~jstrick") print "isabs(UNIX_PATH_1): ",os.path.isabs(UNIX_PATH_1) print "isabs(UNIX_PATH_2): ",os.path.isabs(UNIX_PATH_2) print "os.path.split(UNIX_PATH_1) Head: {0} Tail: {1}".format(
*os.path.split(UNIX_PATH_1)) print "os.path.split(UNIX_PATH_2) Head: {0} Tail: {1}".format(
*os.path.split(UNIX_PATH_2))
def win_examples(): print "dirname(WIN_PATH_1): ",os.path.dirname(WIN_PATH_1) print "dirname(WIN_PATH_2): ",os.path.dirname(WIN_PATH_2) print "basename(WIN_PATH_1): ",os.path.basename(WIN_PATH_1) print "basename(WIN_PATH_2): ",os.path.basename(WIN_PATH_2) print "os.path.split(WIN_PATH_1) Head: {0} Tail: {1}".format(
os.path.split(WIN_PATH_1)) print "os.path.split(WIN_PATH_1) Head: {0} Tail: {1}".format(
os.path.split(WIN_PATH_2)) print "os.path.splitunc(WIN_PATH_1) Head: {0} Tail: {1}".format(
os.path.splitunc(WIN_PATH_1)) print "os.path.splitunc(WIN_PATH_1) Head: {0} Tail: {1}".format(
os.path.splitunc(WIN_PATH_2))
if __name__ == '__main__': main()
paths.py (unix)dirname(unix_p1): /usr/local/bindirname(unix_p2): binbasename(unix_p1): foobasename(unix_p2): foo.txtexpanduser("~"): /home/jstrickexpanduser("~jstrick"): /home/jstrickisabs(unix_p1): Trueisabs(unix_p2): Falseos.path.split(unix_p1) Head: /usr/local/bin Tail: fooos.path.split(unix_p2) Head: bin Tail: foo.txt
2-12 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 OS Services
paths.py (windows)dirname(win_p1): \\marmoset\sharing\technology\docs\bonsaidirname(win_p2): \\marmoset\sharing\technology\docs\bonsaibasename(win_p1): foo.docbasename(win_p2): foo.docos.path.split(win_p1) Head: \\marmoset\sharing\technology\docs\bonsai Tail: foo.docos.path.split(win_p1) Head: bonsai Tail: foo.docos.path.splitunc(win_p1) Head: \\marmoset\sharing Tail: \technology\docs\bonsai\foo.docos.path.splitunc(win_p1) Head: Tail: bonsai\foo.doc
Python II: Applied Python 2-13© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
OS Services rev. 3.0
os.path methodsMethod Description
abspath(path) Return normalized absolutized version of the pathname path.
basename(path) Return the base name of pathname path.
commonprefx(list) Return the longest path prefx (taken character-by-character) that is a prefx of allpaths in list. If list is empty, return the empty string ('').
dirname(path) Return the directory name of pathname path.
exists(path) Return True if path refers to an existing path. Returns False for broken symbolic links. May be subject to permissions
lexists(path) Return True if path refers to an existing path. Returns True for broken symbolic links.
expanduser(path) On Unix, return the argument with an initial component of "~" or "~user" replaced by that user's home directory. Only "~" is supported on Windows.
expandvars(path) Return the argument with environment variables expanded. Substrings of the form "$name" or "${name}" are replaced by the value of environment variable name. Malformed variable names and references to non-existing variables are left unchanged.
getatime(path) Return the time of last access of path. (seconds since epoch).
getmtime(path) Return the time of last modifcation of path. (seconds since epoch).
getctime(path) Return the system's ctime. (seconds since epoch).
getsize(path) Return the size, in bytes, of path. Raise os.error if path does not exist or cannot be accessed.
isabs(path) Return True if path is an absolute pathname (begins with a slash).
isfle(path) Return True if path is an existing regular fle. This follows symbolic links.
isdir(path) Return True if path is an existing directory. Follows symbolic links.
islink(path) Return True if path refers to a directory entry that is a symbolic link. Always False on Windows.
ismount(path) Return True if pathname path is a mount point (Unix only).
join(path1[, path2[, ...]])
Join one or more path components intelligently.
normcase(path) Normalize the case of a pathname. On Unix, this returns the path unchanged; on case-insensitive flesystems, it converts the path to lowercase. On Windows, it also converts forward slashes to backward slashes.
normpath(ph) Normalize a pathname. This collapses redundant separators and up-level references so that A//B, A/./B and A/foo/../B all become A/B.
realpath(path) Return the canonical path of the specifed flename, eliminating any symbolic links encountered in the path.
samefle(path1, path2)
Return True if both pathname arguments refer to the same fle or directory (as indicated by device number and i-node number). Raise an exception if a os.stat()call on either pathname fails. Availability: Macintosh, Unix.
sameopenfle(fp1, fp2)
Return True if the fle descriptors fp1 and fp2 refer to the same fle. Availability: Macintosh, Unix.
samestat(stat1, stat2) Return True if the stat tuples stat1 and stat2 refer to the same fle. These structures may have been returned by fstat(), lstat(), or stat(). Availability: Macintosh, Unix.
split(path) Split the pathname path into a pair, (head, tail) where tail is the last pathname component and head is everything leading up to that. The tail part will never contain a slash.
splitdrive(path) Split the pathname path into a pair (drive, tail) where drive is either a drive
2-14 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 OS Servicesspecifcation or the empty string. On systems which do not use drive specifcations, drive will always be the empty string..
splitext(path) Split the pathname path into a pair (root, ext) such that root + ext == path, and ext is empty or begins with a period and contains at most one period.
splitunc(path) Split the pathname path into a pair (unc, rest) so that unc is the UNC mount point (such as r'\\host\mount'), if present, and rest the rest of the path (such as r'\path\fle.ext'). For paths containing drive letters, unc will always be the empty string. Availability: Windows.
walk(path, visit, arg) Calls the function visit with arguments (arg, dirname, names) for each directory in the directory tree rooted at path (including path itself, if it is a directory). Note: The newer os.walk() generator supplies similar functionality and can be easier to use. (Like File::Find in perl)
supports_unicode_flenames
True if arbitrary Unicode strings can be used as fle names (within limitations imposed by the fle system), and if os.listdir() returns Unicode strings for a Unicode argument. New in version 2.3.
Python II: Applied Python 2-15© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
OS Services rev. 3.0
Walking directory trees
● Use os.walk()
● Returns tuple for each directory starting with the specifed top directory
● Tuple contains full path to directory, list of subdirectories, and list of fles
● syntax:for currdir,dirnames,filenames in os.walk("start-dir"):
pass
The os.walk method provides a way to easily walk a directory tree. It provides an iterator for a directory and all its subdirectories. For each directory, it returns a tuple with three values.
The frst element is the full (absolute) path to the directory; the second element is a list of the directory's subdirectories (relative names); the third element is a list of the non-directory entries in the subdirectory (also relative names).
use os.path.join() to put together the directory and the fle or subdirectory name.
Remember not to use dir as a variable name since dir is a builtin function.
2-16 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 OS ServicesExamplewalk_ex.py
# print size of every python file whose name starts with 'm'import os
START_DIR = ".." # start in root of student files
def main():
for currdir,subdirs,files in os.walk(START_DIR): for file in files: if file.endswith('.py') and file.startswith('m'): fullpath = os.path.join(currdir,file) fsize = os.path.getsize(fullpath) print "{0:8d} {1:s}".format(fsize, fullpath)
if __name__ == '__main__': main()
walk_ex.py 1479 ../EXAMPLES/mongodb_example.py 1076 ../EXAMPLES/mammal.py 400 ../EXAMPLES/monkeypatch.py 211 ../EXAMPLES/mary_gen.py 1010 ../EXAMPLES/magic_methods.py 578 ../SETUP/mkdbms_sqllite.py 2625 ../SETUP/mkpres_xml.pyetc etc
Python II: Applied Python 2-17© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
OS Services rev. 3.0
Dates and times
● Use the datetime module
● Provides several objects
datetime
date
time
timedelta
The datetime module provides several classes for manipulating dats and times.
The datetime class stores both a date and time, while date and time store just a date or a time.
strftime() and strptime() are a pair of methods available on a datetime that format a time string (strftime) or parse a formatted time string (strptime). They both use the sameformat characters, as shown in the table on page 2-22.
The timedelta object provides a time interval in days, seconds, and microseconds. A timedelta can be added to or subtracted from a date, time, or datetime object.
The following are examples of how to initialize these objects.
dt = datetime(2011,7,4,11,30,45)d = date(2011,7,4)t = time(11,30,45)td = timedelta(7,0,0)
2-18 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 OS ServicesExamplesdates_and_times.pyfrom datetime import ( datetime as DateTime, date as Date, time as Time, timedelta as TimeDelta)
def main(): todays_date() sep() current_time() sep() date_deltas() sep()
def sep(): print '-' * 50
def todays_date(): print "date.today():",Date.today()
def current_time(): now = DateTime.now() print "now.day:", now.day print "now.month:", now.month print "now.year:", now.year print "now.hour:", now.hour print "now.minute:", now.minute print "now.second:", now.second
def date_deltas(): d1 = DateTime(2013,6,13) d2 = DateTime(2013,8,24)
d3 = d2 - d1
print "raw time delta:",d3 print "time delta days:",d3.days
t1 = DateTime(2007,8,24,10,4,34) t2 = DateTime(2007,8,24,22,8,1) t3 = t2 - t1
print "t1 = datetime(2007,8,24,10,4,34):",t1 print "t2 = datetime(2007,8,24,22,8,1):",t2 print "time diff (t2 - t1):",t3
if __name__ == '__main__': main()
Python II: Applied Python 2-19© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
OS Services rev. 3.0
dates_and_times.pydate.today(): 2013-12-11--------------------------------------------------now.day: 11now.month: 12now.year: 2013now.hour: 16now.minute: 34now.second: 34--------------------------------------------------raw time delta: 72 days, 0:00:00time delta days: 72t1 = datetime(2007,8,24,10,4,34): 2007-08-24 10:04:34t2 = datetime(2007,8,24,22,8,1): 2007-08-24 22:08:01time diff (t2 - t1): 12:03:27--------------------------------------------------
2-20 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 OS Services
The time module
● Provides time-of-day services
● Common methods: time(), sleep(), asctime(), ctime(), strptime(), strftime()
The time module provides various time-of-day services from the OS. The most basic method is time(), which returns the Unix-style timestamp for the current time (number ofseconds since January 1, 1970).
Two related methods are asctime() and ctime(), both of which return a formatted time string with either the current time (the default), or a specifed time. The format is the same for both; the difference is that asctime() takes a 9-tuple of time felds, and ctime() takes a Unix-style time stamp. With no argument, they are more or less equivalent.
The sleep() method will pause for a specifed number of seconds.
The time module also provides a strftime/strptime pair. strftime() takes a 9-tuple or a struct time (named tuple with 9 time felds) and formats the date and time according to the format string. strptime() returns a struct time.
In Python 2, strftime() and strptime() only work for years >= 1900.
Python II: Applied Python 2-21© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
OS Services rev. 3.0
Exampletime_ex.py
import time
def main(): right_now = time.time() print "It is {0} seconds since 1/1/70".format(right_now) print time.sleep(2) then_ctime = time.ctime(right_now) print "it was", then_ctime
now_asc = time.asctime() print "then it was",now_asc
time.sleep(3)
now_ctime = time.ctime() print "it is now",now_ctime
if __name__ == '__main__': main()
time_ex.pyIt is 1386910386.95 seconds since 1/1/70
it was Thu Dec 12 23:53:06 2013then it was Thu Dec 12 23:53:08 2013it is now Thu Dec 12 23:53:11 2013
2-22 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 OS ServicesExampledate_parsing_and_formatting.py
from datetime import datetime as DateTimeimport time
# September 9, 1941 is Dennis Ritchie's BD. # Look him up if you don't know who he is
def main(): format_time() parse_time()
def format_time(): drbd1 = DateTime(1941, 9, 9) now = time.localtime()
print drbd1.strftime('%m/%d/%y') print drbd1.strftime('%B %d, %Y') print
print time.strftime('%m/%d/%y', now) print time.strftime('%B %d, %Y', now) print
def parse_time(): s1 = "September 09, 1941" s2 = "09/09/41" print DateTime.strptime(s1, "%B %d, %Y") print DateTime.strptime(s2, "%m/%d/%y") #oops!
print time.strptime(s1, "%B %d, %Y") print time.strptime(s2, "%m/%d/%y") #oops!
if __name__ == '__main__': main()
date_parsing_and_formatting.py09/09/41September 09, 1941
12/12/13December 12, 2013
1941-09-09 00:00:002041-09-09 00:00:00time.struct_time(tm_year=1941, tm_mon=9, tm_mday=9, tm_hour=0, tm_min=0, tm_sec=0, tm_wday=1, tm_yday=252, tm_isdst=-1)time.struct_time(tm_year=2041, tm_mon=9, tm_mday=9, tm_hour=0, tm_min=0, tm_sec=0, tm_wday=0, tm_yday=252, tm_isdst=-1)
Python II: Applied Python 2-23© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
OS Services rev. 3.0
Formats for strftime/strptime
Directive Meaning %a Locale’s abbreviated weekday name.
%A Locale’s full weekday name.
%b Locale’s abbreviated month name.
%B Locale’s full month name.
%c Locale’s appropriate date and time representation.
%d Day of the month as a decimal number [01,31].
%H Hour (24-hour clock) as a decimal number [00,23].
%I Hour (12-hour clock) as a decimal number [01,12].
%j Day of the year as a decimal number [001,366].
%m Month as a decimal number [01,12].
%M Minute as a decimal number [00,59].
%p Locale’s equivalent of either AM or PM.
%S Second as a decimal number [00,61].
%U Week number of the year (Sunday as the frst day of the week) as a decimal number [00,53]. All days in a new year preceding the frst Sunday are considered to be in week 0.
%w Weekday as a decimal number [0(Sunday),6].
%W Week number of the year (Monday as the frst day of the week) as a decimal number [00,53]. All days in a new year preceding the frst Monday are considered to be in week 0.
%x Locale’s appropriate date representation.
%X Locale’s appropriate time representation.
%y Year without century as a decimal number [00,99].
%Y Year with century as a decimal number.
%Z Time zone name (no characters if no time zone exists).
%% A literal '%' character.
2-24 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 OS Services
Time-Related Modulesdatetime.date
Attributes valuesD.day Day as integer (1-31)
D.month Month as integer (1-12)
D.year Year as integer (4 digits)
min Minimum date (equal to datetime.date(1, 1, 1)
max Maximum date (equal to datetime.date(9999, 12, 31)
resolution Minimum resolution of date object (equal to datetime.timedelta(1) )
Instance methods returnsctime() String in format Sat Dec 28 11:48:24 2013
isocalendar() 3-Tuple containing ISO year, week number, and day
isoformat() String in format YYYY-MM-DD
isoweekday() Integer day of the week (Monday =1 ...Sunday = 7
replace(fd=newvalue) datetime.date with specifed felds replaced
strftime(fmt) String with date formatted as specifed
timetuple() Named 9-tuple with year, month, day, hour, minute, second, weekday, Julian date, DST fag (compatible with time.localtime() )
weekday() Integer day of the week (Monday =0 ...Sunday = 6
Static methods returnsfromtimestamp(ts) datetime.date from POSIX timestamp ts
today() Current date as datetime.date
datetime.datetime
Attributes valuesday Day as integer (1-31)
month Month as integer (1-12)
year Year as integer (4 digits)
hour Hour as integer (0-23)
minute Minute as integer (0-59)
second Second as integer (0-59)
microsecond Microsecond as integer (0-999999)
min Minimum date (equal to datetime.datetime(1, 1, 1, 0, 0) )
max Maximum date (equal to datetime.datetime (9999, 12, 31, 23, 59, 59, 999999) )
resolution Minimum resolution of date object (equal to datetime.timedelta(0, 0, 1) )
Instance Methods returnscombine(date, time) datetime.datetime from datetime.date date and datetime.time time
ctime() String in format Sat Dec 28 11:48:24 2013
date() datetime.date
isocalendar() 3-Tuple containing ISO year, week number, and day
isoformat() String in format YYYY-MM-DDTHH-:MM:SS[.mmmmmm]
Python II: Applied Python 2-25© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
OS Services rev. 3.0
Time-Related Modulesisoweekday() Integer day of the week (Monday =1 ...Sunday = 7
now() Current date and time as datetime.datetime
replace(fd=newvalue) datetime.datetime with specifed felds replaced
strftime(fmt) String with date formatted as specifed
time() datetime.time
timetuple() Named 9-tuple with year, month, day, hour, minute, second, weekday, Julian date, DST fag (compatible with time.localtime() )
today() Current date/time as datetime.datetime
weekday() Integer day of the week (Monday =0 ...Sunday = 6)
Static Methods returnsfromtimestamp(ts) datetime.datetime from POSIX timestamp ts
strptime(str) Datetime.datetime parsed from string str
datetime.time
Attributes valueshour Hour as integer (0-23)
minute Minute as integer (0-59)
second Second as integer (0-59)
microsecond Microsecond as integer (0-999999)
min Minimum date (equal to datetime.datetime(1, 1, 1, 0, 0) )
max Maximum date (equal to datetime.datetime (9999, 12, 31, 23, 59, 59, 999999) )
resolution Minimum resolution of date object (equal to datetime.timedelta(0, 0, 1) )
Instance Methods returnsreplace(fd=newvalue) datetime.datetime with specifed felds replaced
strftime(fmt) String with time formatted as specifed
time
Static Methods returnsasctime(tt) String from time tuple tt in format Sat Dec 28 11:48:24 2013
clock() Integer seconds since process started
ctime(ts) String from timestamp ts in format Sat Dec 28 11:48:24 2013
localtime(ts) Time tuple from timestamp ts
mktime(tt) Timestamp from time tuple
sleep(seconds) Sleep for seconds seconds
strftime(format, tt) Formatted string from format format and timetuple tt (default current time)
strpime(string, format) Time tuple from string parsed using format
2-26 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 OS Services
The calendar module
● Provides date calculations
● Outputs calendar in text, HTML, and other formats
● Utility methods for testing leap year, getting day of week, etc.
● Methods for weekday of frst day of month, number of days in month, and others
The calendar module provides a wide range of date-related functions. You can output a calendar in text or HTML formats; you can determine whether it's leap year, get the number of days in a given month and year, and many other useful methods.
Python II: Applied Python 2-27© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
OS Services rev. 3.0
Examplecalendar_ex.py
import calendar
def main(): print_text_calendar() print print_html_calendar()
def print_text_calendar(): tcal = calendar.TextCalendar() print tcal.formatmonth(2014,1)
def print_html_calendar(): hcal = calendar.HTMLCalendar() print hcal.formatmonth(2014,1)
if __name__ == '__main__': main()
calendar_ex.py January 2014Mo Tu We Th Fr Sa Su 1 2 3 4 5 6 7 8 9 10 11 1213 14 15 16 17 18 1920 21 22 23 24 25 2627 28 29 30 31
<table border="0" cellpadding="0" cellspacing="0" class="month"><tr><th colspan="7" class="month">January 2014</th></tr><tr><th class="mon">Mon</th><th class="tue">Tue</th><th class="wed">Wed</th><th class="thu">Thu</th><th class="fri">Fri</th><th class="sat">Sat</th><th class="sun">Sun</th></tr><tr><td class="noday"> </td><td class="noday"> </td><tdclass="wed">1</td><td class="thu">2</td><td class="fri">3</td><tdclass="sat">4</td><td class="sun">5</td></tr><tr><td class="mon">6</td><td class="tue">7</td><td class="wed">8</td><td class="thu">9</td><td class="fri">10</td><td class="sat">11</td><td class="sun">12</td></tr><tr><td class="mon">13</td><td class="tue">14</td><td class="wed">15</td><td class="thu">16</td><td class="fri">17</td><td class="sat">18</td><td class="sun">19</td></tr><tr><td class="mon">20</td><td class="tue">21</td><td class="wed">22</td><td class="thu">23</td><td class="fri">24</td><td class="sat">25</td><td class="sun">26</td></tr><tr><td class="mon">27</td><td class="tue">28</td><td class="wed">29</td><td class="thu">30</td><td class="fri">31</td><td class="noday"> </td><td class="noday"> </td></tr></table>
2-28 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 OS Services
Chapter 2 Exercises
Exercise 2-1 (path_files.py)
List each component of your PATH environment variable, together with the number of fles it contains. This is the set of fles you can execute from the command line without specifying a their path. Output should look something like this (for Windows, the paths will look different, but the idea is the same):
/usr/bin 2376/usr/local/bin 17/usr/local/sbin 1/usr/sbin 263
Hint: Use os.listdir to get the contents of a directory
Exercise 2-2 (oldest_file.py)
Write a script that, given a directory on the command line, prints out the date of the oldestfle in that directory (there may be more than one fle with this date).
Hint: make a list of (filename, timestamp) tuples and sort the list, reversed. Use datetime.fromtimestamp() to concert the timestamp and print that.
Exercise 2-3 (elapsed_time.py)
Write a script that accepts a date on the command line in the format YYYY-MM-DD, andprints out the elapsed time since that date in years and days.
Exercise 2-4 (date_diff.py)
Write a script that accepts two dates on the command line, as above, and prints out the time between the dates.
Exercise 2-5 (date_math.py)
Write a script that accepts a date and a positive or negative integer on the command line, and prints out the date that many days before or after the specifed date.
Hint: create a timedelta with the specified number of days
Exercise 2-6 (pres_ages.py)
Write an interactive script that asks for a president's last name. For each president whose last name matches, print out their full name and age when they left offce. For the current president, print their current age.
Python II: Applied Python 2-29© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
OS Services rev. 3.0
2-30 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
Python II: Applied Python
Chapter 9
Database Access
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 Database Access
Chapter 9 Objectives
● Understand the Python DBAPI architecture
● Connect to a database
● Execute simple and parameterized queries
● Fetch single and multiple row results
● Get metadata about a query
● Execute non-query statements
● Start transactions and commit or rollback as needed
Python II: Applied Python 9-1© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
Database Access rev. 3.0
The DB API
● Several ways to access DBMSs from Python
● DB API is most popular
● DB API is sort of an "abstract class"
● Many modules for different DBMSs using DB ABI
● Hides actual DBMS implemenation
To make database programming simpler, Python has the DB API. This is an API to standardize working with databases. When a package is written to access a database, it is written to conform to the API, and thus programmers do not have to learn a new set of methods and functions.
9-2 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 Database Access
Available InterfacesThe following databases have been implemented using the Python DB API-2.0 interface.
Database Python package
IBM DB2 PyDB2
Firebird (and Interbase) KInterbasDB
Informix informixdb
Ingres ingmod
MySQL MySQLdb
Oracle DCOracle2 or
ODBC pyodbc
PostgreSQL pgdb
SAP DB (also known as "MaxDB") sapdbapi
Microsoft SQL Server pymssql
Sybase Sybase
SQLite sqlite3
There may be other interfaces to some of the listed DBMs as well.
Python II: Applied Python 9-3© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
Database Access rev. 3.0
Connecting to a Server
● Import appropriate library
● Use connect() to get a database object
● Specify host, database, username, password
To connect to a database server, import the package for the specifc database. Use the package's connect() method to get a database object, specifying the host, initial database, username, and password. If the username and password are not needed, use None.
Some database modules have nonstandard parameters to the connect() method.
When fnished with the connection, call the close() method on the connection object.
Most database modules support the context manager (with statement), and will automatically close the database when the with block is exited. Check the documentation to see how this is implemented for a specifc database.
Example
import sqlite3
slconn = sqlite3.connect('web_content')
import MySQLdb
myconn = MySQLdb.connect (host = "myserver1", user = "adeveloper", passwd = "s3cr3t", db = "web_content")
Argument names for the connect() method may not be consistent. For instance, MySQL supports the above parameter names, while pymssql does not.
When used with the with statement, MySQLdb.connect() returns acursor object rather than a connection object. A connection is implicitly created (and closed) behind the scenes.
9-4 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 Database Access
Python package
connect() examples
MySQLdb MySQLdb.connect ( host="localhost", user="adeveloper", passwd="$3cr3t", db="testdb",)
sqlite3 sqlite3.connect('testdb')sqlite3.connect(':memory:')
pymssql pymssql.connect ( host="localhost", user="adeveloper", passwd="$3cr3t", db="testdb",)
pymssql.connect ( dsn="DSN",)
pyodbc pyodbc.connect(''' DRIVER={SQL Server}; SERVER=localhost; DATABASE=testdb; UID=adeveloper; PWD=$3cr3t
''')
pyodbc.connect('DSN=testdsn;PWD=$3cr3t')
pgdb pgdb.connect ( host="localhost", user="adeveloper", passwd="$3cr3t", db="testdb",)pgdb.connect ('host:database:user:password:options:tty')felds can be empty, non-trailing colons must be kept
psychopg psycopg2.connect (''' host='localhost' user='adeveloper' password='$3cr3t' dbname='testdb' ''') note connect() has one parameter: a string, not multiple parameters
cx_oracle Ip = 'localhost'port = 1521SID = 'YOURSIDHERE'dsn_tns = cx_Oracle.makedsn(ip, port, SID)
db = cx_Oracle.connect('adeveloper', '$3cr3t', dsn_tns)
Python II: Applied Python 9-5© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
Database Access rev. 3.0
Creating a Cursor
● Cursor is object that can execute SQL statements
● Several kinds of cursors usually available
● Other cursors leave data on server, etc.
Once you have a database object, you can create one or more cursors. A cursor is an object that can execute SQL code and fetch results.
The default cursor for most packages returns each row as a tuple of values. There are different types of cursors that can return data in different formats, or that control whether data is stored on the client or the server.
Example:myconn = MySQLdb.connect (host="myserver1",user="adeveloper",
passwd="s3cr3t", db="web_content")
mycursor = myconn.cursor()
9-6 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 Database Access
Executing a Statement
● Executing cursor sends SQL to server
● Data not returned until asked for
● Returns number of lines in result set for querys
● Returns lines affected for other statements
Once you have a cursor, you can use it to perform queries, or to execute arbitrary SQL statements via the execute() method. The frst argument to execute() is a string containingthe SQL statement to run.
Examplecursor.execute("select hostname,ostype,user from hostinfo")cursor.execute('insert into hostinfo values ("foo",5,"2.6","arch","net",2055,3072,"bob",0)')
Python II: Applied Python 9-7© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
Database Access rev. 3.0
Fetching Data
● Use a fetch methods from the cursor object
● Syntaxrec = cursor.fetchone()
recs = cursor.fetchall()
recs = cursor.fetchmany()
Cursors provide three methods for returning query results. fetchone() returns the next available row from the query results; fetchall() returns a tuple of all rows, and fetchmany(n) returns up to n rows. The latter is useful when the query returns a very large number of rows.
Examplecursor.execute("select color, quest from knights where name = 'Robin'")(color,quest) = cursor.fetchone()
cursor.execute("select color, quest from knights")rows = cursor.fetchall()
cursor.execute("select * from huge_table)while True: rows = cursor.fetchmany(1000) if rows == []: break for row in rows: # process row
9-8 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 Database AccessExampledb_sqlite_basics
import sqlite3
with sqlite3.connect('../DATA/PYTHON2') as s3conn:
s3cursor = s3conn.cursor()
# select first name, last name from all presidents s3cursor.execute(''' select lname, fname from presidents ''')
print "Sqlite3 does not provide a row count\n" for row in s3cursor.fetchmany(10): print ' '.join(row) # note -- Sqlite3 does not return the number of rows print
party_query = ''' select lname, fname from presidents where party = ? '''
for party in 'Federalist', 'Whig': s3cursor.execute(party_query, (party,)) print s3cursor.fetchall() print
db_sqlite_basics.pySqlite3 does not provide a row count
Washington GeorgeAdams John...Tyler John
[(u'Adams', u'John')]
[(u'Harrison', u'William Henry'), (u'Tyler', u'John'), (u'Taylor', u'Zachary'), (u'Fillmore', u'Millard')]
See db_mysql_basics.py and db_postgres_basics.py for examples using those modules.
Python II: Applied Python 9-9© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
Database Access rev. 3.0
Making a generator for large queries
● CURSOR.fetchall() fetches all data
● Memory/time issues for large datasets
● CURSOR.fetchmany() effcient
Suppose you have a query that returns 5 million records Using the fetchall() method, your cursor has to wait for all 5 million records, and transfer them over the network from the server machine to the client machine; even if the script is running on the server, that's still a lot of data.
You could also call fetchone() in a loop, but you lose effciency fetching one at a time across the network.
A good solution is to write a generator function to grab chunks of the result set using fetchmany().
Exampledb_sqlite_fetchiter
import sqlite3
def fetch_iter(cursor,chunk_size): """ Fetch chunk_size records, but parcel them out one at a time """ while True: rows = cursor.fetchmany(chunk_size) if not rows: # no more recs break for row in rows: yield row
with sqlite3.connect('../DATA/PYTHON2') as s3conn: s3cursor = s3conn.cursor()
s3cursor.execute('select fname, lname from presidents') for row in fetch_iter(s3cursor, 10): print ' '.join(row)
db_sqlite_iterfetch.pyGeorge WashingtonJohn AdamsThomas Jefferson...William Jefferson 'Bill' ClintonGeorge Walker BushBarack Hussein Obama
9-10 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 Database Access
Parameterized Statements
● More effcient updates
● Use cursor.execute() or cursor.executemany()
● Use placeholders in query
● Pass iterable of
For effciency, you can iterate over of sequence of datasets when performing a non-query SQL statement. The execute() method takes a query, plus an iterable of values to fll in the placeholders. The database manager will only parse the query once, then reuse it for subsequent calls to execute().
Different database modules use different placeholders. To see what kind of placeholder a module uses, check MODULE.paramstyle. Types include 'format', meaning '%s', and 'qmark', meaning '?'.
The executemany() method takes a query, and an iterable of iterables. It will call execute() once for each nested iterable.
Examplesingledata = ("Smith","John","green"),
multidata = [ ("Smith","John","green"), ("Douglas","Sam","pink"), ("Robinson","Alberta","blue"),]
query = "insert into people (lname,fname,color) values (%s,%s,%s)"
rows_added = cursor.execute(query, singledata)rows_added = cursor.executemany(query, multidata)
Python package Placeholder for parameters1
MySQLdb %s
cx_oracle :param_name
pyodbc ?
pgdb %s
pymssql %d int %s str etc.
Psychopg %s or %(param_name)s
sqlite3 ?
1With the exception of pymssql the same placeholder is used for all column types
Python II: Applied Python 9-11© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
Database Access rev. 3.0
Dictionary Cursors
● Indexed by column name
● Not standardized in the DB API
The standard cursor provided by the DBAPI returns a tuple for each row. Most DB packages provide other kinds of cursors, including user-defned versions.
A very common cursor is a dictionary cursor, which returns a dictionary for each row, where the keys are the column names. Each package that provides a dictionary cursor hasits own way of providing the dictionary cursor, although they all work the same way.
For the packages that don't have a dictionary cursor, you can make a generator function that will emulate one.
9-12 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 Database AccessExampledb_sqlite_extras.py
import sqlite3
s3conn = sqlite3.connect('../DATA/PYTHON2')# uncomment to make _all_ cursors dictionary cursors# conn.row_factory = sqlite3.Row
NAME_QUERY = ''' select lname, fname from presidents where num < 5'''
cur = s3conn.cursor()
# select first name, last name from all presidentscur.execute(NAME_QUERY)
for row in cur.fetchall(): print rowprint '-' * 50
dcur = s3conn.cursor()
# make _this_ cursor a dictionary cursordcur.row_factory = sqlite3.Row
# select first name, last name from all presidentsdcur.execute(NAME_QUERY)
for row in dcur.fetchall(): print row['lname'], row['lname']
print '-' * 50
# make cursors return str rather than unicodes3conn.text_factory = strtcur = s3conn.cursor()tcur.execute(NAME_QUERY)for row in tcur.fetchall(): print row
db_sqlite_extras.py(u'Washington', u'George')...(u'Madison', u'James')--------------------------------------------------Washington Washington...Madison Madison--------------------------------------------------('Washington', 'George')...('Madison', 'James')
Python II: Applied Python 9-13© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
Database Access rev. 3.0
Python package
How to get a dictionary cursor
MySQLdb import MySQLdb.cursorsconn = MySQLdb.connect(..., cursorclass = MySQLdb.cursors.DictCursor)dcur = conn.cursor() all cursors will be dict cursorsordcur = conn.cursor( MySQLdb.cursors.DictCursor)only this cursor will be a dict cursor
cx_oracle Not available directly, but can be emulated using columns from CUR.description
pyodbc Not available directly, but can be emulated using columns from CUR.description
pgdb Not available directly, but can be emulated using columns from CUR.description
pymssql conn = pymssql.connect (..., as_dict=True)
dcur = conn.cursor()
psychopg import psycopg2. extras
dcur = conn.cursor(cursor_factory=psycopg.extras.DictCursor)
sqlite3 conn = sqlite3.connect (..., row_factory=sqlite3.Row)
dcur = conn.cursor()
or
conn.row_factory = sqlite3.Row
dcur = conn.cursor()
9-14 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 Database Access
Metadata
● cursor.description returns tuple of tuples
● Fields
name
type_code
display_size
internal_size
precision
scale
null_ok
Once a query has been executed, the cursor's description() method returns information about the query as a tuple of tuples.
There is one tuple for each column in the query; each tuple contains a tuple of 7 values describing the column.
For instance, to get the names of the columns, you could say:
names = [ d[0] for d in cursor.description ]For non-query statements, cursor.description returns None.
The names are based on the query, and not necessarily on the 'official' names in the database itself.
Python II: Applied Python 9-15© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
Database Access rev. 3.0
Exampledb_sqlite_emulate_dict_cursor.pyNote: Most database modules, including MySQLdb, have a dictionary cursor built in -- this is just for an example you could use with any DBAPI module that does not have this capability. This uses the metadata from the cursor to get the column names, and forms a dictionary by zipping the column names with the column values. Another approach would be to use a named tuple.
import sqlite3
s3conn = sqlite3.connect('../DATA/PYTHON2')
c = s3conn.cursor()
def row_as_dict(cursor): '''Generate rows as dictionaries''' column_names = [desc[0] for desc in cursor.description] for row in cursor.fetchall(): row_dict = dict(zip(column_names, row)) yield row_dict
# select first name, last name from all presidentsnum_recs = c.execute(''' select lname, fname from presidents''')
for row in row_as_dict(c): print row['fname'], row['lname']
db_sqlite_emulate_dict_cursor.pyGeorge Washington John Adams Thomas Jefferson ...William Jefferson 'Bill' Clinton George Walker Bush Barack Hussein Obama
See db_sqlite_named_tuple_cursor.py for a similar example that creates named tuples rather than dictionaries for each row.
9-16 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 Database Access
Transactions
● Transactions allow safer control of updates
● commit() to save transactions
● rollback() to discard
● Default is autocommit off
● Add autocommit=True to connect() method to turn on
To be certain of data integrity, you may want to make batch changes to your database andcommit the changes only when successful. By default, you must explicitly save your data using connection.commit().
To save all transactions for a database since last commit or rollback, use connection.commit(). To discard transactions, use connection.rollback().
You can also turn on autocommit, which calls commit() after every statement that modifes the database.
Exampletry: for info in list_of_tuples: cursor.execute(query,info)except SQLError: dbconn.rollback()else: dbconn.commit()
MySQLdb only supports transaction processing for tables using the InnoDB engine
Python II: Applied Python 9-17© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
Database Access rev. 3.0
Object-relational Mappers
● No SQL required
● Maps a class to a table
● All DB work is done by manipulating objects
● Most popular Python ORMs
● SQLAlchemy
● Django (which is a complete web framework)
An Object-relation mapper is a module or framework that creates a level of abstraction above the actual database tables and SQL queries. As the name implies, a Python class (object) is mapped to the actual table.
Instead of querying the database, you call a search method on an object representing a table. To add a row to the table, you create a new instance of the table class, populate it, and call a method like save(). You can create a large, complex database system, complete with foreign keys, composite indices, and all the other attributes near and dear to a DBA, without writing the frst line of SQL.
You can use Python ORMs in two ways.
One way is to design the database with the ORM. To do this, you create a class for each table in the database, specifying the columns with predefned classes from the ORM. Then you run an ORM command which executes the queries needed to build the database. If you need to make changes, you update the class defnitions, and run an ORM command to synchonize the actual DBMS to your classes.
The second way is to map tables to an existing database. You create the classes to match the schemas that have already been defned in the database.
The two most popular Python ORMs are SQLAlchemy which is a standalone ORM, and Django, which is a very large and complete Web development framework, which provides an ORM as a subpackage. SQLAlchemy is the most fully developed package, and is the ORM used by the Pyramid (formerly Pylons) Web development framework.
9-18 Python II: Applied Python© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.
rev. 3.0 Database Access
Chapter 9 Exercises
Exercise 9-1 (presidentmysql.py, presidentsqlite.py)
For this exercise, you can use the SQLite3 database provided, or use your own DBMS. The mkpres.sql script is generic and should work with any DBMS to create and populate the presidents table. The SQLite3 database is named PYTHON2 and is created in the rootfolder of the student fles.
The data has the following layout
+------------+--------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +------------+--------------+------+-----+---------+-------+ | num | int(11) | YES | | NULL | | | lname | varchar(32) | YES | | NULL | | | fname | varchar(64) | YES | | NULL | | | dstart | date | YES | | NULL | | | dend | date | YES | | NULL | | | birthplace | varchar(128) | YES | | NULL | | | dbirth | date | YES | | NULL | | | ddeath | date | YES | | NULL | | | party | varchar(32) | YES | | NULL | | +------------+--------------+------+-----+---------+-------+
Refactor your president module to get its data from this table, rather than from a fle. Test with your existing scripts.
Exercise 9-2 (add_pres_mysql.py, add_pres_sqlite.py)
Assuming your favorite candidate wins (or has won) the 2016 election, add them to the table.
Hint: If you don't have access to the Internet, make up their information!
SQL syntax for adding a record is
INSERT INTO table ("COL1-NAME",..) VALUES ("VALUE1",...)
Python II: Applied Python 9-19© 2014 CJ Associates
EVALUATION COPY
Unauthorized reproduction or distribution is prohibited.