Spatially-Enabled DevelopmentUtilizing open source tools and resources for GIS data manipulation and analysis

What is GIS? Geographic Information Systems/Science

Spatial Information – visualize the world around us

“five parts of a GIS” (Source: gisandscience.com)

Spatially Enabled Decision Support- the Urban GIS mojo

Why GIS? GIS empowers us to understand the world around us through

data visualization and analysis.

Why GIS? Supports Planning for,

among other things, emergency disaster response

Image here shows the path of the recent EF-4 tornado that hit near Rochelle, IL on April 9, 2015

The image here is a false-color composite (7-5-3), Landsat 8 image with 15m resolution

Additional Storm Data

Source: http://earthobservatory.nasa.gov/NaturalHazards/view.php?id=85689

Why GIS? Pattern Recognition

Credit: Thomas Pingel - Northern Illinois Universityhttp://popular-archaeology.com/issue/spring-2015/article/archaeologist-discovers-mysterious-ancient-maya-citadel

Why GIS? Pattern Recognition

Why GIS? Discover Relationships

Source: http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#//00080000001q000000

Source: https://en.wikipedia.org/wiki/Spatial_relation#/media/File:TopologicSpatialRelarions2.png

What can GIS do for You? Representation of features in geographic space

Augment your feature attribution with GIS data

Extend your data, don't recreate it

Add topology and build relationships between your features

Identify features within 5ft

Features have directionality – water flow modeling, e.g.

GIS Data Repositories

Select GIS Data Repositories City of Chicago Data Portal

OpenStreetMap

NCDC Climate Data

USGS Landcover

Census Data – TIGER Products

USGS Satellite Derived Data

WorldBank

United Nations Environment Program (UNEP)

ASTER Global DEM – NASA

OpenTopography

OpenFlights

Gridded Population of the World (GPW) v3

See More: http://urbangis.com/repositories/

City of Chicago Data Portal https://data.cityofchicago.org

200+ datasets about City departments, services, facilities, performance and statistics

Transportation Buildings Administration Public Safety Community and Economic Development

Access via web or API Socrata Open Data API (SODA) http://dev.socrata.com/

City of Chicago Data Portal##Quick and easy way to visualize the crimes data set using open source techniques##Have more ideas? Tell me about them at @ChicagoCDO

##All you need to do is install R at http://cran.r-project.org/ and run this file

install.packages("googleVis")

library(googleVis)

crimes_stage <- read.csv("http://data.cityofchicago.org/views/x2n5-8w5q/rows.csv")

attach(crimes_stage)

##remove first row with column names

crimes <- crimes_stage[-c(1),]

##create frequency table from crime

freq <- table(PRIMARY.DESCRIPTION)

##change frequency table to data frame

freq.data <- as.data.frame(freq)

pie <- gvisPieChart(freq.data,options=list(height=1200,width=1200))

## Appreciate visual output

plot(pie)

GitHub: Chicago/gist:1165275

City of Chicago Data Portal

Data: freq.data • Chart ID: PieChartID26136638e480 • googleVis-0.5.8R version 3.2.0 (2015-04-16) • Google Terms of Use • Documentation and Data Policy

ARSON

ASSAULT

BATTERY

BURGLARY

CRIM SEXUAL ASSAULT

CRIMINAL DAMAGE

CRIMINAL TRESPASS

DECEPTIVE PRACTICE

GAMBLING

HOMICIDE

INTERFERENCE WITH

PUBLIC OFFICER

MOTOR VEHICLE THEFT

NARCOTICS

OFFENSE INVOLVING

CHILDREN

OTHER OFFENSE

PROSTITUTION

PUBLIC PEACE VIOLATION

ROBBERY

SEX OFFENSE

THEFT

WEAPONS VIOLATION

Other

6.4%

18.6%

5.1%

10.5%

5.2%

22.2%

3.6%

6.4%

9.5%

3.8%

City of Chicago Data Portal

Occurrences of Arson 2001 to Present

OpenStreetMap http://www.openstreetmap.org

Collaborative GIS, Community driven mapping

License: Open Data Commons Open Database License (ODbL)

Data Format: OSM XML

<?xml version="1.0" encoding="UTF-8"?><osm version="0.6" generator="CGImap 0.4.0 (23692 thorn-02.openstreetmap.org)" copyright="OpenStreetMap and contributors" attribution="http://www.openstreetmap.org/copyright"

license="http://opendatacommons.org/licenses/odbl/1-0/"><bounds minlat="41.9345500" minlon="-88.7605200" maxlat="41.9362700" maxlon="-88.7563100"/><node id="234234028" visible="true" version="2" changeset="2419810" timestamp="2009-09-08T22:37:37Z" user="woodpeck_fixbot" uid="147510" lat="41.9342840" lon="-88.7544640"/><node id="234251673" visible="true" version="2" changeset="3566456" timestamp="2010-01-07T22:37:32Z" user="woodpeck_fixbot" uid="147510" lat="41.9346200" lon="-88.7564080"/><node id="234283141" visible="true" version="2" changeset="2419810" timestamp="2009-09-08T22:51:58Z" user="woodpeck_fixbot" uid="147510" lat="41.9355060" lon="-88.7594300"/><node id="339220187" visible="true" version="1" changeset="857236" timestamp="2009-01-30T21:21:45Z" user="Ru55Ht" uid="3859" lat="41.9394849" lon="-88.7511986"/><node id="339220237" visible="true" version="1" changeset="857236" timestamp="2009-01-30T21:21:46Z" user="Ru55Ht" uid="3859" lat="41.9323402" lon="-88.7600563"/><node id="234224085" visible="true" version="2" changeset="2419810" timestamp="2009-09-08T22:34:43Z" user="woodpeck_fixbot" uid="147510" lat="41.9338060" lon="-88.7524030"/><node id="234228505" visible="true" version="2" changeset="2419810" timestamp="2009-09-08T22:36:14Z" user="woodpeck_fixbot" uid="147510" lat="41.9355020" lon="-88.7549950"/><node id="234228507" visible="true" version="2" changeset="2419810" timestamp="2009-09-08T22:36:14Z" user="woodpeck_fixbot" uid="147510" lat="41.9357720" lon="-88.7567720"/><node id="234228509" visible="true" version="2" changeset="2419810" timestamp="2009-09-08T22:36:14Z" user="woodpeck_fixbot" uid="147510" lat="41.9358100" lon="-88.7573010"/><node id="234228511" visible="true" version="2" changeset="2419810" timestamp="2009-09-08T22:36:14Z" user="woodpeck_fixbot" uid="147510" lat="41.9358310" lon="-88.7578040"/><node id="234228516" visible="true" version="2" changeset="2419810" timestamp="2009-09-08T22:36:14Z" user="woodpeck_fixbot" uid="147510" lat="41.9358330" lon="-88.7579830"/><node id="234228519" visible="true" version="2" changeset="2419810" timestamp="2009-09-08T22:36:14Z" user="woodpeck_fixbot" uid="147510" lat="41.9358170" lon="-88.7583470"/><node id="234228522" visible="true" version="2" changeset="2419810" timestamp="2009-09-08T22:36:14Z" user="woodpeck_fixbot" uid="147510" lat="41.9358050" lon="-88.7585070"/><node id="234228525" visible="true" version="2" changeset="2419810" timestamp="2009-09-08T22:36:14Z" user="woodpeck_fixbot" uid="147510" lat="41.9357780" lon="-88.7587070"/><node id="234228528" visible="true" version="2" changeset="2419810" timestamp="2009-09-08T22:36:14Z" user="woodpeck_fixbot" uid="147510" lat="41.9357620" lon="-88.7587820"/><node id="234228532" visible="true" version="2" changeset="2419810" timestamp="2009-09-08T22:36:14Z" user="woodpeck_fixbot" uid="147510" lat="41.9357410" lon="-88.7588550"/><node id="234228536" visible="true" version="2" changeset="2419810" timestamp="2009-09-08T22:36:14Z" user="woodpeck_fixbot" uid="147510" lat="41.9357160" lon="-88.7589170"/><node id="234228540" visible="true" version="2" changeset="2419810" timestamp="2009-09-08T22:36:14Z" user="woodpeck_fixbot" uid="147510" lat="41.9356450" lon="-88.7590470"/><node id="234228543" visible="true" version="2" changeset="2419810" timestamp="2009-09-08T22:36:14Z" user="woodpeck_fixbot" uid="147510" lat="41.9356160" lon="-88.7590920"/>

NCDC Climate Data http://www.ncdc.noaa.gov

Global weather and climate data

Data format: ASCII

Example: Integrated Surface Database (ISD) consists of global hourly and synoptic observations in ASCII

format

35k stations worldwide with some stations providing data since 1901

USGS Landcover http://landcover.usgs.gov/usgslandcover.php

A variety of derived products including: 1 km MODIS-based Maximum Green Vegetation Fraction

0.5 km MODIS-based Global Land Cover Climatology

Ecoregions (multiple)

Global Land Cover (multiple)

Fire Mapping

Data Format: Shapefile, GeoTIFF, others

Census Data – TIGER Products http://www.census.gov/geo/maps-data/data/tiger.html

USGS – Other Products http://earthexplorer.usgs.gov

Data sets include: Aerial Imagery

AVHRR

LIDAR

Land Cover

Landsat Multispectral Imagery

Radar

Data formats: varies, primarily raster based products

Tools for Data Manipulation and Analysis

Python for GIS Spatial Data Interoperability

Pyshp – https://pypi.python.org/pypi/pyshp/ Shapely - https://pypi.python.org/pypi/Shapely GDAL - https://pypi.python.org/pypi/GDAL/ and http://gdal.org/python/ GeoJSON - https://pypi.python.org/pypi/geojson

Spatial Operations/Manipulation PyProj - https://pypi.python.org/pypi/pyproj/ Geopy - https://pypi.python.org/pypi/geopy ArcPy - https://arcpy.wordpress.com/ (Recipes) Numpy - http://www.numpy.org/ PySAL - https://pypi.python.org/pypi/PySAL NetworkX - http://networkx.github.io/

App Dev ScyPy - http://www.scipy.org/ GeoDjango - http://geodjango.org/

See More (with sample code): http://urbangis.com/python/

Example: ArcPy and PyProjimport arcpyimport pyprojimport psycopg2

from arcpy import env

env.workspace = "C:\Users\Owner\AppData\Roaming\ESRI\Desktop10.2\ArcCatalog\xxxx.sde"

# Projection string for NAD83_HARN_Illinois_East_ftUS (EPSG 3435)pnyc = pyproj.Proj('+proj=tmerc +lat_0=36.66666666666666 +lon_0=-88.33333333333333 +k=0.9999749999999999 +x_0=300000.0000000001 +y_0=0 +ellps=GRS80 +datum=NAD83 +to_meter=0.3048006096012192 +no_defs ', preserve_units=True)

conn = psycopg2.connect("dbname=xxxx user=xxxx host=xxxx")cur = conn.cursor()

cur.execute("SELECT globalid, x_coord, y_coord FROM features")#cur.execute("SELECT globalid, ST_X(shape), ST_Y(SHAPE) FROM features")

foo = cur.fetchall()

for row in foo:newCoord = pnyc(row[1], row[2], inverse=True)cur.execute("UPDATE features SET longitude = {0} WHERE globalid = '{1}'".format(newCoord[0], row[0]))cur.execute("UPDATE features SET latitude = {0} WHERE globalid = '{1}'".format(newCoord[1], row[0]))

Example: Scripting Updates with Arcpywith arcpy.da.UpdateCursor("Soil_Landuse_Intersection",("NIPC_CODE","HydrologyG","CN")) as cur:

for row in cur:

whereStmt = """"NIPC_CODE"={0} AND "HYDROGRP"='{1}'""".format(row[0],row[1])

with arcpy.da.SearchCursor("NIPC_sl","CN",whereStmt) as cur1:

for row1 in cur1:

newValue = row1[0]

row[2] = newValue

cur.updateRow(row)

with arcpy.da.UpdateCursor("Melvina_Subbasings_20141219",("FID","AREA")) as cur:

for row in cur:

weightedCN = 0

whereStatement = """ "FID_Melvin"={0} """.format(row[0])

with arcpy.da.SearchCursor("SoilLanduse_clipped",("AREA_NEW","CN"),whereStatement) as cur1:

for row1 in cur1:

weightedCNValue = row1[1]*(row1[0]/row[1])

weightedCN += weightedCNValue

print row[0], weightedCN

Example: Raster processing with Python and GDALdef FileExtractRaster(filename, bbox, verbose=False):

'''Obtain a list of specified elevation chunks from an SRTM raster tile. Uses the gdal module. Takes the path to the tile, with the tile name. Grabs the specified block (with bbox) of raster data from that tile. bbox is a list of dictionaries representing the lower left and upper rightblock corners. Returns a 2D numpy array. I'd like to use gdal.ReadAsArray() but it didn't work.'''

import osgeo.gdalimport gdalconstimport structimport numpyimport math

infile = SRTMPATH + filenameif verbose:

print infile

dataset = osgeo.gdal.Open( infile, gdalconst.GA_ReadOnly )if verbose:

print 'Driver: ', dataset.GetDriver().ShortName,'/', \dataset.GetDriver().LongName

print 'Size is ', dataset.RasterXSize,'x',dataset.RasterYSize, \'x',dataset.RasterCount

print 'Projection is ', dataset.GetProjection()

# Grab the upper left coords and pixel size of the raster gridgeotransform = dataset.GetGeoTransform()if verbose:

if not geotransform is None:print 'Origin = (',geotransform[0], ',',geotransform[3],')'print 'Pixel Size = (',geotransform[1], ',',geotransform[5],')'

Credit: Ashton Shortridge, Michigan State University

Example: Raster processing with Python and GDAL

# Get a pointer to the Raster elevationsband = dataset.GetRasterBand(1)datType = 'GDT_' + osgeo.gdal.GetDataTypeName(band.DataType)

# Figure out the offsets to the coordinates of the upper left corner

xoff = int(abs(abs(geotransform[0]) - abs(float(bbox['llx']))) / geotransform[1])

yoff = int(abs(abs(geotransform[3]) - abs(float(bbox['ury']))) / geotransform[1])

# Figure out the box size in cells

xsize = (float(bbox['urx']) - float(bbox['llx'])) / geotransform[1]

ysize = (float(bbox['ury']) - float(bbox['lly'])) / geotransform[1]

xsize = int(math.ceil(xsize))

ysize = int(math.ceil(ysize))

# Extract the elevation block

rastBlock = band.ReadRaster(xoff, yoff, xsize, ysize, xsize, ysize, band.DataType)

# Convert from binary to ints or floats and append to the coord structure

if band.DataType == 1: numType='B' # unsigned char (this is a guess): MODIS

if band.DataType in (2, 3): numType='H' # unsigned short: SRTM

if band.DataType == 6: numType='f' # float : NED (really?)

rastValues = numpy.array(struct.unpack(numType * (xsize * ysize), rastBlock)).reshape(ysize, xsize)

return rastValues

Credit: Ashton Shortridge, Michigan State University

Example: Raster processing with Python and GDAL# Extract the elevation block

rastBlock = band.ReadRaster(xoff, yoff, xsize, ysize, xsize, ysize, band.DataType)

# Convert from binary to ints or floats and append to the coord structure

if band.DataType == 1: numType='B' # unsigned char (this is a guess): MODIS

if band.DataType in (2, 3): numType='H' # unsigned short: SRTM

if band.DataType == 6: numType='f' # float : NED (really?)

rastValues = numpy.array(struct.unpack(numType * (xsize * ysize), rastBlock)).reshape(ysize, xsize)

return rastValues

Credit: Ashton Shortridge, Michigan State University

Example: Raster processing in the DBimport psycopg2

import numpy

conn = psycopg2.connect("dbname=xxxx user=xxxx password=xxxx")

def processSamples(coordList, winx, winy, source='file', verbose=False, table='srtm_global_100'):

'''Process a list of coordinates. Identify the window around each one

and hand that off to functions that actually do the retrieval, either

from 'file' or 'db'.'''

for c in coordList:

bbox = createBbox(c['lon'], c['lat'], winx, winy)

cur = conn.cursor()

query = "SELECT ST_Neighborhood(rast, ST_GeomFromText('POINT({0} {1})', 4326), {2},{3}) FROM {4}

WHERE ST_Intersects(rast, ST_GeomFromText('POINT({0} {1})', 4326))".format(c['lon'], c['lat'],

(winx-1)/2, (winy-1)/2, table)

cur.execute(query)

for record in cur:

foo = numpy.asarray(record)

Additional Resources

WEB RESOURCES

StackExchangehttp://gis.stackexchange.com/

ArcGIS Developer Resourceshttps://developers.arcgis.com/en/

Open Source Geospatial Foundationhttp://www.osgeo.org/

OpenLayershttp://openlayers.org/

Public Geospatial Data Projecthttp://wiki.osgeo.org/wiki/Public_Geospatial_Data_Project

GeoToolshttp://www.geotools.org/

Geospatial Analysis Guide – a GIS Resourcehttp://www.spatialanalysisonline.com/index.html

BOOKS/MEDIA

Spatial Data Analysis in R

Applied Spatial Data Analysis with R

Python Scripting for ArcGIS

Open Source GIS

ArcGIS Web Development

Python Geospatial Development

Shaun Langleysl@urbangis.com, 312.666.7581

Thank You

A copy of this presentation is available at http://urbangis.com/presentations/