geonode integration with gis and data processing...
TRANSCRIPT
GeoNode Integration with GIS and Data Processing workflows
Salvador Bayarri sbayarrigmailcom
World Bank Consultant
Contents (1)
bull GeoServer components
bull Direct access to GeoServer
ndash Styling with SLD
ndash Adding data from external databases
bull Direct access to GeoNetwork
ndash Searching displaying and editing metadata catalog
bull Direct access to the Geonode spatial database
ndash Examples of DB analysis processes
ndash Projecting data in DB
Contents (2)
bull Access from desktop GIS ndash Connecting and displaying map and feature services
ndash Editing the DB
ndash Projecting data in GIS
ndash Access to metadata catalogs
ndash Exporting styles through SLD
ndash Georeferencing data
ndash Uploading mobile data
ndash Verifying and correcting topology
GeoNode stack
Application layer
Service layer
Data layer
GeoNetwork Web interface
GeoServer Web interface
Direct access to GeoServer bull GeoServer is the component that provides WMS WFS and WCS services
bull It can be used to upload and style data using its own Web front-end
bull However your GeoNode user wonrsquot be recognized by GeoServer you need to use a GeoServer user normally with administration privileges
Connect with your Web browser httpgeonodeURLgeoserver
GeoServer styling with SLD
bull It is easier to work with SLD styles directly within GeoServer
bull Click on the lsquoStylesrsquo link in the left-side menu and select the style you want to edit or add a new one
GeoServer styling with SLD
In the style editor you can bull Change the name of the style
bull Copy the SLD from an existing style
bull Change the code of the SLD
bull Upload an SLD file
bull Validate the SLD to detect errors
GeoServer new styles
bull If you create a new style you need to associate it with a layer
bull To do so select the desired layer in the Layers page
GeoServer new styles
bull Select the lsquoPublishingrsquo tab in the layer properties
In the WMS Settings section you can choose the default style and add more style choices
GeoServer new styles
bull The styling changes will show up in GeoNode (you may need to refresh the page)
GeoServer styling with SLD
bull Exercise
ndash Modify and add new styles for some of your vector layers
ndash Modify the style for the elevation layers
ndash See the changes in GeoNode
GeoServer adding data
bull GeoNode only supports file-based data sources shapefiles and geoTIFF but GeoServer offers more options
bull Data sources are referred in GeoServer as stores
GeoServer vector data stores
bull The uploaded shape files are inserted in the geonode PostGIS database so they are all in the same store called ldquopostgres_importsrdquo
We can access any data from the PostGIS DB using the same store
GeoServer geoTIFF stores
bull The uploaded geoTIFF files are copied in the geonode data folder Each raster file has its own store
We can access any data from the PostGIS DB using the same store
GeoServer adding external data
bull We can add other data sources for GeoNode by creating GeoServer stores
More raster formats
GeoServer adding external data
bull GeoServer has extensions to create stores from other database vendors
bull More information in
httpdocsgeoserverorglatestenuserwebadmindatastoreshtml
GeoServer adding external data
IMPORTANT NOTE
If you add external data directly into GeoServer the GeoNode superuser must run the following shell command to synchronize these new layers to be visible in GeoNode
Unfortunately this updating does yet not work for cascaded WMS layers as we will see
geonode updatelayers
GeoServer adding external data
bull Exercise ndash Add a new store for the GTOPO30
elevation dataset named W200N40dem
ndash After the store is added click on Publish and accept the default
ndash Run the lsquoupdatelayersrsquo command and see if the layer appears in GeoNode
Direct access to GeoNetwork bull GeoNetwork is the component that provides metadata management
and CSW catalog search service
bull It can be used to search and display the GeoNode metadata
bull However do not use Geonetwork to edit the GeoNode metadata bull If you have administration privileges you can still use Geonetwork to create non-spatial
metadata and harvest metadata from other CSW servers
Connect with your Web browser httpgeonodeURLgeonetwork
Direct access to GeoNetwork
bull Exercise ndash Log in Geonetwork as user lsquoadminrsquo with password lsquoadminrsquo
ndash Go to the Administration New metadata menu
ndash Select the Vector data ISO19139 template
Direct access to GeoNetwork
bull Exercise ndash In the medata form type a title like lsquoOnline documentrsquo
ndash Scroll down and type a Web address as online resource
ndash Click on lsquoSave and Closersquo
Direct access to GeoNetwork
bull Exercise ndash In the medata preview click on lsquoPrivilegesrsquo
ndash Set access privileges for All and Submit
ndash Go back and search for your new lsquodatarsquo link in GeoNode
Direct access to the GeoNode DB
bull GeoNode uses a PostGIS database management system to store uploaded vector data as well as the metadata
bull To see the contents of this DB open the pgAdmin Postgres DB administration tool (this has been installed for the workshop)
Direct access to the GeoNode DB
bull Use the lsquoAdd Serverrsquo command to connect to the geonode DB
Use the password that is in the DATABASE_PASSWORD variable in this text file etcgeonodelocal_settingspy
Direct access to the GeoNode DB
Under DatabasesgeonodeSchemas publicTables we will see the tables corresponding to our vector data By right-clicking on a table we have access to some interesting tools
Direct access to the GeoNode DB
bull We can create reports like a data dictionary report
Direct access to the GeoNode DB
bull We can view the data records and filter them
Direct access to the GeoNode DB
bull We can execute SQL queries to select insert or delete records
We can apply operators to perform calculations like average count standard deviation summation etc
Direct access to the GeoNode DB
bull Exercise ndash Use simple SQL queries to extract columns from some layers
SELECT some_columns
FROM some_data_source
WHERE some_condition
Remember column names in UPPERCASE must use QUOTES ldquoLIKETHISrdquo
ndash Use Sum() and Count() operators with SELECT to aggregate results
For instance find out how many different polygons make each District and what is their total areas
Direct access to the GeoNode DB
There are also spatial operators that perform calculations on geometries
bull ST_GeometryType(geometry) returns the type of the geometry
bull ST_SRID(geometry) returns the spatial reference identifier number of the geometry
bull ST_X(point) returns the X ordinate
bull ST_Y(point) returns the Y ordinate
bull ST_Length(linestring) returns the length of the linestring
bull ST_StartPoint(geometry) returns the first coordinate as a point
bull ST_EndPoint(geometry) returns the last coordinate as a point
bull ST_NPoints(geometry) returns the number of coordinates in the linestring
bull ST_Area(geometry) returns the area of the polygons
bull ST_NRings(geometry) returns the number of rings (usually 1 more if there are holes)
bull ST_ExteriorRing(polygon) returns the outer ring as a linestring
bull ST_InteriorRingN(polygon integer) returns a specified interior ring as a linestring
bull ST_Perimeter(geometry) returns the length of all the rings
Direct access to the GeoNode DB
And operators that generate a text description of the geometries in different formats or vice versa
bull ST_AsText(geometry) returns WKT text
bull ST_AsEWKT(geometry) returns EWKT text
bull ST_GeomFromWKB(bytea) returns geometry
bull ST_AsBinary(geometry) returns WKB bytea
bull ST_AsEWKB(geometry) returns EWKB bytea
bull ST_GeomFromGML(text) returns geometry
bull ST_AsGML(geometry) returns GML text
bull ST_GeomFromKML(text) returns geometry
bull ST_AsKML(geometry) returns KML text
bull ST_AsGeoJSON(geometry) returns JSON text
bull ST_AsSVG(geometry) returns SVG text
Analysis with GeoNode DB We can use the DB to perform more complex spatial analysis operations
Many lsquoGIS analysisrsquo tasks are based on a combination of geometric operations between features (intersection distance) and aggregation
PostGIS provides the following geometric operators
bull ST_Contains(geometry A geometry B)
bull ST_Crosses(geometry A geometry B)
bull ST_Disjoint(geometry A geometry B)
bull ST_Distance(geometry A geometry B)
bull ST_DWithin(geometry A geometry B radius)
bull ST_Equals(geometry A geometry B)
bull ST_Intersects(geometry A geometry B)
bull ST_Overlaps(geometry A geometry B)
bull ST_Touches(geometry A geometry B)
bull ST_Within(geometry A geometry B)
Analysis with GeoNode DB spatial joins
The SQL JOIN operator allows us to add fields to a table by performing spatial tests
For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to
We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Contents (1)
bull GeoServer components
bull Direct access to GeoServer
ndash Styling with SLD
ndash Adding data from external databases
bull Direct access to GeoNetwork
ndash Searching displaying and editing metadata catalog
bull Direct access to the Geonode spatial database
ndash Examples of DB analysis processes
ndash Projecting data in DB
Contents (2)
bull Access from desktop GIS ndash Connecting and displaying map and feature services
ndash Editing the DB
ndash Projecting data in GIS
ndash Access to metadata catalogs
ndash Exporting styles through SLD
ndash Georeferencing data
ndash Uploading mobile data
ndash Verifying and correcting topology
GeoNode stack
Application layer
Service layer
Data layer
GeoNetwork Web interface
GeoServer Web interface
Direct access to GeoServer bull GeoServer is the component that provides WMS WFS and WCS services
bull It can be used to upload and style data using its own Web front-end
bull However your GeoNode user wonrsquot be recognized by GeoServer you need to use a GeoServer user normally with administration privileges
Connect with your Web browser httpgeonodeURLgeoserver
GeoServer styling with SLD
bull It is easier to work with SLD styles directly within GeoServer
bull Click on the lsquoStylesrsquo link in the left-side menu and select the style you want to edit or add a new one
GeoServer styling with SLD
In the style editor you can bull Change the name of the style
bull Copy the SLD from an existing style
bull Change the code of the SLD
bull Upload an SLD file
bull Validate the SLD to detect errors
GeoServer new styles
bull If you create a new style you need to associate it with a layer
bull To do so select the desired layer in the Layers page
GeoServer new styles
bull Select the lsquoPublishingrsquo tab in the layer properties
In the WMS Settings section you can choose the default style and add more style choices
GeoServer new styles
bull The styling changes will show up in GeoNode (you may need to refresh the page)
GeoServer styling with SLD
bull Exercise
ndash Modify and add new styles for some of your vector layers
ndash Modify the style for the elevation layers
ndash See the changes in GeoNode
GeoServer adding data
bull GeoNode only supports file-based data sources shapefiles and geoTIFF but GeoServer offers more options
bull Data sources are referred in GeoServer as stores
GeoServer vector data stores
bull The uploaded shape files are inserted in the geonode PostGIS database so they are all in the same store called ldquopostgres_importsrdquo
We can access any data from the PostGIS DB using the same store
GeoServer geoTIFF stores
bull The uploaded geoTIFF files are copied in the geonode data folder Each raster file has its own store
We can access any data from the PostGIS DB using the same store
GeoServer adding external data
bull We can add other data sources for GeoNode by creating GeoServer stores
More raster formats
GeoServer adding external data
bull GeoServer has extensions to create stores from other database vendors
bull More information in
httpdocsgeoserverorglatestenuserwebadmindatastoreshtml
GeoServer adding external data
IMPORTANT NOTE
If you add external data directly into GeoServer the GeoNode superuser must run the following shell command to synchronize these new layers to be visible in GeoNode
Unfortunately this updating does yet not work for cascaded WMS layers as we will see
geonode updatelayers
GeoServer adding external data
bull Exercise ndash Add a new store for the GTOPO30
elevation dataset named W200N40dem
ndash After the store is added click on Publish and accept the default
ndash Run the lsquoupdatelayersrsquo command and see if the layer appears in GeoNode
Direct access to GeoNetwork bull GeoNetwork is the component that provides metadata management
and CSW catalog search service
bull It can be used to search and display the GeoNode metadata
bull However do not use Geonetwork to edit the GeoNode metadata bull If you have administration privileges you can still use Geonetwork to create non-spatial
metadata and harvest metadata from other CSW servers
Connect with your Web browser httpgeonodeURLgeonetwork
Direct access to GeoNetwork
bull Exercise ndash Log in Geonetwork as user lsquoadminrsquo with password lsquoadminrsquo
ndash Go to the Administration New metadata menu
ndash Select the Vector data ISO19139 template
Direct access to GeoNetwork
bull Exercise ndash In the medata form type a title like lsquoOnline documentrsquo
ndash Scroll down and type a Web address as online resource
ndash Click on lsquoSave and Closersquo
Direct access to GeoNetwork
bull Exercise ndash In the medata preview click on lsquoPrivilegesrsquo
ndash Set access privileges for All and Submit
ndash Go back and search for your new lsquodatarsquo link in GeoNode
Direct access to the GeoNode DB
bull GeoNode uses a PostGIS database management system to store uploaded vector data as well as the metadata
bull To see the contents of this DB open the pgAdmin Postgres DB administration tool (this has been installed for the workshop)
Direct access to the GeoNode DB
bull Use the lsquoAdd Serverrsquo command to connect to the geonode DB
Use the password that is in the DATABASE_PASSWORD variable in this text file etcgeonodelocal_settingspy
Direct access to the GeoNode DB
Under DatabasesgeonodeSchemas publicTables we will see the tables corresponding to our vector data By right-clicking on a table we have access to some interesting tools
Direct access to the GeoNode DB
bull We can create reports like a data dictionary report
Direct access to the GeoNode DB
bull We can view the data records and filter them
Direct access to the GeoNode DB
bull We can execute SQL queries to select insert or delete records
We can apply operators to perform calculations like average count standard deviation summation etc
Direct access to the GeoNode DB
bull Exercise ndash Use simple SQL queries to extract columns from some layers
SELECT some_columns
FROM some_data_source
WHERE some_condition
Remember column names in UPPERCASE must use QUOTES ldquoLIKETHISrdquo
ndash Use Sum() and Count() operators with SELECT to aggregate results
For instance find out how many different polygons make each District and what is their total areas
Direct access to the GeoNode DB
There are also spatial operators that perform calculations on geometries
bull ST_GeometryType(geometry) returns the type of the geometry
bull ST_SRID(geometry) returns the spatial reference identifier number of the geometry
bull ST_X(point) returns the X ordinate
bull ST_Y(point) returns the Y ordinate
bull ST_Length(linestring) returns the length of the linestring
bull ST_StartPoint(geometry) returns the first coordinate as a point
bull ST_EndPoint(geometry) returns the last coordinate as a point
bull ST_NPoints(geometry) returns the number of coordinates in the linestring
bull ST_Area(geometry) returns the area of the polygons
bull ST_NRings(geometry) returns the number of rings (usually 1 more if there are holes)
bull ST_ExteriorRing(polygon) returns the outer ring as a linestring
bull ST_InteriorRingN(polygon integer) returns a specified interior ring as a linestring
bull ST_Perimeter(geometry) returns the length of all the rings
Direct access to the GeoNode DB
And operators that generate a text description of the geometries in different formats or vice versa
bull ST_AsText(geometry) returns WKT text
bull ST_AsEWKT(geometry) returns EWKT text
bull ST_GeomFromWKB(bytea) returns geometry
bull ST_AsBinary(geometry) returns WKB bytea
bull ST_AsEWKB(geometry) returns EWKB bytea
bull ST_GeomFromGML(text) returns geometry
bull ST_AsGML(geometry) returns GML text
bull ST_GeomFromKML(text) returns geometry
bull ST_AsKML(geometry) returns KML text
bull ST_AsGeoJSON(geometry) returns JSON text
bull ST_AsSVG(geometry) returns SVG text
Analysis with GeoNode DB We can use the DB to perform more complex spatial analysis operations
Many lsquoGIS analysisrsquo tasks are based on a combination of geometric operations between features (intersection distance) and aggregation
PostGIS provides the following geometric operators
bull ST_Contains(geometry A geometry B)
bull ST_Crosses(geometry A geometry B)
bull ST_Disjoint(geometry A geometry B)
bull ST_Distance(geometry A geometry B)
bull ST_DWithin(geometry A geometry B radius)
bull ST_Equals(geometry A geometry B)
bull ST_Intersects(geometry A geometry B)
bull ST_Overlaps(geometry A geometry B)
bull ST_Touches(geometry A geometry B)
bull ST_Within(geometry A geometry B)
Analysis with GeoNode DB spatial joins
The SQL JOIN operator allows us to add fields to a table by performing spatial tests
For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to
We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Contents (2)
bull Access from desktop GIS ndash Connecting and displaying map and feature services
ndash Editing the DB
ndash Projecting data in GIS
ndash Access to metadata catalogs
ndash Exporting styles through SLD
ndash Georeferencing data
ndash Uploading mobile data
ndash Verifying and correcting topology
GeoNode stack
Application layer
Service layer
Data layer
GeoNetwork Web interface
GeoServer Web interface
Direct access to GeoServer bull GeoServer is the component that provides WMS WFS and WCS services
bull It can be used to upload and style data using its own Web front-end
bull However your GeoNode user wonrsquot be recognized by GeoServer you need to use a GeoServer user normally with administration privileges
Connect with your Web browser httpgeonodeURLgeoserver
GeoServer styling with SLD
bull It is easier to work with SLD styles directly within GeoServer
bull Click on the lsquoStylesrsquo link in the left-side menu and select the style you want to edit or add a new one
GeoServer styling with SLD
In the style editor you can bull Change the name of the style
bull Copy the SLD from an existing style
bull Change the code of the SLD
bull Upload an SLD file
bull Validate the SLD to detect errors
GeoServer new styles
bull If you create a new style you need to associate it with a layer
bull To do so select the desired layer in the Layers page
GeoServer new styles
bull Select the lsquoPublishingrsquo tab in the layer properties
In the WMS Settings section you can choose the default style and add more style choices
GeoServer new styles
bull The styling changes will show up in GeoNode (you may need to refresh the page)
GeoServer styling with SLD
bull Exercise
ndash Modify and add new styles for some of your vector layers
ndash Modify the style for the elevation layers
ndash See the changes in GeoNode
GeoServer adding data
bull GeoNode only supports file-based data sources shapefiles and geoTIFF but GeoServer offers more options
bull Data sources are referred in GeoServer as stores
GeoServer vector data stores
bull The uploaded shape files are inserted in the geonode PostGIS database so they are all in the same store called ldquopostgres_importsrdquo
We can access any data from the PostGIS DB using the same store
GeoServer geoTIFF stores
bull The uploaded geoTIFF files are copied in the geonode data folder Each raster file has its own store
We can access any data from the PostGIS DB using the same store
GeoServer adding external data
bull We can add other data sources for GeoNode by creating GeoServer stores
More raster formats
GeoServer adding external data
bull GeoServer has extensions to create stores from other database vendors
bull More information in
httpdocsgeoserverorglatestenuserwebadmindatastoreshtml
GeoServer adding external data
IMPORTANT NOTE
If you add external data directly into GeoServer the GeoNode superuser must run the following shell command to synchronize these new layers to be visible in GeoNode
Unfortunately this updating does yet not work for cascaded WMS layers as we will see
geonode updatelayers
GeoServer adding external data
bull Exercise ndash Add a new store for the GTOPO30
elevation dataset named W200N40dem
ndash After the store is added click on Publish and accept the default
ndash Run the lsquoupdatelayersrsquo command and see if the layer appears in GeoNode
Direct access to GeoNetwork bull GeoNetwork is the component that provides metadata management
and CSW catalog search service
bull It can be used to search and display the GeoNode metadata
bull However do not use Geonetwork to edit the GeoNode metadata bull If you have administration privileges you can still use Geonetwork to create non-spatial
metadata and harvest metadata from other CSW servers
Connect with your Web browser httpgeonodeURLgeonetwork
Direct access to GeoNetwork
bull Exercise ndash Log in Geonetwork as user lsquoadminrsquo with password lsquoadminrsquo
ndash Go to the Administration New metadata menu
ndash Select the Vector data ISO19139 template
Direct access to GeoNetwork
bull Exercise ndash In the medata form type a title like lsquoOnline documentrsquo
ndash Scroll down and type a Web address as online resource
ndash Click on lsquoSave and Closersquo
Direct access to GeoNetwork
bull Exercise ndash In the medata preview click on lsquoPrivilegesrsquo
ndash Set access privileges for All and Submit
ndash Go back and search for your new lsquodatarsquo link in GeoNode
Direct access to the GeoNode DB
bull GeoNode uses a PostGIS database management system to store uploaded vector data as well as the metadata
bull To see the contents of this DB open the pgAdmin Postgres DB administration tool (this has been installed for the workshop)
Direct access to the GeoNode DB
bull Use the lsquoAdd Serverrsquo command to connect to the geonode DB
Use the password that is in the DATABASE_PASSWORD variable in this text file etcgeonodelocal_settingspy
Direct access to the GeoNode DB
Under DatabasesgeonodeSchemas publicTables we will see the tables corresponding to our vector data By right-clicking on a table we have access to some interesting tools
Direct access to the GeoNode DB
bull We can create reports like a data dictionary report
Direct access to the GeoNode DB
bull We can view the data records and filter them
Direct access to the GeoNode DB
bull We can execute SQL queries to select insert or delete records
We can apply operators to perform calculations like average count standard deviation summation etc
Direct access to the GeoNode DB
bull Exercise ndash Use simple SQL queries to extract columns from some layers
SELECT some_columns
FROM some_data_source
WHERE some_condition
Remember column names in UPPERCASE must use QUOTES ldquoLIKETHISrdquo
ndash Use Sum() and Count() operators with SELECT to aggregate results
For instance find out how many different polygons make each District and what is their total areas
Direct access to the GeoNode DB
There are also spatial operators that perform calculations on geometries
bull ST_GeometryType(geometry) returns the type of the geometry
bull ST_SRID(geometry) returns the spatial reference identifier number of the geometry
bull ST_X(point) returns the X ordinate
bull ST_Y(point) returns the Y ordinate
bull ST_Length(linestring) returns the length of the linestring
bull ST_StartPoint(geometry) returns the first coordinate as a point
bull ST_EndPoint(geometry) returns the last coordinate as a point
bull ST_NPoints(geometry) returns the number of coordinates in the linestring
bull ST_Area(geometry) returns the area of the polygons
bull ST_NRings(geometry) returns the number of rings (usually 1 more if there are holes)
bull ST_ExteriorRing(polygon) returns the outer ring as a linestring
bull ST_InteriorRingN(polygon integer) returns a specified interior ring as a linestring
bull ST_Perimeter(geometry) returns the length of all the rings
Direct access to the GeoNode DB
And operators that generate a text description of the geometries in different formats or vice versa
bull ST_AsText(geometry) returns WKT text
bull ST_AsEWKT(geometry) returns EWKT text
bull ST_GeomFromWKB(bytea) returns geometry
bull ST_AsBinary(geometry) returns WKB bytea
bull ST_AsEWKB(geometry) returns EWKB bytea
bull ST_GeomFromGML(text) returns geometry
bull ST_AsGML(geometry) returns GML text
bull ST_GeomFromKML(text) returns geometry
bull ST_AsKML(geometry) returns KML text
bull ST_AsGeoJSON(geometry) returns JSON text
bull ST_AsSVG(geometry) returns SVG text
Analysis with GeoNode DB We can use the DB to perform more complex spatial analysis operations
Many lsquoGIS analysisrsquo tasks are based on a combination of geometric operations between features (intersection distance) and aggregation
PostGIS provides the following geometric operators
bull ST_Contains(geometry A geometry B)
bull ST_Crosses(geometry A geometry B)
bull ST_Disjoint(geometry A geometry B)
bull ST_Distance(geometry A geometry B)
bull ST_DWithin(geometry A geometry B radius)
bull ST_Equals(geometry A geometry B)
bull ST_Intersects(geometry A geometry B)
bull ST_Overlaps(geometry A geometry B)
bull ST_Touches(geometry A geometry B)
bull ST_Within(geometry A geometry B)
Analysis with GeoNode DB spatial joins
The SQL JOIN operator allows us to add fields to a table by performing spatial tests
For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to
We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
GeoNode stack
Application layer
Service layer
Data layer
GeoNetwork Web interface
GeoServer Web interface
Direct access to GeoServer bull GeoServer is the component that provides WMS WFS and WCS services
bull It can be used to upload and style data using its own Web front-end
bull However your GeoNode user wonrsquot be recognized by GeoServer you need to use a GeoServer user normally with administration privileges
Connect with your Web browser httpgeonodeURLgeoserver
GeoServer styling with SLD
bull It is easier to work with SLD styles directly within GeoServer
bull Click on the lsquoStylesrsquo link in the left-side menu and select the style you want to edit or add a new one
GeoServer styling with SLD
In the style editor you can bull Change the name of the style
bull Copy the SLD from an existing style
bull Change the code of the SLD
bull Upload an SLD file
bull Validate the SLD to detect errors
GeoServer new styles
bull If you create a new style you need to associate it with a layer
bull To do so select the desired layer in the Layers page
GeoServer new styles
bull Select the lsquoPublishingrsquo tab in the layer properties
In the WMS Settings section you can choose the default style and add more style choices
GeoServer new styles
bull The styling changes will show up in GeoNode (you may need to refresh the page)
GeoServer styling with SLD
bull Exercise
ndash Modify and add new styles for some of your vector layers
ndash Modify the style for the elevation layers
ndash See the changes in GeoNode
GeoServer adding data
bull GeoNode only supports file-based data sources shapefiles and geoTIFF but GeoServer offers more options
bull Data sources are referred in GeoServer as stores
GeoServer vector data stores
bull The uploaded shape files are inserted in the geonode PostGIS database so they are all in the same store called ldquopostgres_importsrdquo
We can access any data from the PostGIS DB using the same store
GeoServer geoTIFF stores
bull The uploaded geoTIFF files are copied in the geonode data folder Each raster file has its own store
We can access any data from the PostGIS DB using the same store
GeoServer adding external data
bull We can add other data sources for GeoNode by creating GeoServer stores
More raster formats
GeoServer adding external data
bull GeoServer has extensions to create stores from other database vendors
bull More information in
httpdocsgeoserverorglatestenuserwebadmindatastoreshtml
GeoServer adding external data
IMPORTANT NOTE
If you add external data directly into GeoServer the GeoNode superuser must run the following shell command to synchronize these new layers to be visible in GeoNode
Unfortunately this updating does yet not work for cascaded WMS layers as we will see
geonode updatelayers
GeoServer adding external data
bull Exercise ndash Add a new store for the GTOPO30
elevation dataset named W200N40dem
ndash After the store is added click on Publish and accept the default
ndash Run the lsquoupdatelayersrsquo command and see if the layer appears in GeoNode
Direct access to GeoNetwork bull GeoNetwork is the component that provides metadata management
and CSW catalog search service
bull It can be used to search and display the GeoNode metadata
bull However do not use Geonetwork to edit the GeoNode metadata bull If you have administration privileges you can still use Geonetwork to create non-spatial
metadata and harvest metadata from other CSW servers
Connect with your Web browser httpgeonodeURLgeonetwork
Direct access to GeoNetwork
bull Exercise ndash Log in Geonetwork as user lsquoadminrsquo with password lsquoadminrsquo
ndash Go to the Administration New metadata menu
ndash Select the Vector data ISO19139 template
Direct access to GeoNetwork
bull Exercise ndash In the medata form type a title like lsquoOnline documentrsquo
ndash Scroll down and type a Web address as online resource
ndash Click on lsquoSave and Closersquo
Direct access to GeoNetwork
bull Exercise ndash In the medata preview click on lsquoPrivilegesrsquo
ndash Set access privileges for All and Submit
ndash Go back and search for your new lsquodatarsquo link in GeoNode
Direct access to the GeoNode DB
bull GeoNode uses a PostGIS database management system to store uploaded vector data as well as the metadata
bull To see the contents of this DB open the pgAdmin Postgres DB administration tool (this has been installed for the workshop)
Direct access to the GeoNode DB
bull Use the lsquoAdd Serverrsquo command to connect to the geonode DB
Use the password that is in the DATABASE_PASSWORD variable in this text file etcgeonodelocal_settingspy
Direct access to the GeoNode DB
Under DatabasesgeonodeSchemas publicTables we will see the tables corresponding to our vector data By right-clicking on a table we have access to some interesting tools
Direct access to the GeoNode DB
bull We can create reports like a data dictionary report
Direct access to the GeoNode DB
bull We can view the data records and filter them
Direct access to the GeoNode DB
bull We can execute SQL queries to select insert or delete records
We can apply operators to perform calculations like average count standard deviation summation etc
Direct access to the GeoNode DB
bull Exercise ndash Use simple SQL queries to extract columns from some layers
SELECT some_columns
FROM some_data_source
WHERE some_condition
Remember column names in UPPERCASE must use QUOTES ldquoLIKETHISrdquo
ndash Use Sum() and Count() operators with SELECT to aggregate results
For instance find out how many different polygons make each District and what is their total areas
Direct access to the GeoNode DB
There are also spatial operators that perform calculations on geometries
bull ST_GeometryType(geometry) returns the type of the geometry
bull ST_SRID(geometry) returns the spatial reference identifier number of the geometry
bull ST_X(point) returns the X ordinate
bull ST_Y(point) returns the Y ordinate
bull ST_Length(linestring) returns the length of the linestring
bull ST_StartPoint(geometry) returns the first coordinate as a point
bull ST_EndPoint(geometry) returns the last coordinate as a point
bull ST_NPoints(geometry) returns the number of coordinates in the linestring
bull ST_Area(geometry) returns the area of the polygons
bull ST_NRings(geometry) returns the number of rings (usually 1 more if there are holes)
bull ST_ExteriorRing(polygon) returns the outer ring as a linestring
bull ST_InteriorRingN(polygon integer) returns a specified interior ring as a linestring
bull ST_Perimeter(geometry) returns the length of all the rings
Direct access to the GeoNode DB
And operators that generate a text description of the geometries in different formats or vice versa
bull ST_AsText(geometry) returns WKT text
bull ST_AsEWKT(geometry) returns EWKT text
bull ST_GeomFromWKB(bytea) returns geometry
bull ST_AsBinary(geometry) returns WKB bytea
bull ST_AsEWKB(geometry) returns EWKB bytea
bull ST_GeomFromGML(text) returns geometry
bull ST_AsGML(geometry) returns GML text
bull ST_GeomFromKML(text) returns geometry
bull ST_AsKML(geometry) returns KML text
bull ST_AsGeoJSON(geometry) returns JSON text
bull ST_AsSVG(geometry) returns SVG text
Analysis with GeoNode DB We can use the DB to perform more complex spatial analysis operations
Many lsquoGIS analysisrsquo tasks are based on a combination of geometric operations between features (intersection distance) and aggregation
PostGIS provides the following geometric operators
bull ST_Contains(geometry A geometry B)
bull ST_Crosses(geometry A geometry B)
bull ST_Disjoint(geometry A geometry B)
bull ST_Distance(geometry A geometry B)
bull ST_DWithin(geometry A geometry B radius)
bull ST_Equals(geometry A geometry B)
bull ST_Intersects(geometry A geometry B)
bull ST_Overlaps(geometry A geometry B)
bull ST_Touches(geometry A geometry B)
bull ST_Within(geometry A geometry B)
Analysis with GeoNode DB spatial joins
The SQL JOIN operator allows us to add fields to a table by performing spatial tests
For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to
We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Direct access to GeoServer bull GeoServer is the component that provides WMS WFS and WCS services
bull It can be used to upload and style data using its own Web front-end
bull However your GeoNode user wonrsquot be recognized by GeoServer you need to use a GeoServer user normally with administration privileges
Connect with your Web browser httpgeonodeURLgeoserver
GeoServer styling with SLD
bull It is easier to work with SLD styles directly within GeoServer
bull Click on the lsquoStylesrsquo link in the left-side menu and select the style you want to edit or add a new one
GeoServer styling with SLD
In the style editor you can bull Change the name of the style
bull Copy the SLD from an existing style
bull Change the code of the SLD
bull Upload an SLD file
bull Validate the SLD to detect errors
GeoServer new styles
bull If you create a new style you need to associate it with a layer
bull To do so select the desired layer in the Layers page
GeoServer new styles
bull Select the lsquoPublishingrsquo tab in the layer properties
In the WMS Settings section you can choose the default style and add more style choices
GeoServer new styles
bull The styling changes will show up in GeoNode (you may need to refresh the page)
GeoServer styling with SLD
bull Exercise
ndash Modify and add new styles for some of your vector layers
ndash Modify the style for the elevation layers
ndash See the changes in GeoNode
GeoServer adding data
bull GeoNode only supports file-based data sources shapefiles and geoTIFF but GeoServer offers more options
bull Data sources are referred in GeoServer as stores
GeoServer vector data stores
bull The uploaded shape files are inserted in the geonode PostGIS database so they are all in the same store called ldquopostgres_importsrdquo
We can access any data from the PostGIS DB using the same store
GeoServer geoTIFF stores
bull The uploaded geoTIFF files are copied in the geonode data folder Each raster file has its own store
We can access any data from the PostGIS DB using the same store
GeoServer adding external data
bull We can add other data sources for GeoNode by creating GeoServer stores
More raster formats
GeoServer adding external data
bull GeoServer has extensions to create stores from other database vendors
bull More information in
httpdocsgeoserverorglatestenuserwebadmindatastoreshtml
GeoServer adding external data
IMPORTANT NOTE
If you add external data directly into GeoServer the GeoNode superuser must run the following shell command to synchronize these new layers to be visible in GeoNode
Unfortunately this updating does yet not work for cascaded WMS layers as we will see
geonode updatelayers
GeoServer adding external data
bull Exercise ndash Add a new store for the GTOPO30
elevation dataset named W200N40dem
ndash After the store is added click on Publish and accept the default
ndash Run the lsquoupdatelayersrsquo command and see if the layer appears in GeoNode
Direct access to GeoNetwork bull GeoNetwork is the component that provides metadata management
and CSW catalog search service
bull It can be used to search and display the GeoNode metadata
bull However do not use Geonetwork to edit the GeoNode metadata bull If you have administration privileges you can still use Geonetwork to create non-spatial
metadata and harvest metadata from other CSW servers
Connect with your Web browser httpgeonodeURLgeonetwork
Direct access to GeoNetwork
bull Exercise ndash Log in Geonetwork as user lsquoadminrsquo with password lsquoadminrsquo
ndash Go to the Administration New metadata menu
ndash Select the Vector data ISO19139 template
Direct access to GeoNetwork
bull Exercise ndash In the medata form type a title like lsquoOnline documentrsquo
ndash Scroll down and type a Web address as online resource
ndash Click on lsquoSave and Closersquo
Direct access to GeoNetwork
bull Exercise ndash In the medata preview click on lsquoPrivilegesrsquo
ndash Set access privileges for All and Submit
ndash Go back and search for your new lsquodatarsquo link in GeoNode
Direct access to the GeoNode DB
bull GeoNode uses a PostGIS database management system to store uploaded vector data as well as the metadata
bull To see the contents of this DB open the pgAdmin Postgres DB administration tool (this has been installed for the workshop)
Direct access to the GeoNode DB
bull Use the lsquoAdd Serverrsquo command to connect to the geonode DB
Use the password that is in the DATABASE_PASSWORD variable in this text file etcgeonodelocal_settingspy
Direct access to the GeoNode DB
Under DatabasesgeonodeSchemas publicTables we will see the tables corresponding to our vector data By right-clicking on a table we have access to some interesting tools
Direct access to the GeoNode DB
bull We can create reports like a data dictionary report
Direct access to the GeoNode DB
bull We can view the data records and filter them
Direct access to the GeoNode DB
bull We can execute SQL queries to select insert or delete records
We can apply operators to perform calculations like average count standard deviation summation etc
Direct access to the GeoNode DB
bull Exercise ndash Use simple SQL queries to extract columns from some layers
SELECT some_columns
FROM some_data_source
WHERE some_condition
Remember column names in UPPERCASE must use QUOTES ldquoLIKETHISrdquo
ndash Use Sum() and Count() operators with SELECT to aggregate results
For instance find out how many different polygons make each District and what is their total areas
Direct access to the GeoNode DB
There are also spatial operators that perform calculations on geometries
bull ST_GeometryType(geometry) returns the type of the geometry
bull ST_SRID(geometry) returns the spatial reference identifier number of the geometry
bull ST_X(point) returns the X ordinate
bull ST_Y(point) returns the Y ordinate
bull ST_Length(linestring) returns the length of the linestring
bull ST_StartPoint(geometry) returns the first coordinate as a point
bull ST_EndPoint(geometry) returns the last coordinate as a point
bull ST_NPoints(geometry) returns the number of coordinates in the linestring
bull ST_Area(geometry) returns the area of the polygons
bull ST_NRings(geometry) returns the number of rings (usually 1 more if there are holes)
bull ST_ExteriorRing(polygon) returns the outer ring as a linestring
bull ST_InteriorRingN(polygon integer) returns a specified interior ring as a linestring
bull ST_Perimeter(geometry) returns the length of all the rings
Direct access to the GeoNode DB
And operators that generate a text description of the geometries in different formats or vice versa
bull ST_AsText(geometry) returns WKT text
bull ST_AsEWKT(geometry) returns EWKT text
bull ST_GeomFromWKB(bytea) returns geometry
bull ST_AsBinary(geometry) returns WKB bytea
bull ST_AsEWKB(geometry) returns EWKB bytea
bull ST_GeomFromGML(text) returns geometry
bull ST_AsGML(geometry) returns GML text
bull ST_GeomFromKML(text) returns geometry
bull ST_AsKML(geometry) returns KML text
bull ST_AsGeoJSON(geometry) returns JSON text
bull ST_AsSVG(geometry) returns SVG text
Analysis with GeoNode DB We can use the DB to perform more complex spatial analysis operations
Many lsquoGIS analysisrsquo tasks are based on a combination of geometric operations between features (intersection distance) and aggregation
PostGIS provides the following geometric operators
bull ST_Contains(geometry A geometry B)
bull ST_Crosses(geometry A geometry B)
bull ST_Disjoint(geometry A geometry B)
bull ST_Distance(geometry A geometry B)
bull ST_DWithin(geometry A geometry B radius)
bull ST_Equals(geometry A geometry B)
bull ST_Intersects(geometry A geometry B)
bull ST_Overlaps(geometry A geometry B)
bull ST_Touches(geometry A geometry B)
bull ST_Within(geometry A geometry B)
Analysis with GeoNode DB spatial joins
The SQL JOIN operator allows us to add fields to a table by performing spatial tests
For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to
We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
GeoServer styling with SLD
bull It is easier to work with SLD styles directly within GeoServer
bull Click on the lsquoStylesrsquo link in the left-side menu and select the style you want to edit or add a new one
GeoServer styling with SLD
In the style editor you can bull Change the name of the style
bull Copy the SLD from an existing style
bull Change the code of the SLD
bull Upload an SLD file
bull Validate the SLD to detect errors
GeoServer new styles
bull If you create a new style you need to associate it with a layer
bull To do so select the desired layer in the Layers page
GeoServer new styles
bull Select the lsquoPublishingrsquo tab in the layer properties
In the WMS Settings section you can choose the default style and add more style choices
GeoServer new styles
bull The styling changes will show up in GeoNode (you may need to refresh the page)
GeoServer styling with SLD
bull Exercise
ndash Modify and add new styles for some of your vector layers
ndash Modify the style for the elevation layers
ndash See the changes in GeoNode
GeoServer adding data
bull GeoNode only supports file-based data sources shapefiles and geoTIFF but GeoServer offers more options
bull Data sources are referred in GeoServer as stores
GeoServer vector data stores
bull The uploaded shape files are inserted in the geonode PostGIS database so they are all in the same store called ldquopostgres_importsrdquo
We can access any data from the PostGIS DB using the same store
GeoServer geoTIFF stores
bull The uploaded geoTIFF files are copied in the geonode data folder Each raster file has its own store
We can access any data from the PostGIS DB using the same store
GeoServer adding external data
bull We can add other data sources for GeoNode by creating GeoServer stores
More raster formats
GeoServer adding external data
bull GeoServer has extensions to create stores from other database vendors
bull More information in
httpdocsgeoserverorglatestenuserwebadmindatastoreshtml
GeoServer adding external data
IMPORTANT NOTE
If you add external data directly into GeoServer the GeoNode superuser must run the following shell command to synchronize these new layers to be visible in GeoNode
Unfortunately this updating does yet not work for cascaded WMS layers as we will see
geonode updatelayers
GeoServer adding external data
bull Exercise ndash Add a new store for the GTOPO30
elevation dataset named W200N40dem
ndash After the store is added click on Publish and accept the default
ndash Run the lsquoupdatelayersrsquo command and see if the layer appears in GeoNode
Direct access to GeoNetwork bull GeoNetwork is the component that provides metadata management
and CSW catalog search service
bull It can be used to search and display the GeoNode metadata
bull However do not use Geonetwork to edit the GeoNode metadata bull If you have administration privileges you can still use Geonetwork to create non-spatial
metadata and harvest metadata from other CSW servers
Connect with your Web browser httpgeonodeURLgeonetwork
Direct access to GeoNetwork
bull Exercise ndash Log in Geonetwork as user lsquoadminrsquo with password lsquoadminrsquo
ndash Go to the Administration New metadata menu
ndash Select the Vector data ISO19139 template
Direct access to GeoNetwork
bull Exercise ndash In the medata form type a title like lsquoOnline documentrsquo
ndash Scroll down and type a Web address as online resource
ndash Click on lsquoSave and Closersquo
Direct access to GeoNetwork
bull Exercise ndash In the medata preview click on lsquoPrivilegesrsquo
ndash Set access privileges for All and Submit
ndash Go back and search for your new lsquodatarsquo link in GeoNode
Direct access to the GeoNode DB
bull GeoNode uses a PostGIS database management system to store uploaded vector data as well as the metadata
bull To see the contents of this DB open the pgAdmin Postgres DB administration tool (this has been installed for the workshop)
Direct access to the GeoNode DB
bull Use the lsquoAdd Serverrsquo command to connect to the geonode DB
Use the password that is in the DATABASE_PASSWORD variable in this text file etcgeonodelocal_settingspy
Direct access to the GeoNode DB
Under DatabasesgeonodeSchemas publicTables we will see the tables corresponding to our vector data By right-clicking on a table we have access to some interesting tools
Direct access to the GeoNode DB
bull We can create reports like a data dictionary report
Direct access to the GeoNode DB
bull We can view the data records and filter them
Direct access to the GeoNode DB
bull We can execute SQL queries to select insert or delete records
We can apply operators to perform calculations like average count standard deviation summation etc
Direct access to the GeoNode DB
bull Exercise ndash Use simple SQL queries to extract columns from some layers
SELECT some_columns
FROM some_data_source
WHERE some_condition
Remember column names in UPPERCASE must use QUOTES ldquoLIKETHISrdquo
ndash Use Sum() and Count() operators with SELECT to aggregate results
For instance find out how many different polygons make each District and what is their total areas
Direct access to the GeoNode DB
There are also spatial operators that perform calculations on geometries
bull ST_GeometryType(geometry) returns the type of the geometry
bull ST_SRID(geometry) returns the spatial reference identifier number of the geometry
bull ST_X(point) returns the X ordinate
bull ST_Y(point) returns the Y ordinate
bull ST_Length(linestring) returns the length of the linestring
bull ST_StartPoint(geometry) returns the first coordinate as a point
bull ST_EndPoint(geometry) returns the last coordinate as a point
bull ST_NPoints(geometry) returns the number of coordinates in the linestring
bull ST_Area(geometry) returns the area of the polygons
bull ST_NRings(geometry) returns the number of rings (usually 1 more if there are holes)
bull ST_ExteriorRing(polygon) returns the outer ring as a linestring
bull ST_InteriorRingN(polygon integer) returns a specified interior ring as a linestring
bull ST_Perimeter(geometry) returns the length of all the rings
Direct access to the GeoNode DB
And operators that generate a text description of the geometries in different formats or vice versa
bull ST_AsText(geometry) returns WKT text
bull ST_AsEWKT(geometry) returns EWKT text
bull ST_GeomFromWKB(bytea) returns geometry
bull ST_AsBinary(geometry) returns WKB bytea
bull ST_AsEWKB(geometry) returns EWKB bytea
bull ST_GeomFromGML(text) returns geometry
bull ST_AsGML(geometry) returns GML text
bull ST_GeomFromKML(text) returns geometry
bull ST_AsKML(geometry) returns KML text
bull ST_AsGeoJSON(geometry) returns JSON text
bull ST_AsSVG(geometry) returns SVG text
Analysis with GeoNode DB We can use the DB to perform more complex spatial analysis operations
Many lsquoGIS analysisrsquo tasks are based on a combination of geometric operations between features (intersection distance) and aggregation
PostGIS provides the following geometric operators
bull ST_Contains(geometry A geometry B)
bull ST_Crosses(geometry A geometry B)
bull ST_Disjoint(geometry A geometry B)
bull ST_Distance(geometry A geometry B)
bull ST_DWithin(geometry A geometry B radius)
bull ST_Equals(geometry A geometry B)
bull ST_Intersects(geometry A geometry B)
bull ST_Overlaps(geometry A geometry B)
bull ST_Touches(geometry A geometry B)
bull ST_Within(geometry A geometry B)
Analysis with GeoNode DB spatial joins
The SQL JOIN operator allows us to add fields to a table by performing spatial tests
For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to
We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
GeoServer styling with SLD
In the style editor you can bull Change the name of the style
bull Copy the SLD from an existing style
bull Change the code of the SLD
bull Upload an SLD file
bull Validate the SLD to detect errors
GeoServer new styles
bull If you create a new style you need to associate it with a layer
bull To do so select the desired layer in the Layers page
GeoServer new styles
bull Select the lsquoPublishingrsquo tab in the layer properties
In the WMS Settings section you can choose the default style and add more style choices
GeoServer new styles
bull The styling changes will show up in GeoNode (you may need to refresh the page)
GeoServer styling with SLD
bull Exercise
ndash Modify and add new styles for some of your vector layers
ndash Modify the style for the elevation layers
ndash See the changes in GeoNode
GeoServer adding data
bull GeoNode only supports file-based data sources shapefiles and geoTIFF but GeoServer offers more options
bull Data sources are referred in GeoServer as stores
GeoServer vector data stores
bull The uploaded shape files are inserted in the geonode PostGIS database so they are all in the same store called ldquopostgres_importsrdquo
We can access any data from the PostGIS DB using the same store
GeoServer geoTIFF stores
bull The uploaded geoTIFF files are copied in the geonode data folder Each raster file has its own store
We can access any data from the PostGIS DB using the same store
GeoServer adding external data
bull We can add other data sources for GeoNode by creating GeoServer stores
More raster formats
GeoServer adding external data
bull GeoServer has extensions to create stores from other database vendors
bull More information in
httpdocsgeoserverorglatestenuserwebadmindatastoreshtml
GeoServer adding external data
IMPORTANT NOTE
If you add external data directly into GeoServer the GeoNode superuser must run the following shell command to synchronize these new layers to be visible in GeoNode
Unfortunately this updating does yet not work for cascaded WMS layers as we will see
geonode updatelayers
GeoServer adding external data
bull Exercise ndash Add a new store for the GTOPO30
elevation dataset named W200N40dem
ndash After the store is added click on Publish and accept the default
ndash Run the lsquoupdatelayersrsquo command and see if the layer appears in GeoNode
Direct access to GeoNetwork bull GeoNetwork is the component that provides metadata management
and CSW catalog search service
bull It can be used to search and display the GeoNode metadata
bull However do not use Geonetwork to edit the GeoNode metadata bull If you have administration privileges you can still use Geonetwork to create non-spatial
metadata and harvest metadata from other CSW servers
Connect with your Web browser httpgeonodeURLgeonetwork
Direct access to GeoNetwork
bull Exercise ndash Log in Geonetwork as user lsquoadminrsquo with password lsquoadminrsquo
ndash Go to the Administration New metadata menu
ndash Select the Vector data ISO19139 template
Direct access to GeoNetwork
bull Exercise ndash In the medata form type a title like lsquoOnline documentrsquo
ndash Scroll down and type a Web address as online resource
ndash Click on lsquoSave and Closersquo
Direct access to GeoNetwork
bull Exercise ndash In the medata preview click on lsquoPrivilegesrsquo
ndash Set access privileges for All and Submit
ndash Go back and search for your new lsquodatarsquo link in GeoNode
Direct access to the GeoNode DB
bull GeoNode uses a PostGIS database management system to store uploaded vector data as well as the metadata
bull To see the contents of this DB open the pgAdmin Postgres DB administration tool (this has been installed for the workshop)
Direct access to the GeoNode DB
bull Use the lsquoAdd Serverrsquo command to connect to the geonode DB
Use the password that is in the DATABASE_PASSWORD variable in this text file etcgeonodelocal_settingspy
Direct access to the GeoNode DB
Under DatabasesgeonodeSchemas publicTables we will see the tables corresponding to our vector data By right-clicking on a table we have access to some interesting tools
Direct access to the GeoNode DB
bull We can create reports like a data dictionary report
Direct access to the GeoNode DB
bull We can view the data records and filter them
Direct access to the GeoNode DB
bull We can execute SQL queries to select insert or delete records
We can apply operators to perform calculations like average count standard deviation summation etc
Direct access to the GeoNode DB
bull Exercise ndash Use simple SQL queries to extract columns from some layers
SELECT some_columns
FROM some_data_source
WHERE some_condition
Remember column names in UPPERCASE must use QUOTES ldquoLIKETHISrdquo
ndash Use Sum() and Count() operators with SELECT to aggregate results
For instance find out how many different polygons make each District and what is their total areas
Direct access to the GeoNode DB
There are also spatial operators that perform calculations on geometries
bull ST_GeometryType(geometry) returns the type of the geometry
bull ST_SRID(geometry) returns the spatial reference identifier number of the geometry
bull ST_X(point) returns the X ordinate
bull ST_Y(point) returns the Y ordinate
bull ST_Length(linestring) returns the length of the linestring
bull ST_StartPoint(geometry) returns the first coordinate as a point
bull ST_EndPoint(geometry) returns the last coordinate as a point
bull ST_NPoints(geometry) returns the number of coordinates in the linestring
bull ST_Area(geometry) returns the area of the polygons
bull ST_NRings(geometry) returns the number of rings (usually 1 more if there are holes)
bull ST_ExteriorRing(polygon) returns the outer ring as a linestring
bull ST_InteriorRingN(polygon integer) returns a specified interior ring as a linestring
bull ST_Perimeter(geometry) returns the length of all the rings
Direct access to the GeoNode DB
And operators that generate a text description of the geometries in different formats or vice versa
bull ST_AsText(geometry) returns WKT text
bull ST_AsEWKT(geometry) returns EWKT text
bull ST_GeomFromWKB(bytea) returns geometry
bull ST_AsBinary(geometry) returns WKB bytea
bull ST_AsEWKB(geometry) returns EWKB bytea
bull ST_GeomFromGML(text) returns geometry
bull ST_AsGML(geometry) returns GML text
bull ST_GeomFromKML(text) returns geometry
bull ST_AsKML(geometry) returns KML text
bull ST_AsGeoJSON(geometry) returns JSON text
bull ST_AsSVG(geometry) returns SVG text
Analysis with GeoNode DB We can use the DB to perform more complex spatial analysis operations
Many lsquoGIS analysisrsquo tasks are based on a combination of geometric operations between features (intersection distance) and aggregation
PostGIS provides the following geometric operators
bull ST_Contains(geometry A geometry B)
bull ST_Crosses(geometry A geometry B)
bull ST_Disjoint(geometry A geometry B)
bull ST_Distance(geometry A geometry B)
bull ST_DWithin(geometry A geometry B radius)
bull ST_Equals(geometry A geometry B)
bull ST_Intersects(geometry A geometry B)
bull ST_Overlaps(geometry A geometry B)
bull ST_Touches(geometry A geometry B)
bull ST_Within(geometry A geometry B)
Analysis with GeoNode DB spatial joins
The SQL JOIN operator allows us to add fields to a table by performing spatial tests
For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to
We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
GeoServer new styles
bull If you create a new style you need to associate it with a layer
bull To do so select the desired layer in the Layers page
GeoServer new styles
bull Select the lsquoPublishingrsquo tab in the layer properties
In the WMS Settings section you can choose the default style and add more style choices
GeoServer new styles
bull The styling changes will show up in GeoNode (you may need to refresh the page)
GeoServer styling with SLD
bull Exercise
ndash Modify and add new styles for some of your vector layers
ndash Modify the style for the elevation layers
ndash See the changes in GeoNode
GeoServer adding data
bull GeoNode only supports file-based data sources shapefiles and geoTIFF but GeoServer offers more options
bull Data sources are referred in GeoServer as stores
GeoServer vector data stores
bull The uploaded shape files are inserted in the geonode PostGIS database so they are all in the same store called ldquopostgres_importsrdquo
We can access any data from the PostGIS DB using the same store
GeoServer geoTIFF stores
bull The uploaded geoTIFF files are copied in the geonode data folder Each raster file has its own store
We can access any data from the PostGIS DB using the same store
GeoServer adding external data
bull We can add other data sources for GeoNode by creating GeoServer stores
More raster formats
GeoServer adding external data
bull GeoServer has extensions to create stores from other database vendors
bull More information in
httpdocsgeoserverorglatestenuserwebadmindatastoreshtml
GeoServer adding external data
IMPORTANT NOTE
If you add external data directly into GeoServer the GeoNode superuser must run the following shell command to synchronize these new layers to be visible in GeoNode
Unfortunately this updating does yet not work for cascaded WMS layers as we will see
geonode updatelayers
GeoServer adding external data
bull Exercise ndash Add a new store for the GTOPO30
elevation dataset named W200N40dem
ndash After the store is added click on Publish and accept the default
ndash Run the lsquoupdatelayersrsquo command and see if the layer appears in GeoNode
Direct access to GeoNetwork bull GeoNetwork is the component that provides metadata management
and CSW catalog search service
bull It can be used to search and display the GeoNode metadata
bull However do not use Geonetwork to edit the GeoNode metadata bull If you have administration privileges you can still use Geonetwork to create non-spatial
metadata and harvest metadata from other CSW servers
Connect with your Web browser httpgeonodeURLgeonetwork
Direct access to GeoNetwork
bull Exercise ndash Log in Geonetwork as user lsquoadminrsquo with password lsquoadminrsquo
ndash Go to the Administration New metadata menu
ndash Select the Vector data ISO19139 template
Direct access to GeoNetwork
bull Exercise ndash In the medata form type a title like lsquoOnline documentrsquo
ndash Scroll down and type a Web address as online resource
ndash Click on lsquoSave and Closersquo
Direct access to GeoNetwork
bull Exercise ndash In the medata preview click on lsquoPrivilegesrsquo
ndash Set access privileges for All and Submit
ndash Go back and search for your new lsquodatarsquo link in GeoNode
Direct access to the GeoNode DB
bull GeoNode uses a PostGIS database management system to store uploaded vector data as well as the metadata
bull To see the contents of this DB open the pgAdmin Postgres DB administration tool (this has been installed for the workshop)
Direct access to the GeoNode DB
bull Use the lsquoAdd Serverrsquo command to connect to the geonode DB
Use the password that is in the DATABASE_PASSWORD variable in this text file etcgeonodelocal_settingspy
Direct access to the GeoNode DB
Under DatabasesgeonodeSchemas publicTables we will see the tables corresponding to our vector data By right-clicking on a table we have access to some interesting tools
Direct access to the GeoNode DB
bull We can create reports like a data dictionary report
Direct access to the GeoNode DB
bull We can view the data records and filter them
Direct access to the GeoNode DB
bull We can execute SQL queries to select insert or delete records
We can apply operators to perform calculations like average count standard deviation summation etc
Direct access to the GeoNode DB
bull Exercise ndash Use simple SQL queries to extract columns from some layers
SELECT some_columns
FROM some_data_source
WHERE some_condition
Remember column names in UPPERCASE must use QUOTES ldquoLIKETHISrdquo
ndash Use Sum() and Count() operators with SELECT to aggregate results
For instance find out how many different polygons make each District and what is their total areas
Direct access to the GeoNode DB
There are also spatial operators that perform calculations on geometries
bull ST_GeometryType(geometry) returns the type of the geometry
bull ST_SRID(geometry) returns the spatial reference identifier number of the geometry
bull ST_X(point) returns the X ordinate
bull ST_Y(point) returns the Y ordinate
bull ST_Length(linestring) returns the length of the linestring
bull ST_StartPoint(geometry) returns the first coordinate as a point
bull ST_EndPoint(geometry) returns the last coordinate as a point
bull ST_NPoints(geometry) returns the number of coordinates in the linestring
bull ST_Area(geometry) returns the area of the polygons
bull ST_NRings(geometry) returns the number of rings (usually 1 more if there are holes)
bull ST_ExteriorRing(polygon) returns the outer ring as a linestring
bull ST_InteriorRingN(polygon integer) returns a specified interior ring as a linestring
bull ST_Perimeter(geometry) returns the length of all the rings
Direct access to the GeoNode DB
And operators that generate a text description of the geometries in different formats or vice versa
bull ST_AsText(geometry) returns WKT text
bull ST_AsEWKT(geometry) returns EWKT text
bull ST_GeomFromWKB(bytea) returns geometry
bull ST_AsBinary(geometry) returns WKB bytea
bull ST_AsEWKB(geometry) returns EWKB bytea
bull ST_GeomFromGML(text) returns geometry
bull ST_AsGML(geometry) returns GML text
bull ST_GeomFromKML(text) returns geometry
bull ST_AsKML(geometry) returns KML text
bull ST_AsGeoJSON(geometry) returns JSON text
bull ST_AsSVG(geometry) returns SVG text
Analysis with GeoNode DB We can use the DB to perform more complex spatial analysis operations
Many lsquoGIS analysisrsquo tasks are based on a combination of geometric operations between features (intersection distance) and aggregation
PostGIS provides the following geometric operators
bull ST_Contains(geometry A geometry B)
bull ST_Crosses(geometry A geometry B)
bull ST_Disjoint(geometry A geometry B)
bull ST_Distance(geometry A geometry B)
bull ST_DWithin(geometry A geometry B radius)
bull ST_Equals(geometry A geometry B)
bull ST_Intersects(geometry A geometry B)
bull ST_Overlaps(geometry A geometry B)
bull ST_Touches(geometry A geometry B)
bull ST_Within(geometry A geometry B)
Analysis with GeoNode DB spatial joins
The SQL JOIN operator allows us to add fields to a table by performing spatial tests
For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to
We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
GeoServer new styles
bull Select the lsquoPublishingrsquo tab in the layer properties
In the WMS Settings section you can choose the default style and add more style choices
GeoServer new styles
bull The styling changes will show up in GeoNode (you may need to refresh the page)
GeoServer styling with SLD
bull Exercise
ndash Modify and add new styles for some of your vector layers
ndash Modify the style for the elevation layers
ndash See the changes in GeoNode
GeoServer adding data
bull GeoNode only supports file-based data sources shapefiles and geoTIFF but GeoServer offers more options
bull Data sources are referred in GeoServer as stores
GeoServer vector data stores
bull The uploaded shape files are inserted in the geonode PostGIS database so they are all in the same store called ldquopostgres_importsrdquo
We can access any data from the PostGIS DB using the same store
GeoServer geoTIFF stores
bull The uploaded geoTIFF files are copied in the geonode data folder Each raster file has its own store
We can access any data from the PostGIS DB using the same store
GeoServer adding external data
bull We can add other data sources for GeoNode by creating GeoServer stores
More raster formats
GeoServer adding external data
bull GeoServer has extensions to create stores from other database vendors
bull More information in
httpdocsgeoserverorglatestenuserwebadmindatastoreshtml
GeoServer adding external data
IMPORTANT NOTE
If you add external data directly into GeoServer the GeoNode superuser must run the following shell command to synchronize these new layers to be visible in GeoNode
Unfortunately this updating does yet not work for cascaded WMS layers as we will see
geonode updatelayers
GeoServer adding external data
bull Exercise ndash Add a new store for the GTOPO30
elevation dataset named W200N40dem
ndash After the store is added click on Publish and accept the default
ndash Run the lsquoupdatelayersrsquo command and see if the layer appears in GeoNode
Direct access to GeoNetwork bull GeoNetwork is the component that provides metadata management
and CSW catalog search service
bull It can be used to search and display the GeoNode metadata
bull However do not use Geonetwork to edit the GeoNode metadata bull If you have administration privileges you can still use Geonetwork to create non-spatial
metadata and harvest metadata from other CSW servers
Connect with your Web browser httpgeonodeURLgeonetwork
Direct access to GeoNetwork
bull Exercise ndash Log in Geonetwork as user lsquoadminrsquo with password lsquoadminrsquo
ndash Go to the Administration New metadata menu
ndash Select the Vector data ISO19139 template
Direct access to GeoNetwork
bull Exercise ndash In the medata form type a title like lsquoOnline documentrsquo
ndash Scroll down and type a Web address as online resource
ndash Click on lsquoSave and Closersquo
Direct access to GeoNetwork
bull Exercise ndash In the medata preview click on lsquoPrivilegesrsquo
ndash Set access privileges for All and Submit
ndash Go back and search for your new lsquodatarsquo link in GeoNode
Direct access to the GeoNode DB
bull GeoNode uses a PostGIS database management system to store uploaded vector data as well as the metadata
bull To see the contents of this DB open the pgAdmin Postgres DB administration tool (this has been installed for the workshop)
Direct access to the GeoNode DB
bull Use the lsquoAdd Serverrsquo command to connect to the geonode DB
Use the password that is in the DATABASE_PASSWORD variable in this text file etcgeonodelocal_settingspy
Direct access to the GeoNode DB
Under DatabasesgeonodeSchemas publicTables we will see the tables corresponding to our vector data By right-clicking on a table we have access to some interesting tools
Direct access to the GeoNode DB
bull We can create reports like a data dictionary report
Direct access to the GeoNode DB
bull We can view the data records and filter them
Direct access to the GeoNode DB
bull We can execute SQL queries to select insert or delete records
We can apply operators to perform calculations like average count standard deviation summation etc
Direct access to the GeoNode DB
bull Exercise ndash Use simple SQL queries to extract columns from some layers
SELECT some_columns
FROM some_data_source
WHERE some_condition
Remember column names in UPPERCASE must use QUOTES ldquoLIKETHISrdquo
ndash Use Sum() and Count() operators with SELECT to aggregate results
For instance find out how many different polygons make each District and what is their total areas
Direct access to the GeoNode DB
There are also spatial operators that perform calculations on geometries
bull ST_GeometryType(geometry) returns the type of the geometry
bull ST_SRID(geometry) returns the spatial reference identifier number of the geometry
bull ST_X(point) returns the X ordinate
bull ST_Y(point) returns the Y ordinate
bull ST_Length(linestring) returns the length of the linestring
bull ST_StartPoint(geometry) returns the first coordinate as a point
bull ST_EndPoint(geometry) returns the last coordinate as a point
bull ST_NPoints(geometry) returns the number of coordinates in the linestring
bull ST_Area(geometry) returns the area of the polygons
bull ST_NRings(geometry) returns the number of rings (usually 1 more if there are holes)
bull ST_ExteriorRing(polygon) returns the outer ring as a linestring
bull ST_InteriorRingN(polygon integer) returns a specified interior ring as a linestring
bull ST_Perimeter(geometry) returns the length of all the rings
Direct access to the GeoNode DB
And operators that generate a text description of the geometries in different formats or vice versa
bull ST_AsText(geometry) returns WKT text
bull ST_AsEWKT(geometry) returns EWKT text
bull ST_GeomFromWKB(bytea) returns geometry
bull ST_AsBinary(geometry) returns WKB bytea
bull ST_AsEWKB(geometry) returns EWKB bytea
bull ST_GeomFromGML(text) returns geometry
bull ST_AsGML(geometry) returns GML text
bull ST_GeomFromKML(text) returns geometry
bull ST_AsKML(geometry) returns KML text
bull ST_AsGeoJSON(geometry) returns JSON text
bull ST_AsSVG(geometry) returns SVG text
Analysis with GeoNode DB We can use the DB to perform more complex spatial analysis operations
Many lsquoGIS analysisrsquo tasks are based on a combination of geometric operations between features (intersection distance) and aggregation
PostGIS provides the following geometric operators
bull ST_Contains(geometry A geometry B)
bull ST_Crosses(geometry A geometry B)
bull ST_Disjoint(geometry A geometry B)
bull ST_Distance(geometry A geometry B)
bull ST_DWithin(geometry A geometry B radius)
bull ST_Equals(geometry A geometry B)
bull ST_Intersects(geometry A geometry B)
bull ST_Overlaps(geometry A geometry B)
bull ST_Touches(geometry A geometry B)
bull ST_Within(geometry A geometry B)
Analysis with GeoNode DB spatial joins
The SQL JOIN operator allows us to add fields to a table by performing spatial tests
For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to
We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
GeoServer new styles
bull The styling changes will show up in GeoNode (you may need to refresh the page)
GeoServer styling with SLD
bull Exercise
ndash Modify and add new styles for some of your vector layers
ndash Modify the style for the elevation layers
ndash See the changes in GeoNode
GeoServer adding data
bull GeoNode only supports file-based data sources shapefiles and geoTIFF but GeoServer offers more options
bull Data sources are referred in GeoServer as stores
GeoServer vector data stores
bull The uploaded shape files are inserted in the geonode PostGIS database so they are all in the same store called ldquopostgres_importsrdquo
We can access any data from the PostGIS DB using the same store
GeoServer geoTIFF stores
bull The uploaded geoTIFF files are copied in the geonode data folder Each raster file has its own store
We can access any data from the PostGIS DB using the same store
GeoServer adding external data
bull We can add other data sources for GeoNode by creating GeoServer stores
More raster formats
GeoServer adding external data
bull GeoServer has extensions to create stores from other database vendors
bull More information in
httpdocsgeoserverorglatestenuserwebadmindatastoreshtml
GeoServer adding external data
IMPORTANT NOTE
If you add external data directly into GeoServer the GeoNode superuser must run the following shell command to synchronize these new layers to be visible in GeoNode
Unfortunately this updating does yet not work for cascaded WMS layers as we will see
geonode updatelayers
GeoServer adding external data
bull Exercise ndash Add a new store for the GTOPO30
elevation dataset named W200N40dem
ndash After the store is added click on Publish and accept the default
ndash Run the lsquoupdatelayersrsquo command and see if the layer appears in GeoNode
Direct access to GeoNetwork bull GeoNetwork is the component that provides metadata management
and CSW catalog search service
bull It can be used to search and display the GeoNode metadata
bull However do not use Geonetwork to edit the GeoNode metadata bull If you have administration privileges you can still use Geonetwork to create non-spatial
metadata and harvest metadata from other CSW servers
Connect with your Web browser httpgeonodeURLgeonetwork
Direct access to GeoNetwork
bull Exercise ndash Log in Geonetwork as user lsquoadminrsquo with password lsquoadminrsquo
ndash Go to the Administration New metadata menu
ndash Select the Vector data ISO19139 template
Direct access to GeoNetwork
bull Exercise ndash In the medata form type a title like lsquoOnline documentrsquo
ndash Scroll down and type a Web address as online resource
ndash Click on lsquoSave and Closersquo
Direct access to GeoNetwork
bull Exercise ndash In the medata preview click on lsquoPrivilegesrsquo
ndash Set access privileges for All and Submit
ndash Go back and search for your new lsquodatarsquo link in GeoNode
Direct access to the GeoNode DB
bull GeoNode uses a PostGIS database management system to store uploaded vector data as well as the metadata
bull To see the contents of this DB open the pgAdmin Postgres DB administration tool (this has been installed for the workshop)
Direct access to the GeoNode DB
bull Use the lsquoAdd Serverrsquo command to connect to the geonode DB
Use the password that is in the DATABASE_PASSWORD variable in this text file etcgeonodelocal_settingspy
Direct access to the GeoNode DB
Under DatabasesgeonodeSchemas publicTables we will see the tables corresponding to our vector data By right-clicking on a table we have access to some interesting tools
Direct access to the GeoNode DB
bull We can create reports like a data dictionary report
Direct access to the GeoNode DB
bull We can view the data records and filter them
Direct access to the GeoNode DB
bull We can execute SQL queries to select insert or delete records
We can apply operators to perform calculations like average count standard deviation summation etc
Direct access to the GeoNode DB
bull Exercise ndash Use simple SQL queries to extract columns from some layers
SELECT some_columns
FROM some_data_source
WHERE some_condition
Remember column names in UPPERCASE must use QUOTES ldquoLIKETHISrdquo
ndash Use Sum() and Count() operators with SELECT to aggregate results
For instance find out how many different polygons make each District and what is their total areas
Direct access to the GeoNode DB
There are also spatial operators that perform calculations on geometries
bull ST_GeometryType(geometry) returns the type of the geometry
bull ST_SRID(geometry) returns the spatial reference identifier number of the geometry
bull ST_X(point) returns the X ordinate
bull ST_Y(point) returns the Y ordinate
bull ST_Length(linestring) returns the length of the linestring
bull ST_StartPoint(geometry) returns the first coordinate as a point
bull ST_EndPoint(geometry) returns the last coordinate as a point
bull ST_NPoints(geometry) returns the number of coordinates in the linestring
bull ST_Area(geometry) returns the area of the polygons
bull ST_NRings(geometry) returns the number of rings (usually 1 more if there are holes)
bull ST_ExteriorRing(polygon) returns the outer ring as a linestring
bull ST_InteriorRingN(polygon integer) returns a specified interior ring as a linestring
bull ST_Perimeter(geometry) returns the length of all the rings
Direct access to the GeoNode DB
And operators that generate a text description of the geometries in different formats or vice versa
bull ST_AsText(geometry) returns WKT text
bull ST_AsEWKT(geometry) returns EWKT text
bull ST_GeomFromWKB(bytea) returns geometry
bull ST_AsBinary(geometry) returns WKB bytea
bull ST_AsEWKB(geometry) returns EWKB bytea
bull ST_GeomFromGML(text) returns geometry
bull ST_AsGML(geometry) returns GML text
bull ST_GeomFromKML(text) returns geometry
bull ST_AsKML(geometry) returns KML text
bull ST_AsGeoJSON(geometry) returns JSON text
bull ST_AsSVG(geometry) returns SVG text
Analysis with GeoNode DB We can use the DB to perform more complex spatial analysis operations
Many lsquoGIS analysisrsquo tasks are based on a combination of geometric operations between features (intersection distance) and aggregation
PostGIS provides the following geometric operators
bull ST_Contains(geometry A geometry B)
bull ST_Crosses(geometry A geometry B)
bull ST_Disjoint(geometry A geometry B)
bull ST_Distance(geometry A geometry B)
bull ST_DWithin(geometry A geometry B radius)
bull ST_Equals(geometry A geometry B)
bull ST_Intersects(geometry A geometry B)
bull ST_Overlaps(geometry A geometry B)
bull ST_Touches(geometry A geometry B)
bull ST_Within(geometry A geometry B)
Analysis with GeoNode DB spatial joins
The SQL JOIN operator allows us to add fields to a table by performing spatial tests
For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to
We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
GeoServer styling with SLD
bull Exercise
ndash Modify and add new styles for some of your vector layers
ndash Modify the style for the elevation layers
ndash See the changes in GeoNode
GeoServer adding data
bull GeoNode only supports file-based data sources shapefiles and geoTIFF but GeoServer offers more options
bull Data sources are referred in GeoServer as stores
GeoServer vector data stores
bull The uploaded shape files are inserted in the geonode PostGIS database so they are all in the same store called ldquopostgres_importsrdquo
We can access any data from the PostGIS DB using the same store
GeoServer geoTIFF stores
bull The uploaded geoTIFF files are copied in the geonode data folder Each raster file has its own store
We can access any data from the PostGIS DB using the same store
GeoServer adding external data
bull We can add other data sources for GeoNode by creating GeoServer stores
More raster formats
GeoServer adding external data
bull GeoServer has extensions to create stores from other database vendors
bull More information in
httpdocsgeoserverorglatestenuserwebadmindatastoreshtml
GeoServer adding external data
IMPORTANT NOTE
If you add external data directly into GeoServer the GeoNode superuser must run the following shell command to synchronize these new layers to be visible in GeoNode
Unfortunately this updating does yet not work for cascaded WMS layers as we will see
geonode updatelayers
GeoServer adding external data
bull Exercise ndash Add a new store for the GTOPO30
elevation dataset named W200N40dem
ndash After the store is added click on Publish and accept the default
ndash Run the lsquoupdatelayersrsquo command and see if the layer appears in GeoNode
Direct access to GeoNetwork bull GeoNetwork is the component that provides metadata management
and CSW catalog search service
bull It can be used to search and display the GeoNode metadata
bull However do not use Geonetwork to edit the GeoNode metadata bull If you have administration privileges you can still use Geonetwork to create non-spatial
metadata and harvest metadata from other CSW servers
Connect with your Web browser httpgeonodeURLgeonetwork
Direct access to GeoNetwork
bull Exercise ndash Log in Geonetwork as user lsquoadminrsquo with password lsquoadminrsquo
ndash Go to the Administration New metadata menu
ndash Select the Vector data ISO19139 template
Direct access to GeoNetwork
bull Exercise ndash In the medata form type a title like lsquoOnline documentrsquo
ndash Scroll down and type a Web address as online resource
ndash Click on lsquoSave and Closersquo
Direct access to GeoNetwork
bull Exercise ndash In the medata preview click on lsquoPrivilegesrsquo
ndash Set access privileges for All and Submit
ndash Go back and search for your new lsquodatarsquo link in GeoNode
Direct access to the GeoNode DB
bull GeoNode uses a PostGIS database management system to store uploaded vector data as well as the metadata
bull To see the contents of this DB open the pgAdmin Postgres DB administration tool (this has been installed for the workshop)
Direct access to the GeoNode DB
bull Use the lsquoAdd Serverrsquo command to connect to the geonode DB
Use the password that is in the DATABASE_PASSWORD variable in this text file etcgeonodelocal_settingspy
Direct access to the GeoNode DB
Under DatabasesgeonodeSchemas publicTables we will see the tables corresponding to our vector data By right-clicking on a table we have access to some interesting tools
Direct access to the GeoNode DB
bull We can create reports like a data dictionary report
Direct access to the GeoNode DB
bull We can view the data records and filter them
Direct access to the GeoNode DB
bull We can execute SQL queries to select insert or delete records
We can apply operators to perform calculations like average count standard deviation summation etc
Direct access to the GeoNode DB
bull Exercise ndash Use simple SQL queries to extract columns from some layers
SELECT some_columns
FROM some_data_source
WHERE some_condition
Remember column names in UPPERCASE must use QUOTES ldquoLIKETHISrdquo
ndash Use Sum() and Count() operators with SELECT to aggregate results
For instance find out how many different polygons make each District and what is their total areas
Direct access to the GeoNode DB
There are also spatial operators that perform calculations on geometries
bull ST_GeometryType(geometry) returns the type of the geometry
bull ST_SRID(geometry) returns the spatial reference identifier number of the geometry
bull ST_X(point) returns the X ordinate
bull ST_Y(point) returns the Y ordinate
bull ST_Length(linestring) returns the length of the linestring
bull ST_StartPoint(geometry) returns the first coordinate as a point
bull ST_EndPoint(geometry) returns the last coordinate as a point
bull ST_NPoints(geometry) returns the number of coordinates in the linestring
bull ST_Area(geometry) returns the area of the polygons
bull ST_NRings(geometry) returns the number of rings (usually 1 more if there are holes)
bull ST_ExteriorRing(polygon) returns the outer ring as a linestring
bull ST_InteriorRingN(polygon integer) returns a specified interior ring as a linestring
bull ST_Perimeter(geometry) returns the length of all the rings
Direct access to the GeoNode DB
And operators that generate a text description of the geometries in different formats or vice versa
bull ST_AsText(geometry) returns WKT text
bull ST_AsEWKT(geometry) returns EWKT text
bull ST_GeomFromWKB(bytea) returns geometry
bull ST_AsBinary(geometry) returns WKB bytea
bull ST_AsEWKB(geometry) returns EWKB bytea
bull ST_GeomFromGML(text) returns geometry
bull ST_AsGML(geometry) returns GML text
bull ST_GeomFromKML(text) returns geometry
bull ST_AsKML(geometry) returns KML text
bull ST_AsGeoJSON(geometry) returns JSON text
bull ST_AsSVG(geometry) returns SVG text
Analysis with GeoNode DB We can use the DB to perform more complex spatial analysis operations
Many lsquoGIS analysisrsquo tasks are based on a combination of geometric operations between features (intersection distance) and aggregation
PostGIS provides the following geometric operators
bull ST_Contains(geometry A geometry B)
bull ST_Crosses(geometry A geometry B)
bull ST_Disjoint(geometry A geometry B)
bull ST_Distance(geometry A geometry B)
bull ST_DWithin(geometry A geometry B radius)
bull ST_Equals(geometry A geometry B)
bull ST_Intersects(geometry A geometry B)
bull ST_Overlaps(geometry A geometry B)
bull ST_Touches(geometry A geometry B)
bull ST_Within(geometry A geometry B)
Analysis with GeoNode DB spatial joins
The SQL JOIN operator allows us to add fields to a table by performing spatial tests
For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to
We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
GeoServer adding data
bull GeoNode only supports file-based data sources shapefiles and geoTIFF but GeoServer offers more options
bull Data sources are referred in GeoServer as stores
GeoServer vector data stores
bull The uploaded shape files are inserted in the geonode PostGIS database so they are all in the same store called ldquopostgres_importsrdquo
We can access any data from the PostGIS DB using the same store
GeoServer geoTIFF stores
bull The uploaded geoTIFF files are copied in the geonode data folder Each raster file has its own store
We can access any data from the PostGIS DB using the same store
GeoServer adding external data
bull We can add other data sources for GeoNode by creating GeoServer stores
More raster formats
GeoServer adding external data
bull GeoServer has extensions to create stores from other database vendors
bull More information in
httpdocsgeoserverorglatestenuserwebadmindatastoreshtml
GeoServer adding external data
IMPORTANT NOTE
If you add external data directly into GeoServer the GeoNode superuser must run the following shell command to synchronize these new layers to be visible in GeoNode
Unfortunately this updating does yet not work for cascaded WMS layers as we will see
geonode updatelayers
GeoServer adding external data
bull Exercise ndash Add a new store for the GTOPO30
elevation dataset named W200N40dem
ndash After the store is added click on Publish and accept the default
ndash Run the lsquoupdatelayersrsquo command and see if the layer appears in GeoNode
Direct access to GeoNetwork bull GeoNetwork is the component that provides metadata management
and CSW catalog search service
bull It can be used to search and display the GeoNode metadata
bull However do not use Geonetwork to edit the GeoNode metadata bull If you have administration privileges you can still use Geonetwork to create non-spatial
metadata and harvest metadata from other CSW servers
Connect with your Web browser httpgeonodeURLgeonetwork
Direct access to GeoNetwork
bull Exercise ndash Log in Geonetwork as user lsquoadminrsquo with password lsquoadminrsquo
ndash Go to the Administration New metadata menu
ndash Select the Vector data ISO19139 template
Direct access to GeoNetwork
bull Exercise ndash In the medata form type a title like lsquoOnline documentrsquo
ndash Scroll down and type a Web address as online resource
ndash Click on lsquoSave and Closersquo
Direct access to GeoNetwork
bull Exercise ndash In the medata preview click on lsquoPrivilegesrsquo
ndash Set access privileges for All and Submit
ndash Go back and search for your new lsquodatarsquo link in GeoNode
Direct access to the GeoNode DB
bull GeoNode uses a PostGIS database management system to store uploaded vector data as well as the metadata
bull To see the contents of this DB open the pgAdmin Postgres DB administration tool (this has been installed for the workshop)
Direct access to the GeoNode DB
bull Use the lsquoAdd Serverrsquo command to connect to the geonode DB
Use the password that is in the DATABASE_PASSWORD variable in this text file etcgeonodelocal_settingspy
Direct access to the GeoNode DB
Under DatabasesgeonodeSchemas publicTables we will see the tables corresponding to our vector data By right-clicking on a table we have access to some interesting tools
Direct access to the GeoNode DB
bull We can create reports like a data dictionary report
Direct access to the GeoNode DB
bull We can view the data records and filter them
Direct access to the GeoNode DB
bull We can execute SQL queries to select insert or delete records
We can apply operators to perform calculations like average count standard deviation summation etc
Direct access to the GeoNode DB
bull Exercise ndash Use simple SQL queries to extract columns from some layers
SELECT some_columns
FROM some_data_source
WHERE some_condition
Remember column names in UPPERCASE must use QUOTES ldquoLIKETHISrdquo
ndash Use Sum() and Count() operators with SELECT to aggregate results
For instance find out how many different polygons make each District and what is their total areas
Direct access to the GeoNode DB
There are also spatial operators that perform calculations on geometries
bull ST_GeometryType(geometry) returns the type of the geometry
bull ST_SRID(geometry) returns the spatial reference identifier number of the geometry
bull ST_X(point) returns the X ordinate
bull ST_Y(point) returns the Y ordinate
bull ST_Length(linestring) returns the length of the linestring
bull ST_StartPoint(geometry) returns the first coordinate as a point
bull ST_EndPoint(geometry) returns the last coordinate as a point
bull ST_NPoints(geometry) returns the number of coordinates in the linestring
bull ST_Area(geometry) returns the area of the polygons
bull ST_NRings(geometry) returns the number of rings (usually 1 more if there are holes)
bull ST_ExteriorRing(polygon) returns the outer ring as a linestring
bull ST_InteriorRingN(polygon integer) returns a specified interior ring as a linestring
bull ST_Perimeter(geometry) returns the length of all the rings
Direct access to the GeoNode DB
And operators that generate a text description of the geometries in different formats or vice versa
bull ST_AsText(geometry) returns WKT text
bull ST_AsEWKT(geometry) returns EWKT text
bull ST_GeomFromWKB(bytea) returns geometry
bull ST_AsBinary(geometry) returns WKB bytea
bull ST_AsEWKB(geometry) returns EWKB bytea
bull ST_GeomFromGML(text) returns geometry
bull ST_AsGML(geometry) returns GML text
bull ST_GeomFromKML(text) returns geometry
bull ST_AsKML(geometry) returns KML text
bull ST_AsGeoJSON(geometry) returns JSON text
bull ST_AsSVG(geometry) returns SVG text
Analysis with GeoNode DB We can use the DB to perform more complex spatial analysis operations
Many lsquoGIS analysisrsquo tasks are based on a combination of geometric operations between features (intersection distance) and aggregation
PostGIS provides the following geometric operators
bull ST_Contains(geometry A geometry B)
bull ST_Crosses(geometry A geometry B)
bull ST_Disjoint(geometry A geometry B)
bull ST_Distance(geometry A geometry B)
bull ST_DWithin(geometry A geometry B radius)
bull ST_Equals(geometry A geometry B)
bull ST_Intersects(geometry A geometry B)
bull ST_Overlaps(geometry A geometry B)
bull ST_Touches(geometry A geometry B)
bull ST_Within(geometry A geometry B)
Analysis with GeoNode DB spatial joins
The SQL JOIN operator allows us to add fields to a table by performing spatial tests
For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to
We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
GeoServer vector data stores
bull The uploaded shape files are inserted in the geonode PostGIS database so they are all in the same store called ldquopostgres_importsrdquo
We can access any data from the PostGIS DB using the same store
GeoServer geoTIFF stores
bull The uploaded geoTIFF files are copied in the geonode data folder Each raster file has its own store
We can access any data from the PostGIS DB using the same store
GeoServer adding external data
bull We can add other data sources for GeoNode by creating GeoServer stores
More raster formats
GeoServer adding external data
bull GeoServer has extensions to create stores from other database vendors
bull More information in
httpdocsgeoserverorglatestenuserwebadmindatastoreshtml
GeoServer adding external data
IMPORTANT NOTE
If you add external data directly into GeoServer the GeoNode superuser must run the following shell command to synchronize these new layers to be visible in GeoNode
Unfortunately this updating does yet not work for cascaded WMS layers as we will see
geonode updatelayers
GeoServer adding external data
bull Exercise ndash Add a new store for the GTOPO30
elevation dataset named W200N40dem
ndash After the store is added click on Publish and accept the default
ndash Run the lsquoupdatelayersrsquo command and see if the layer appears in GeoNode
Direct access to GeoNetwork bull GeoNetwork is the component that provides metadata management
and CSW catalog search service
bull It can be used to search and display the GeoNode metadata
bull However do not use Geonetwork to edit the GeoNode metadata bull If you have administration privileges you can still use Geonetwork to create non-spatial
metadata and harvest metadata from other CSW servers
Connect with your Web browser httpgeonodeURLgeonetwork
Direct access to GeoNetwork
bull Exercise ndash Log in Geonetwork as user lsquoadminrsquo with password lsquoadminrsquo
ndash Go to the Administration New metadata menu
ndash Select the Vector data ISO19139 template
Direct access to GeoNetwork
bull Exercise ndash In the medata form type a title like lsquoOnline documentrsquo
ndash Scroll down and type a Web address as online resource
ndash Click on lsquoSave and Closersquo
Direct access to GeoNetwork
bull Exercise ndash In the medata preview click on lsquoPrivilegesrsquo
ndash Set access privileges for All and Submit
ndash Go back and search for your new lsquodatarsquo link in GeoNode
Direct access to the GeoNode DB
bull GeoNode uses a PostGIS database management system to store uploaded vector data as well as the metadata
bull To see the contents of this DB open the pgAdmin Postgres DB administration tool (this has been installed for the workshop)
Direct access to the GeoNode DB
bull Use the lsquoAdd Serverrsquo command to connect to the geonode DB
Use the password that is in the DATABASE_PASSWORD variable in this text file etcgeonodelocal_settingspy
Direct access to the GeoNode DB
Under DatabasesgeonodeSchemas publicTables we will see the tables corresponding to our vector data By right-clicking on a table we have access to some interesting tools
Direct access to the GeoNode DB
bull We can create reports like a data dictionary report
Direct access to the GeoNode DB
bull We can view the data records and filter them
Direct access to the GeoNode DB
bull We can execute SQL queries to select insert or delete records
We can apply operators to perform calculations like average count standard deviation summation etc
Direct access to the GeoNode DB
bull Exercise ndash Use simple SQL queries to extract columns from some layers
SELECT some_columns
FROM some_data_source
WHERE some_condition
Remember column names in UPPERCASE must use QUOTES ldquoLIKETHISrdquo
ndash Use Sum() and Count() operators with SELECT to aggregate results
For instance find out how many different polygons make each District and what is their total areas
Direct access to the GeoNode DB
There are also spatial operators that perform calculations on geometries
bull ST_GeometryType(geometry) returns the type of the geometry
bull ST_SRID(geometry) returns the spatial reference identifier number of the geometry
bull ST_X(point) returns the X ordinate
bull ST_Y(point) returns the Y ordinate
bull ST_Length(linestring) returns the length of the linestring
bull ST_StartPoint(geometry) returns the first coordinate as a point
bull ST_EndPoint(geometry) returns the last coordinate as a point
bull ST_NPoints(geometry) returns the number of coordinates in the linestring
bull ST_Area(geometry) returns the area of the polygons
bull ST_NRings(geometry) returns the number of rings (usually 1 more if there are holes)
bull ST_ExteriorRing(polygon) returns the outer ring as a linestring
bull ST_InteriorRingN(polygon integer) returns a specified interior ring as a linestring
bull ST_Perimeter(geometry) returns the length of all the rings
Direct access to the GeoNode DB
And operators that generate a text description of the geometries in different formats or vice versa
bull ST_AsText(geometry) returns WKT text
bull ST_AsEWKT(geometry) returns EWKT text
bull ST_GeomFromWKB(bytea) returns geometry
bull ST_AsBinary(geometry) returns WKB bytea
bull ST_AsEWKB(geometry) returns EWKB bytea
bull ST_GeomFromGML(text) returns geometry
bull ST_AsGML(geometry) returns GML text
bull ST_GeomFromKML(text) returns geometry
bull ST_AsKML(geometry) returns KML text
bull ST_AsGeoJSON(geometry) returns JSON text
bull ST_AsSVG(geometry) returns SVG text
Analysis with GeoNode DB We can use the DB to perform more complex spatial analysis operations
Many lsquoGIS analysisrsquo tasks are based on a combination of geometric operations between features (intersection distance) and aggregation
PostGIS provides the following geometric operators
bull ST_Contains(geometry A geometry B)
bull ST_Crosses(geometry A geometry B)
bull ST_Disjoint(geometry A geometry B)
bull ST_Distance(geometry A geometry B)
bull ST_DWithin(geometry A geometry B radius)
bull ST_Equals(geometry A geometry B)
bull ST_Intersects(geometry A geometry B)
bull ST_Overlaps(geometry A geometry B)
bull ST_Touches(geometry A geometry B)
bull ST_Within(geometry A geometry B)
Analysis with GeoNode DB spatial joins
The SQL JOIN operator allows us to add fields to a table by performing spatial tests
For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to
We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
GeoServer geoTIFF stores
bull The uploaded geoTIFF files are copied in the geonode data folder Each raster file has its own store
We can access any data from the PostGIS DB using the same store
GeoServer adding external data
bull We can add other data sources for GeoNode by creating GeoServer stores
More raster formats
GeoServer adding external data
bull GeoServer has extensions to create stores from other database vendors
bull More information in
httpdocsgeoserverorglatestenuserwebadmindatastoreshtml
GeoServer adding external data
IMPORTANT NOTE
If you add external data directly into GeoServer the GeoNode superuser must run the following shell command to synchronize these new layers to be visible in GeoNode
Unfortunately this updating does yet not work for cascaded WMS layers as we will see
geonode updatelayers
GeoServer adding external data
bull Exercise ndash Add a new store for the GTOPO30
elevation dataset named W200N40dem
ndash After the store is added click on Publish and accept the default
ndash Run the lsquoupdatelayersrsquo command and see if the layer appears in GeoNode
Direct access to GeoNetwork bull GeoNetwork is the component that provides metadata management
and CSW catalog search service
bull It can be used to search and display the GeoNode metadata
bull However do not use Geonetwork to edit the GeoNode metadata bull If you have administration privileges you can still use Geonetwork to create non-spatial
metadata and harvest metadata from other CSW servers
Connect with your Web browser httpgeonodeURLgeonetwork
Direct access to GeoNetwork
bull Exercise ndash Log in Geonetwork as user lsquoadminrsquo with password lsquoadminrsquo
ndash Go to the Administration New metadata menu
ndash Select the Vector data ISO19139 template
Direct access to GeoNetwork
bull Exercise ndash In the medata form type a title like lsquoOnline documentrsquo
ndash Scroll down and type a Web address as online resource
ndash Click on lsquoSave and Closersquo
Direct access to GeoNetwork
bull Exercise ndash In the medata preview click on lsquoPrivilegesrsquo
ndash Set access privileges for All and Submit
ndash Go back and search for your new lsquodatarsquo link in GeoNode
Direct access to the GeoNode DB
bull GeoNode uses a PostGIS database management system to store uploaded vector data as well as the metadata
bull To see the contents of this DB open the pgAdmin Postgres DB administration tool (this has been installed for the workshop)
Direct access to the GeoNode DB
bull Use the lsquoAdd Serverrsquo command to connect to the geonode DB
Use the password that is in the DATABASE_PASSWORD variable in this text file etcgeonodelocal_settingspy
Direct access to the GeoNode DB
Under DatabasesgeonodeSchemas publicTables we will see the tables corresponding to our vector data By right-clicking on a table we have access to some interesting tools
Direct access to the GeoNode DB
bull We can create reports like a data dictionary report
Direct access to the GeoNode DB
bull We can view the data records and filter them
Direct access to the GeoNode DB
bull We can execute SQL queries to select insert or delete records
We can apply operators to perform calculations like average count standard deviation summation etc
Direct access to the GeoNode DB
bull Exercise ndash Use simple SQL queries to extract columns from some layers
SELECT some_columns
FROM some_data_source
WHERE some_condition
Remember column names in UPPERCASE must use QUOTES ldquoLIKETHISrdquo
ndash Use Sum() and Count() operators with SELECT to aggregate results
For instance find out how many different polygons make each District and what is their total areas
Direct access to the GeoNode DB
There are also spatial operators that perform calculations on geometries
bull ST_GeometryType(geometry) returns the type of the geometry
bull ST_SRID(geometry) returns the spatial reference identifier number of the geometry
bull ST_X(point) returns the X ordinate
bull ST_Y(point) returns the Y ordinate
bull ST_Length(linestring) returns the length of the linestring
bull ST_StartPoint(geometry) returns the first coordinate as a point
bull ST_EndPoint(geometry) returns the last coordinate as a point
bull ST_NPoints(geometry) returns the number of coordinates in the linestring
bull ST_Area(geometry) returns the area of the polygons
bull ST_NRings(geometry) returns the number of rings (usually 1 more if there are holes)
bull ST_ExteriorRing(polygon) returns the outer ring as a linestring
bull ST_InteriorRingN(polygon integer) returns a specified interior ring as a linestring
bull ST_Perimeter(geometry) returns the length of all the rings
Direct access to the GeoNode DB
And operators that generate a text description of the geometries in different formats or vice versa
bull ST_AsText(geometry) returns WKT text
bull ST_AsEWKT(geometry) returns EWKT text
bull ST_GeomFromWKB(bytea) returns geometry
bull ST_AsBinary(geometry) returns WKB bytea
bull ST_AsEWKB(geometry) returns EWKB bytea
bull ST_GeomFromGML(text) returns geometry
bull ST_AsGML(geometry) returns GML text
bull ST_GeomFromKML(text) returns geometry
bull ST_AsKML(geometry) returns KML text
bull ST_AsGeoJSON(geometry) returns JSON text
bull ST_AsSVG(geometry) returns SVG text
Analysis with GeoNode DB We can use the DB to perform more complex spatial analysis operations
Many lsquoGIS analysisrsquo tasks are based on a combination of geometric operations between features (intersection distance) and aggregation
PostGIS provides the following geometric operators
bull ST_Contains(geometry A geometry B)
bull ST_Crosses(geometry A geometry B)
bull ST_Disjoint(geometry A geometry B)
bull ST_Distance(geometry A geometry B)
bull ST_DWithin(geometry A geometry B radius)
bull ST_Equals(geometry A geometry B)
bull ST_Intersects(geometry A geometry B)
bull ST_Overlaps(geometry A geometry B)
bull ST_Touches(geometry A geometry B)
bull ST_Within(geometry A geometry B)
Analysis with GeoNode DB spatial joins
The SQL JOIN operator allows us to add fields to a table by performing spatial tests
For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to
We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
GeoServer adding external data
bull We can add other data sources for GeoNode by creating GeoServer stores
More raster formats
GeoServer adding external data
bull GeoServer has extensions to create stores from other database vendors
bull More information in
httpdocsgeoserverorglatestenuserwebadmindatastoreshtml
GeoServer adding external data
IMPORTANT NOTE
If you add external data directly into GeoServer the GeoNode superuser must run the following shell command to synchronize these new layers to be visible in GeoNode
Unfortunately this updating does yet not work for cascaded WMS layers as we will see
geonode updatelayers
GeoServer adding external data
bull Exercise ndash Add a new store for the GTOPO30
elevation dataset named W200N40dem
ndash After the store is added click on Publish and accept the default
ndash Run the lsquoupdatelayersrsquo command and see if the layer appears in GeoNode
Direct access to GeoNetwork bull GeoNetwork is the component that provides metadata management
and CSW catalog search service
bull It can be used to search and display the GeoNode metadata
bull However do not use Geonetwork to edit the GeoNode metadata bull If you have administration privileges you can still use Geonetwork to create non-spatial
metadata and harvest metadata from other CSW servers
Connect with your Web browser httpgeonodeURLgeonetwork
Direct access to GeoNetwork
bull Exercise ndash Log in Geonetwork as user lsquoadminrsquo with password lsquoadminrsquo
ndash Go to the Administration New metadata menu
ndash Select the Vector data ISO19139 template
Direct access to GeoNetwork
bull Exercise ndash In the medata form type a title like lsquoOnline documentrsquo
ndash Scroll down and type a Web address as online resource
ndash Click on lsquoSave and Closersquo
Direct access to GeoNetwork
bull Exercise ndash In the medata preview click on lsquoPrivilegesrsquo
ndash Set access privileges for All and Submit
ndash Go back and search for your new lsquodatarsquo link in GeoNode
Direct access to the GeoNode DB
bull GeoNode uses a PostGIS database management system to store uploaded vector data as well as the metadata
bull To see the contents of this DB open the pgAdmin Postgres DB administration tool (this has been installed for the workshop)
Direct access to the GeoNode DB
bull Use the lsquoAdd Serverrsquo command to connect to the geonode DB
Use the password that is in the DATABASE_PASSWORD variable in this text file etcgeonodelocal_settingspy
Direct access to the GeoNode DB
Under DatabasesgeonodeSchemas publicTables we will see the tables corresponding to our vector data By right-clicking on a table we have access to some interesting tools
Direct access to the GeoNode DB
bull We can create reports like a data dictionary report
Direct access to the GeoNode DB
bull We can view the data records and filter them
Direct access to the GeoNode DB
bull We can execute SQL queries to select insert or delete records
We can apply operators to perform calculations like average count standard deviation summation etc
Direct access to the GeoNode DB
bull Exercise ndash Use simple SQL queries to extract columns from some layers
SELECT some_columns
FROM some_data_source
WHERE some_condition
Remember column names in UPPERCASE must use QUOTES ldquoLIKETHISrdquo
ndash Use Sum() and Count() operators with SELECT to aggregate results
For instance find out how many different polygons make each District and what is their total areas
Direct access to the GeoNode DB
There are also spatial operators that perform calculations on geometries
bull ST_GeometryType(geometry) returns the type of the geometry
bull ST_SRID(geometry) returns the spatial reference identifier number of the geometry
bull ST_X(point) returns the X ordinate
bull ST_Y(point) returns the Y ordinate
bull ST_Length(linestring) returns the length of the linestring
bull ST_StartPoint(geometry) returns the first coordinate as a point
bull ST_EndPoint(geometry) returns the last coordinate as a point
bull ST_NPoints(geometry) returns the number of coordinates in the linestring
bull ST_Area(geometry) returns the area of the polygons
bull ST_NRings(geometry) returns the number of rings (usually 1 more if there are holes)
bull ST_ExteriorRing(polygon) returns the outer ring as a linestring
bull ST_InteriorRingN(polygon integer) returns a specified interior ring as a linestring
bull ST_Perimeter(geometry) returns the length of all the rings
Direct access to the GeoNode DB
And operators that generate a text description of the geometries in different formats or vice versa
bull ST_AsText(geometry) returns WKT text
bull ST_AsEWKT(geometry) returns EWKT text
bull ST_GeomFromWKB(bytea) returns geometry
bull ST_AsBinary(geometry) returns WKB bytea
bull ST_AsEWKB(geometry) returns EWKB bytea
bull ST_GeomFromGML(text) returns geometry
bull ST_AsGML(geometry) returns GML text
bull ST_GeomFromKML(text) returns geometry
bull ST_AsKML(geometry) returns KML text
bull ST_AsGeoJSON(geometry) returns JSON text
bull ST_AsSVG(geometry) returns SVG text
Analysis with GeoNode DB We can use the DB to perform more complex spatial analysis operations
Many lsquoGIS analysisrsquo tasks are based on a combination of geometric operations between features (intersection distance) and aggregation
PostGIS provides the following geometric operators
bull ST_Contains(geometry A geometry B)
bull ST_Crosses(geometry A geometry B)
bull ST_Disjoint(geometry A geometry B)
bull ST_Distance(geometry A geometry B)
bull ST_DWithin(geometry A geometry B radius)
bull ST_Equals(geometry A geometry B)
bull ST_Intersects(geometry A geometry B)
bull ST_Overlaps(geometry A geometry B)
bull ST_Touches(geometry A geometry B)
bull ST_Within(geometry A geometry B)
Analysis with GeoNode DB spatial joins
The SQL JOIN operator allows us to add fields to a table by performing spatial tests
For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to
We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
GeoServer adding external data
bull GeoServer has extensions to create stores from other database vendors
bull More information in
httpdocsgeoserverorglatestenuserwebadmindatastoreshtml
GeoServer adding external data
IMPORTANT NOTE
If you add external data directly into GeoServer the GeoNode superuser must run the following shell command to synchronize these new layers to be visible in GeoNode
Unfortunately this updating does yet not work for cascaded WMS layers as we will see
geonode updatelayers
GeoServer adding external data
bull Exercise ndash Add a new store for the GTOPO30
elevation dataset named W200N40dem
ndash After the store is added click on Publish and accept the default
ndash Run the lsquoupdatelayersrsquo command and see if the layer appears in GeoNode
Direct access to GeoNetwork bull GeoNetwork is the component that provides metadata management
and CSW catalog search service
bull It can be used to search and display the GeoNode metadata
bull However do not use Geonetwork to edit the GeoNode metadata bull If you have administration privileges you can still use Geonetwork to create non-spatial
metadata and harvest metadata from other CSW servers
Connect with your Web browser httpgeonodeURLgeonetwork
Direct access to GeoNetwork
bull Exercise ndash Log in Geonetwork as user lsquoadminrsquo with password lsquoadminrsquo
ndash Go to the Administration New metadata menu
ndash Select the Vector data ISO19139 template
Direct access to GeoNetwork
bull Exercise ndash In the medata form type a title like lsquoOnline documentrsquo
ndash Scroll down and type a Web address as online resource
ndash Click on lsquoSave and Closersquo
Direct access to GeoNetwork
bull Exercise ndash In the medata preview click on lsquoPrivilegesrsquo
ndash Set access privileges for All and Submit
ndash Go back and search for your new lsquodatarsquo link in GeoNode
Direct access to the GeoNode DB
bull GeoNode uses a PostGIS database management system to store uploaded vector data as well as the metadata
bull To see the contents of this DB open the pgAdmin Postgres DB administration tool (this has been installed for the workshop)
Direct access to the GeoNode DB
bull Use the lsquoAdd Serverrsquo command to connect to the geonode DB
Use the password that is in the DATABASE_PASSWORD variable in this text file etcgeonodelocal_settingspy
Direct access to the GeoNode DB
Under DatabasesgeonodeSchemas publicTables we will see the tables corresponding to our vector data By right-clicking on a table we have access to some interesting tools
Direct access to the GeoNode DB
bull We can create reports like a data dictionary report
Direct access to the GeoNode DB
bull We can view the data records and filter them
Direct access to the GeoNode DB
bull We can execute SQL queries to select insert or delete records
We can apply operators to perform calculations like average count standard deviation summation etc
Direct access to the GeoNode DB
bull Exercise ndash Use simple SQL queries to extract columns from some layers
SELECT some_columns
FROM some_data_source
WHERE some_condition
Remember column names in UPPERCASE must use QUOTES ldquoLIKETHISrdquo
ndash Use Sum() and Count() operators with SELECT to aggregate results
For instance find out how many different polygons make each District and what is their total areas
Direct access to the GeoNode DB
There are also spatial operators that perform calculations on geometries
bull ST_GeometryType(geometry) returns the type of the geometry
bull ST_SRID(geometry) returns the spatial reference identifier number of the geometry
bull ST_X(point) returns the X ordinate
bull ST_Y(point) returns the Y ordinate
bull ST_Length(linestring) returns the length of the linestring
bull ST_StartPoint(geometry) returns the first coordinate as a point
bull ST_EndPoint(geometry) returns the last coordinate as a point
bull ST_NPoints(geometry) returns the number of coordinates in the linestring
bull ST_Area(geometry) returns the area of the polygons
bull ST_NRings(geometry) returns the number of rings (usually 1 more if there are holes)
bull ST_ExteriorRing(polygon) returns the outer ring as a linestring
bull ST_InteriorRingN(polygon integer) returns a specified interior ring as a linestring
bull ST_Perimeter(geometry) returns the length of all the rings
Direct access to the GeoNode DB
And operators that generate a text description of the geometries in different formats or vice versa
bull ST_AsText(geometry) returns WKT text
bull ST_AsEWKT(geometry) returns EWKT text
bull ST_GeomFromWKB(bytea) returns geometry
bull ST_AsBinary(geometry) returns WKB bytea
bull ST_AsEWKB(geometry) returns EWKB bytea
bull ST_GeomFromGML(text) returns geometry
bull ST_AsGML(geometry) returns GML text
bull ST_GeomFromKML(text) returns geometry
bull ST_AsKML(geometry) returns KML text
bull ST_AsGeoJSON(geometry) returns JSON text
bull ST_AsSVG(geometry) returns SVG text
Analysis with GeoNode DB We can use the DB to perform more complex spatial analysis operations
Many lsquoGIS analysisrsquo tasks are based on a combination of geometric operations between features (intersection distance) and aggregation
PostGIS provides the following geometric operators
bull ST_Contains(geometry A geometry B)
bull ST_Crosses(geometry A geometry B)
bull ST_Disjoint(geometry A geometry B)
bull ST_Distance(geometry A geometry B)
bull ST_DWithin(geometry A geometry B radius)
bull ST_Equals(geometry A geometry B)
bull ST_Intersects(geometry A geometry B)
bull ST_Overlaps(geometry A geometry B)
bull ST_Touches(geometry A geometry B)
bull ST_Within(geometry A geometry B)
Analysis with GeoNode DB spatial joins
The SQL JOIN operator allows us to add fields to a table by performing spatial tests
For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to
We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
GeoServer adding external data
IMPORTANT NOTE
If you add external data directly into GeoServer the GeoNode superuser must run the following shell command to synchronize these new layers to be visible in GeoNode
Unfortunately this updating does yet not work for cascaded WMS layers as we will see
geonode updatelayers
GeoServer adding external data
bull Exercise ndash Add a new store for the GTOPO30
elevation dataset named W200N40dem
ndash After the store is added click on Publish and accept the default
ndash Run the lsquoupdatelayersrsquo command and see if the layer appears in GeoNode
Direct access to GeoNetwork bull GeoNetwork is the component that provides metadata management
and CSW catalog search service
bull It can be used to search and display the GeoNode metadata
bull However do not use Geonetwork to edit the GeoNode metadata bull If you have administration privileges you can still use Geonetwork to create non-spatial
metadata and harvest metadata from other CSW servers
Connect with your Web browser httpgeonodeURLgeonetwork
Direct access to GeoNetwork
bull Exercise ndash Log in Geonetwork as user lsquoadminrsquo with password lsquoadminrsquo
ndash Go to the Administration New metadata menu
ndash Select the Vector data ISO19139 template
Direct access to GeoNetwork
bull Exercise ndash In the medata form type a title like lsquoOnline documentrsquo
ndash Scroll down and type a Web address as online resource
ndash Click on lsquoSave and Closersquo
Direct access to GeoNetwork
bull Exercise ndash In the medata preview click on lsquoPrivilegesrsquo
ndash Set access privileges for All and Submit
ndash Go back and search for your new lsquodatarsquo link in GeoNode
Direct access to the GeoNode DB
bull GeoNode uses a PostGIS database management system to store uploaded vector data as well as the metadata
bull To see the contents of this DB open the pgAdmin Postgres DB administration tool (this has been installed for the workshop)
Direct access to the GeoNode DB
bull Use the lsquoAdd Serverrsquo command to connect to the geonode DB
Use the password that is in the DATABASE_PASSWORD variable in this text file etcgeonodelocal_settingspy
Direct access to the GeoNode DB
Under DatabasesgeonodeSchemas publicTables we will see the tables corresponding to our vector data By right-clicking on a table we have access to some interesting tools
Direct access to the GeoNode DB
bull We can create reports like a data dictionary report
Direct access to the GeoNode DB
bull We can view the data records and filter them
Direct access to the GeoNode DB
bull We can execute SQL queries to select insert or delete records
We can apply operators to perform calculations like average count standard deviation summation etc
Direct access to the GeoNode DB
bull Exercise ndash Use simple SQL queries to extract columns from some layers
SELECT some_columns
FROM some_data_source
WHERE some_condition
Remember column names in UPPERCASE must use QUOTES ldquoLIKETHISrdquo
ndash Use Sum() and Count() operators with SELECT to aggregate results
For instance find out how many different polygons make each District and what is their total areas
Direct access to the GeoNode DB
There are also spatial operators that perform calculations on geometries
bull ST_GeometryType(geometry) returns the type of the geometry
bull ST_SRID(geometry) returns the spatial reference identifier number of the geometry
bull ST_X(point) returns the X ordinate
bull ST_Y(point) returns the Y ordinate
bull ST_Length(linestring) returns the length of the linestring
bull ST_StartPoint(geometry) returns the first coordinate as a point
bull ST_EndPoint(geometry) returns the last coordinate as a point
bull ST_NPoints(geometry) returns the number of coordinates in the linestring
bull ST_Area(geometry) returns the area of the polygons
bull ST_NRings(geometry) returns the number of rings (usually 1 more if there are holes)
bull ST_ExteriorRing(polygon) returns the outer ring as a linestring
bull ST_InteriorRingN(polygon integer) returns a specified interior ring as a linestring
bull ST_Perimeter(geometry) returns the length of all the rings
Direct access to the GeoNode DB
And operators that generate a text description of the geometries in different formats or vice versa
bull ST_AsText(geometry) returns WKT text
bull ST_AsEWKT(geometry) returns EWKT text
bull ST_GeomFromWKB(bytea) returns geometry
bull ST_AsBinary(geometry) returns WKB bytea
bull ST_AsEWKB(geometry) returns EWKB bytea
bull ST_GeomFromGML(text) returns geometry
bull ST_AsGML(geometry) returns GML text
bull ST_GeomFromKML(text) returns geometry
bull ST_AsKML(geometry) returns KML text
bull ST_AsGeoJSON(geometry) returns JSON text
bull ST_AsSVG(geometry) returns SVG text
Analysis with GeoNode DB We can use the DB to perform more complex spatial analysis operations
Many lsquoGIS analysisrsquo tasks are based on a combination of geometric operations between features (intersection distance) and aggregation
PostGIS provides the following geometric operators
bull ST_Contains(geometry A geometry B)
bull ST_Crosses(geometry A geometry B)
bull ST_Disjoint(geometry A geometry B)
bull ST_Distance(geometry A geometry B)
bull ST_DWithin(geometry A geometry B radius)
bull ST_Equals(geometry A geometry B)
bull ST_Intersects(geometry A geometry B)
bull ST_Overlaps(geometry A geometry B)
bull ST_Touches(geometry A geometry B)
bull ST_Within(geometry A geometry B)
Analysis with GeoNode DB spatial joins
The SQL JOIN operator allows us to add fields to a table by performing spatial tests
For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to
We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
GeoServer adding external data
bull Exercise ndash Add a new store for the GTOPO30
elevation dataset named W200N40dem
ndash After the store is added click on Publish and accept the default
ndash Run the lsquoupdatelayersrsquo command and see if the layer appears in GeoNode
Direct access to GeoNetwork bull GeoNetwork is the component that provides metadata management
and CSW catalog search service
bull It can be used to search and display the GeoNode metadata
bull However do not use Geonetwork to edit the GeoNode metadata bull If you have administration privileges you can still use Geonetwork to create non-spatial
metadata and harvest metadata from other CSW servers
Connect with your Web browser httpgeonodeURLgeonetwork
Direct access to GeoNetwork
bull Exercise ndash Log in Geonetwork as user lsquoadminrsquo with password lsquoadminrsquo
ndash Go to the Administration New metadata menu
ndash Select the Vector data ISO19139 template
Direct access to GeoNetwork
bull Exercise ndash In the medata form type a title like lsquoOnline documentrsquo
ndash Scroll down and type a Web address as online resource
ndash Click on lsquoSave and Closersquo
Direct access to GeoNetwork
bull Exercise ndash In the medata preview click on lsquoPrivilegesrsquo
ndash Set access privileges for All and Submit
ndash Go back and search for your new lsquodatarsquo link in GeoNode
Direct access to the GeoNode DB
bull GeoNode uses a PostGIS database management system to store uploaded vector data as well as the metadata
bull To see the contents of this DB open the pgAdmin Postgres DB administration tool (this has been installed for the workshop)
Direct access to the GeoNode DB
bull Use the lsquoAdd Serverrsquo command to connect to the geonode DB
Use the password that is in the DATABASE_PASSWORD variable in this text file etcgeonodelocal_settingspy
Direct access to the GeoNode DB
Under DatabasesgeonodeSchemas publicTables we will see the tables corresponding to our vector data By right-clicking on a table we have access to some interesting tools
Direct access to the GeoNode DB
bull We can create reports like a data dictionary report
Direct access to the GeoNode DB
bull We can view the data records and filter them
Direct access to the GeoNode DB
bull We can execute SQL queries to select insert or delete records
We can apply operators to perform calculations like average count standard deviation summation etc
Direct access to the GeoNode DB
bull Exercise ndash Use simple SQL queries to extract columns from some layers
SELECT some_columns
FROM some_data_source
WHERE some_condition
Remember column names in UPPERCASE must use QUOTES ldquoLIKETHISrdquo
ndash Use Sum() and Count() operators with SELECT to aggregate results
For instance find out how many different polygons make each District and what is their total areas
Direct access to the GeoNode DB
There are also spatial operators that perform calculations on geometries
bull ST_GeometryType(geometry) returns the type of the geometry
bull ST_SRID(geometry) returns the spatial reference identifier number of the geometry
bull ST_X(point) returns the X ordinate
bull ST_Y(point) returns the Y ordinate
bull ST_Length(linestring) returns the length of the linestring
bull ST_StartPoint(geometry) returns the first coordinate as a point
bull ST_EndPoint(geometry) returns the last coordinate as a point
bull ST_NPoints(geometry) returns the number of coordinates in the linestring
bull ST_Area(geometry) returns the area of the polygons
bull ST_NRings(geometry) returns the number of rings (usually 1 more if there are holes)
bull ST_ExteriorRing(polygon) returns the outer ring as a linestring
bull ST_InteriorRingN(polygon integer) returns a specified interior ring as a linestring
bull ST_Perimeter(geometry) returns the length of all the rings
Direct access to the GeoNode DB
And operators that generate a text description of the geometries in different formats or vice versa
bull ST_AsText(geometry) returns WKT text
bull ST_AsEWKT(geometry) returns EWKT text
bull ST_GeomFromWKB(bytea) returns geometry
bull ST_AsBinary(geometry) returns WKB bytea
bull ST_AsEWKB(geometry) returns EWKB bytea
bull ST_GeomFromGML(text) returns geometry
bull ST_AsGML(geometry) returns GML text
bull ST_GeomFromKML(text) returns geometry
bull ST_AsKML(geometry) returns KML text
bull ST_AsGeoJSON(geometry) returns JSON text
bull ST_AsSVG(geometry) returns SVG text
Analysis with GeoNode DB We can use the DB to perform more complex spatial analysis operations
Many lsquoGIS analysisrsquo tasks are based on a combination of geometric operations between features (intersection distance) and aggregation
PostGIS provides the following geometric operators
bull ST_Contains(geometry A geometry B)
bull ST_Crosses(geometry A geometry B)
bull ST_Disjoint(geometry A geometry B)
bull ST_Distance(geometry A geometry B)
bull ST_DWithin(geometry A geometry B radius)
bull ST_Equals(geometry A geometry B)
bull ST_Intersects(geometry A geometry B)
bull ST_Overlaps(geometry A geometry B)
bull ST_Touches(geometry A geometry B)
bull ST_Within(geometry A geometry B)
Analysis with GeoNode DB spatial joins
The SQL JOIN operator allows us to add fields to a table by performing spatial tests
For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to
We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Direct access to GeoNetwork bull GeoNetwork is the component that provides metadata management
and CSW catalog search service
bull It can be used to search and display the GeoNode metadata
bull However do not use Geonetwork to edit the GeoNode metadata bull If you have administration privileges you can still use Geonetwork to create non-spatial
metadata and harvest metadata from other CSW servers
Connect with your Web browser httpgeonodeURLgeonetwork
Direct access to GeoNetwork
bull Exercise ndash Log in Geonetwork as user lsquoadminrsquo with password lsquoadminrsquo
ndash Go to the Administration New metadata menu
ndash Select the Vector data ISO19139 template
Direct access to GeoNetwork
bull Exercise ndash In the medata form type a title like lsquoOnline documentrsquo
ndash Scroll down and type a Web address as online resource
ndash Click on lsquoSave and Closersquo
Direct access to GeoNetwork
bull Exercise ndash In the medata preview click on lsquoPrivilegesrsquo
ndash Set access privileges for All and Submit
ndash Go back and search for your new lsquodatarsquo link in GeoNode
Direct access to the GeoNode DB
bull GeoNode uses a PostGIS database management system to store uploaded vector data as well as the metadata
bull To see the contents of this DB open the pgAdmin Postgres DB administration tool (this has been installed for the workshop)
Direct access to the GeoNode DB
bull Use the lsquoAdd Serverrsquo command to connect to the geonode DB
Use the password that is in the DATABASE_PASSWORD variable in this text file etcgeonodelocal_settingspy
Direct access to the GeoNode DB
Under DatabasesgeonodeSchemas publicTables we will see the tables corresponding to our vector data By right-clicking on a table we have access to some interesting tools
Direct access to the GeoNode DB
bull We can create reports like a data dictionary report
Direct access to the GeoNode DB
bull We can view the data records and filter them
Direct access to the GeoNode DB
bull We can execute SQL queries to select insert or delete records
We can apply operators to perform calculations like average count standard deviation summation etc
Direct access to the GeoNode DB
bull Exercise ndash Use simple SQL queries to extract columns from some layers
SELECT some_columns
FROM some_data_source
WHERE some_condition
Remember column names in UPPERCASE must use QUOTES ldquoLIKETHISrdquo
ndash Use Sum() and Count() operators with SELECT to aggregate results
For instance find out how many different polygons make each District and what is their total areas
Direct access to the GeoNode DB
There are also spatial operators that perform calculations on geometries
bull ST_GeometryType(geometry) returns the type of the geometry
bull ST_SRID(geometry) returns the spatial reference identifier number of the geometry
bull ST_X(point) returns the X ordinate
bull ST_Y(point) returns the Y ordinate
bull ST_Length(linestring) returns the length of the linestring
bull ST_StartPoint(geometry) returns the first coordinate as a point
bull ST_EndPoint(geometry) returns the last coordinate as a point
bull ST_NPoints(geometry) returns the number of coordinates in the linestring
bull ST_Area(geometry) returns the area of the polygons
bull ST_NRings(geometry) returns the number of rings (usually 1 more if there are holes)
bull ST_ExteriorRing(polygon) returns the outer ring as a linestring
bull ST_InteriorRingN(polygon integer) returns a specified interior ring as a linestring
bull ST_Perimeter(geometry) returns the length of all the rings
Direct access to the GeoNode DB
And operators that generate a text description of the geometries in different formats or vice versa
bull ST_AsText(geometry) returns WKT text
bull ST_AsEWKT(geometry) returns EWKT text
bull ST_GeomFromWKB(bytea) returns geometry
bull ST_AsBinary(geometry) returns WKB bytea
bull ST_AsEWKB(geometry) returns EWKB bytea
bull ST_GeomFromGML(text) returns geometry
bull ST_AsGML(geometry) returns GML text
bull ST_GeomFromKML(text) returns geometry
bull ST_AsKML(geometry) returns KML text
bull ST_AsGeoJSON(geometry) returns JSON text
bull ST_AsSVG(geometry) returns SVG text
Analysis with GeoNode DB We can use the DB to perform more complex spatial analysis operations
Many lsquoGIS analysisrsquo tasks are based on a combination of geometric operations between features (intersection distance) and aggregation
PostGIS provides the following geometric operators
bull ST_Contains(geometry A geometry B)
bull ST_Crosses(geometry A geometry B)
bull ST_Disjoint(geometry A geometry B)
bull ST_Distance(geometry A geometry B)
bull ST_DWithin(geometry A geometry B radius)
bull ST_Equals(geometry A geometry B)
bull ST_Intersects(geometry A geometry B)
bull ST_Overlaps(geometry A geometry B)
bull ST_Touches(geometry A geometry B)
bull ST_Within(geometry A geometry B)
Analysis with GeoNode DB spatial joins
The SQL JOIN operator allows us to add fields to a table by performing spatial tests
For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to
We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Direct access to GeoNetwork
bull Exercise ndash Log in Geonetwork as user lsquoadminrsquo with password lsquoadminrsquo
ndash Go to the Administration New metadata menu
ndash Select the Vector data ISO19139 template
Direct access to GeoNetwork
bull Exercise ndash In the medata form type a title like lsquoOnline documentrsquo
ndash Scroll down and type a Web address as online resource
ndash Click on lsquoSave and Closersquo
Direct access to GeoNetwork
bull Exercise ndash In the medata preview click on lsquoPrivilegesrsquo
ndash Set access privileges for All and Submit
ndash Go back and search for your new lsquodatarsquo link in GeoNode
Direct access to the GeoNode DB
bull GeoNode uses a PostGIS database management system to store uploaded vector data as well as the metadata
bull To see the contents of this DB open the pgAdmin Postgres DB administration tool (this has been installed for the workshop)
Direct access to the GeoNode DB
bull Use the lsquoAdd Serverrsquo command to connect to the geonode DB
Use the password that is in the DATABASE_PASSWORD variable in this text file etcgeonodelocal_settingspy
Direct access to the GeoNode DB
Under DatabasesgeonodeSchemas publicTables we will see the tables corresponding to our vector data By right-clicking on a table we have access to some interesting tools
Direct access to the GeoNode DB
bull We can create reports like a data dictionary report
Direct access to the GeoNode DB
bull We can view the data records and filter them
Direct access to the GeoNode DB
bull We can execute SQL queries to select insert or delete records
We can apply operators to perform calculations like average count standard deviation summation etc
Direct access to the GeoNode DB
bull Exercise ndash Use simple SQL queries to extract columns from some layers
SELECT some_columns
FROM some_data_source
WHERE some_condition
Remember column names in UPPERCASE must use QUOTES ldquoLIKETHISrdquo
ndash Use Sum() and Count() operators with SELECT to aggregate results
For instance find out how many different polygons make each District and what is their total areas
Direct access to the GeoNode DB
There are also spatial operators that perform calculations on geometries
bull ST_GeometryType(geometry) returns the type of the geometry
bull ST_SRID(geometry) returns the spatial reference identifier number of the geometry
bull ST_X(point) returns the X ordinate
bull ST_Y(point) returns the Y ordinate
bull ST_Length(linestring) returns the length of the linestring
bull ST_StartPoint(geometry) returns the first coordinate as a point
bull ST_EndPoint(geometry) returns the last coordinate as a point
bull ST_NPoints(geometry) returns the number of coordinates in the linestring
bull ST_Area(geometry) returns the area of the polygons
bull ST_NRings(geometry) returns the number of rings (usually 1 more if there are holes)
bull ST_ExteriorRing(polygon) returns the outer ring as a linestring
bull ST_InteriorRingN(polygon integer) returns a specified interior ring as a linestring
bull ST_Perimeter(geometry) returns the length of all the rings
Direct access to the GeoNode DB
And operators that generate a text description of the geometries in different formats or vice versa
bull ST_AsText(geometry) returns WKT text
bull ST_AsEWKT(geometry) returns EWKT text
bull ST_GeomFromWKB(bytea) returns geometry
bull ST_AsBinary(geometry) returns WKB bytea
bull ST_AsEWKB(geometry) returns EWKB bytea
bull ST_GeomFromGML(text) returns geometry
bull ST_AsGML(geometry) returns GML text
bull ST_GeomFromKML(text) returns geometry
bull ST_AsKML(geometry) returns KML text
bull ST_AsGeoJSON(geometry) returns JSON text
bull ST_AsSVG(geometry) returns SVG text
Analysis with GeoNode DB We can use the DB to perform more complex spatial analysis operations
Many lsquoGIS analysisrsquo tasks are based on a combination of geometric operations between features (intersection distance) and aggregation
PostGIS provides the following geometric operators
bull ST_Contains(geometry A geometry B)
bull ST_Crosses(geometry A geometry B)
bull ST_Disjoint(geometry A geometry B)
bull ST_Distance(geometry A geometry B)
bull ST_DWithin(geometry A geometry B radius)
bull ST_Equals(geometry A geometry B)
bull ST_Intersects(geometry A geometry B)
bull ST_Overlaps(geometry A geometry B)
bull ST_Touches(geometry A geometry B)
bull ST_Within(geometry A geometry B)
Analysis with GeoNode DB spatial joins
The SQL JOIN operator allows us to add fields to a table by performing spatial tests
For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to
We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Direct access to GeoNetwork
bull Exercise ndash In the medata form type a title like lsquoOnline documentrsquo
ndash Scroll down and type a Web address as online resource
ndash Click on lsquoSave and Closersquo
Direct access to GeoNetwork
bull Exercise ndash In the medata preview click on lsquoPrivilegesrsquo
ndash Set access privileges for All and Submit
ndash Go back and search for your new lsquodatarsquo link in GeoNode
Direct access to the GeoNode DB
bull GeoNode uses a PostGIS database management system to store uploaded vector data as well as the metadata
bull To see the contents of this DB open the pgAdmin Postgres DB administration tool (this has been installed for the workshop)
Direct access to the GeoNode DB
bull Use the lsquoAdd Serverrsquo command to connect to the geonode DB
Use the password that is in the DATABASE_PASSWORD variable in this text file etcgeonodelocal_settingspy
Direct access to the GeoNode DB
Under DatabasesgeonodeSchemas publicTables we will see the tables corresponding to our vector data By right-clicking on a table we have access to some interesting tools
Direct access to the GeoNode DB
bull We can create reports like a data dictionary report
Direct access to the GeoNode DB
bull We can view the data records and filter them
Direct access to the GeoNode DB
bull We can execute SQL queries to select insert or delete records
We can apply operators to perform calculations like average count standard deviation summation etc
Direct access to the GeoNode DB
bull Exercise ndash Use simple SQL queries to extract columns from some layers
SELECT some_columns
FROM some_data_source
WHERE some_condition
Remember column names in UPPERCASE must use QUOTES ldquoLIKETHISrdquo
ndash Use Sum() and Count() operators with SELECT to aggregate results
For instance find out how many different polygons make each District and what is their total areas
Direct access to the GeoNode DB
There are also spatial operators that perform calculations on geometries
bull ST_GeometryType(geometry) returns the type of the geometry
bull ST_SRID(geometry) returns the spatial reference identifier number of the geometry
bull ST_X(point) returns the X ordinate
bull ST_Y(point) returns the Y ordinate
bull ST_Length(linestring) returns the length of the linestring
bull ST_StartPoint(geometry) returns the first coordinate as a point
bull ST_EndPoint(geometry) returns the last coordinate as a point
bull ST_NPoints(geometry) returns the number of coordinates in the linestring
bull ST_Area(geometry) returns the area of the polygons
bull ST_NRings(geometry) returns the number of rings (usually 1 more if there are holes)
bull ST_ExteriorRing(polygon) returns the outer ring as a linestring
bull ST_InteriorRingN(polygon integer) returns a specified interior ring as a linestring
bull ST_Perimeter(geometry) returns the length of all the rings
Direct access to the GeoNode DB
And operators that generate a text description of the geometries in different formats or vice versa
bull ST_AsText(geometry) returns WKT text
bull ST_AsEWKT(geometry) returns EWKT text
bull ST_GeomFromWKB(bytea) returns geometry
bull ST_AsBinary(geometry) returns WKB bytea
bull ST_AsEWKB(geometry) returns EWKB bytea
bull ST_GeomFromGML(text) returns geometry
bull ST_AsGML(geometry) returns GML text
bull ST_GeomFromKML(text) returns geometry
bull ST_AsKML(geometry) returns KML text
bull ST_AsGeoJSON(geometry) returns JSON text
bull ST_AsSVG(geometry) returns SVG text
Analysis with GeoNode DB We can use the DB to perform more complex spatial analysis operations
Many lsquoGIS analysisrsquo tasks are based on a combination of geometric operations between features (intersection distance) and aggregation
PostGIS provides the following geometric operators
bull ST_Contains(geometry A geometry B)
bull ST_Crosses(geometry A geometry B)
bull ST_Disjoint(geometry A geometry B)
bull ST_Distance(geometry A geometry B)
bull ST_DWithin(geometry A geometry B radius)
bull ST_Equals(geometry A geometry B)
bull ST_Intersects(geometry A geometry B)
bull ST_Overlaps(geometry A geometry B)
bull ST_Touches(geometry A geometry B)
bull ST_Within(geometry A geometry B)
Analysis with GeoNode DB spatial joins
The SQL JOIN operator allows us to add fields to a table by performing spatial tests
For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to
We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Direct access to GeoNetwork
bull Exercise ndash In the medata preview click on lsquoPrivilegesrsquo
ndash Set access privileges for All and Submit
ndash Go back and search for your new lsquodatarsquo link in GeoNode
Direct access to the GeoNode DB
bull GeoNode uses a PostGIS database management system to store uploaded vector data as well as the metadata
bull To see the contents of this DB open the pgAdmin Postgres DB administration tool (this has been installed for the workshop)
Direct access to the GeoNode DB
bull Use the lsquoAdd Serverrsquo command to connect to the geonode DB
Use the password that is in the DATABASE_PASSWORD variable in this text file etcgeonodelocal_settingspy
Direct access to the GeoNode DB
Under DatabasesgeonodeSchemas publicTables we will see the tables corresponding to our vector data By right-clicking on a table we have access to some interesting tools
Direct access to the GeoNode DB
bull We can create reports like a data dictionary report
Direct access to the GeoNode DB
bull We can view the data records and filter them
Direct access to the GeoNode DB
bull We can execute SQL queries to select insert or delete records
We can apply operators to perform calculations like average count standard deviation summation etc
Direct access to the GeoNode DB
bull Exercise ndash Use simple SQL queries to extract columns from some layers
SELECT some_columns
FROM some_data_source
WHERE some_condition
Remember column names in UPPERCASE must use QUOTES ldquoLIKETHISrdquo
ndash Use Sum() and Count() operators with SELECT to aggregate results
For instance find out how many different polygons make each District and what is their total areas
Direct access to the GeoNode DB
There are also spatial operators that perform calculations on geometries
bull ST_GeometryType(geometry) returns the type of the geometry
bull ST_SRID(geometry) returns the spatial reference identifier number of the geometry
bull ST_X(point) returns the X ordinate
bull ST_Y(point) returns the Y ordinate
bull ST_Length(linestring) returns the length of the linestring
bull ST_StartPoint(geometry) returns the first coordinate as a point
bull ST_EndPoint(geometry) returns the last coordinate as a point
bull ST_NPoints(geometry) returns the number of coordinates in the linestring
bull ST_Area(geometry) returns the area of the polygons
bull ST_NRings(geometry) returns the number of rings (usually 1 more if there are holes)
bull ST_ExteriorRing(polygon) returns the outer ring as a linestring
bull ST_InteriorRingN(polygon integer) returns a specified interior ring as a linestring
bull ST_Perimeter(geometry) returns the length of all the rings
Direct access to the GeoNode DB
And operators that generate a text description of the geometries in different formats or vice versa
bull ST_AsText(geometry) returns WKT text
bull ST_AsEWKT(geometry) returns EWKT text
bull ST_GeomFromWKB(bytea) returns geometry
bull ST_AsBinary(geometry) returns WKB bytea
bull ST_AsEWKB(geometry) returns EWKB bytea
bull ST_GeomFromGML(text) returns geometry
bull ST_AsGML(geometry) returns GML text
bull ST_GeomFromKML(text) returns geometry
bull ST_AsKML(geometry) returns KML text
bull ST_AsGeoJSON(geometry) returns JSON text
bull ST_AsSVG(geometry) returns SVG text
Analysis with GeoNode DB We can use the DB to perform more complex spatial analysis operations
Many lsquoGIS analysisrsquo tasks are based on a combination of geometric operations between features (intersection distance) and aggregation
PostGIS provides the following geometric operators
bull ST_Contains(geometry A geometry B)
bull ST_Crosses(geometry A geometry B)
bull ST_Disjoint(geometry A geometry B)
bull ST_Distance(geometry A geometry B)
bull ST_DWithin(geometry A geometry B radius)
bull ST_Equals(geometry A geometry B)
bull ST_Intersects(geometry A geometry B)
bull ST_Overlaps(geometry A geometry B)
bull ST_Touches(geometry A geometry B)
bull ST_Within(geometry A geometry B)
Analysis with GeoNode DB spatial joins
The SQL JOIN operator allows us to add fields to a table by performing spatial tests
For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to
We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Direct access to the GeoNode DB
bull GeoNode uses a PostGIS database management system to store uploaded vector data as well as the metadata
bull To see the contents of this DB open the pgAdmin Postgres DB administration tool (this has been installed for the workshop)
Direct access to the GeoNode DB
bull Use the lsquoAdd Serverrsquo command to connect to the geonode DB
Use the password that is in the DATABASE_PASSWORD variable in this text file etcgeonodelocal_settingspy
Direct access to the GeoNode DB
Under DatabasesgeonodeSchemas publicTables we will see the tables corresponding to our vector data By right-clicking on a table we have access to some interesting tools
Direct access to the GeoNode DB
bull We can create reports like a data dictionary report
Direct access to the GeoNode DB
bull We can view the data records and filter them
Direct access to the GeoNode DB
bull We can execute SQL queries to select insert or delete records
We can apply operators to perform calculations like average count standard deviation summation etc
Direct access to the GeoNode DB
bull Exercise ndash Use simple SQL queries to extract columns from some layers
SELECT some_columns
FROM some_data_source
WHERE some_condition
Remember column names in UPPERCASE must use QUOTES ldquoLIKETHISrdquo
ndash Use Sum() and Count() operators with SELECT to aggregate results
For instance find out how many different polygons make each District and what is their total areas
Direct access to the GeoNode DB
There are also spatial operators that perform calculations on geometries
bull ST_GeometryType(geometry) returns the type of the geometry
bull ST_SRID(geometry) returns the spatial reference identifier number of the geometry
bull ST_X(point) returns the X ordinate
bull ST_Y(point) returns the Y ordinate
bull ST_Length(linestring) returns the length of the linestring
bull ST_StartPoint(geometry) returns the first coordinate as a point
bull ST_EndPoint(geometry) returns the last coordinate as a point
bull ST_NPoints(geometry) returns the number of coordinates in the linestring
bull ST_Area(geometry) returns the area of the polygons
bull ST_NRings(geometry) returns the number of rings (usually 1 more if there are holes)
bull ST_ExteriorRing(polygon) returns the outer ring as a linestring
bull ST_InteriorRingN(polygon integer) returns a specified interior ring as a linestring
bull ST_Perimeter(geometry) returns the length of all the rings
Direct access to the GeoNode DB
And operators that generate a text description of the geometries in different formats or vice versa
bull ST_AsText(geometry) returns WKT text
bull ST_AsEWKT(geometry) returns EWKT text
bull ST_GeomFromWKB(bytea) returns geometry
bull ST_AsBinary(geometry) returns WKB bytea
bull ST_AsEWKB(geometry) returns EWKB bytea
bull ST_GeomFromGML(text) returns geometry
bull ST_AsGML(geometry) returns GML text
bull ST_GeomFromKML(text) returns geometry
bull ST_AsKML(geometry) returns KML text
bull ST_AsGeoJSON(geometry) returns JSON text
bull ST_AsSVG(geometry) returns SVG text
Analysis with GeoNode DB We can use the DB to perform more complex spatial analysis operations
Many lsquoGIS analysisrsquo tasks are based on a combination of geometric operations between features (intersection distance) and aggregation
PostGIS provides the following geometric operators
bull ST_Contains(geometry A geometry B)
bull ST_Crosses(geometry A geometry B)
bull ST_Disjoint(geometry A geometry B)
bull ST_Distance(geometry A geometry B)
bull ST_DWithin(geometry A geometry B radius)
bull ST_Equals(geometry A geometry B)
bull ST_Intersects(geometry A geometry B)
bull ST_Overlaps(geometry A geometry B)
bull ST_Touches(geometry A geometry B)
bull ST_Within(geometry A geometry B)
Analysis with GeoNode DB spatial joins
The SQL JOIN operator allows us to add fields to a table by performing spatial tests
For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to
We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Direct access to the GeoNode DB
bull Use the lsquoAdd Serverrsquo command to connect to the geonode DB
Use the password that is in the DATABASE_PASSWORD variable in this text file etcgeonodelocal_settingspy
Direct access to the GeoNode DB
Under DatabasesgeonodeSchemas publicTables we will see the tables corresponding to our vector data By right-clicking on a table we have access to some interesting tools
Direct access to the GeoNode DB
bull We can create reports like a data dictionary report
Direct access to the GeoNode DB
bull We can view the data records and filter them
Direct access to the GeoNode DB
bull We can execute SQL queries to select insert or delete records
We can apply operators to perform calculations like average count standard deviation summation etc
Direct access to the GeoNode DB
bull Exercise ndash Use simple SQL queries to extract columns from some layers
SELECT some_columns
FROM some_data_source
WHERE some_condition
Remember column names in UPPERCASE must use QUOTES ldquoLIKETHISrdquo
ndash Use Sum() and Count() operators with SELECT to aggregate results
For instance find out how many different polygons make each District and what is their total areas
Direct access to the GeoNode DB
There are also spatial operators that perform calculations on geometries
bull ST_GeometryType(geometry) returns the type of the geometry
bull ST_SRID(geometry) returns the spatial reference identifier number of the geometry
bull ST_X(point) returns the X ordinate
bull ST_Y(point) returns the Y ordinate
bull ST_Length(linestring) returns the length of the linestring
bull ST_StartPoint(geometry) returns the first coordinate as a point
bull ST_EndPoint(geometry) returns the last coordinate as a point
bull ST_NPoints(geometry) returns the number of coordinates in the linestring
bull ST_Area(geometry) returns the area of the polygons
bull ST_NRings(geometry) returns the number of rings (usually 1 more if there are holes)
bull ST_ExteriorRing(polygon) returns the outer ring as a linestring
bull ST_InteriorRingN(polygon integer) returns a specified interior ring as a linestring
bull ST_Perimeter(geometry) returns the length of all the rings
Direct access to the GeoNode DB
And operators that generate a text description of the geometries in different formats or vice versa
bull ST_AsText(geometry) returns WKT text
bull ST_AsEWKT(geometry) returns EWKT text
bull ST_GeomFromWKB(bytea) returns geometry
bull ST_AsBinary(geometry) returns WKB bytea
bull ST_AsEWKB(geometry) returns EWKB bytea
bull ST_GeomFromGML(text) returns geometry
bull ST_AsGML(geometry) returns GML text
bull ST_GeomFromKML(text) returns geometry
bull ST_AsKML(geometry) returns KML text
bull ST_AsGeoJSON(geometry) returns JSON text
bull ST_AsSVG(geometry) returns SVG text
Analysis with GeoNode DB We can use the DB to perform more complex spatial analysis operations
Many lsquoGIS analysisrsquo tasks are based on a combination of geometric operations between features (intersection distance) and aggregation
PostGIS provides the following geometric operators
bull ST_Contains(geometry A geometry B)
bull ST_Crosses(geometry A geometry B)
bull ST_Disjoint(geometry A geometry B)
bull ST_Distance(geometry A geometry B)
bull ST_DWithin(geometry A geometry B radius)
bull ST_Equals(geometry A geometry B)
bull ST_Intersects(geometry A geometry B)
bull ST_Overlaps(geometry A geometry B)
bull ST_Touches(geometry A geometry B)
bull ST_Within(geometry A geometry B)
Analysis with GeoNode DB spatial joins
The SQL JOIN operator allows us to add fields to a table by performing spatial tests
For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to
We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Direct access to the GeoNode DB
Under DatabasesgeonodeSchemas publicTables we will see the tables corresponding to our vector data By right-clicking on a table we have access to some interesting tools
Direct access to the GeoNode DB
bull We can create reports like a data dictionary report
Direct access to the GeoNode DB
bull We can view the data records and filter them
Direct access to the GeoNode DB
bull We can execute SQL queries to select insert or delete records
We can apply operators to perform calculations like average count standard deviation summation etc
Direct access to the GeoNode DB
bull Exercise ndash Use simple SQL queries to extract columns from some layers
SELECT some_columns
FROM some_data_source
WHERE some_condition
Remember column names in UPPERCASE must use QUOTES ldquoLIKETHISrdquo
ndash Use Sum() and Count() operators with SELECT to aggregate results
For instance find out how many different polygons make each District and what is their total areas
Direct access to the GeoNode DB
There are also spatial operators that perform calculations on geometries
bull ST_GeometryType(geometry) returns the type of the geometry
bull ST_SRID(geometry) returns the spatial reference identifier number of the geometry
bull ST_X(point) returns the X ordinate
bull ST_Y(point) returns the Y ordinate
bull ST_Length(linestring) returns the length of the linestring
bull ST_StartPoint(geometry) returns the first coordinate as a point
bull ST_EndPoint(geometry) returns the last coordinate as a point
bull ST_NPoints(geometry) returns the number of coordinates in the linestring
bull ST_Area(geometry) returns the area of the polygons
bull ST_NRings(geometry) returns the number of rings (usually 1 more if there are holes)
bull ST_ExteriorRing(polygon) returns the outer ring as a linestring
bull ST_InteriorRingN(polygon integer) returns a specified interior ring as a linestring
bull ST_Perimeter(geometry) returns the length of all the rings
Direct access to the GeoNode DB
And operators that generate a text description of the geometries in different formats or vice versa
bull ST_AsText(geometry) returns WKT text
bull ST_AsEWKT(geometry) returns EWKT text
bull ST_GeomFromWKB(bytea) returns geometry
bull ST_AsBinary(geometry) returns WKB bytea
bull ST_AsEWKB(geometry) returns EWKB bytea
bull ST_GeomFromGML(text) returns geometry
bull ST_AsGML(geometry) returns GML text
bull ST_GeomFromKML(text) returns geometry
bull ST_AsKML(geometry) returns KML text
bull ST_AsGeoJSON(geometry) returns JSON text
bull ST_AsSVG(geometry) returns SVG text
Analysis with GeoNode DB We can use the DB to perform more complex spatial analysis operations
Many lsquoGIS analysisrsquo tasks are based on a combination of geometric operations between features (intersection distance) and aggregation
PostGIS provides the following geometric operators
bull ST_Contains(geometry A geometry B)
bull ST_Crosses(geometry A geometry B)
bull ST_Disjoint(geometry A geometry B)
bull ST_Distance(geometry A geometry B)
bull ST_DWithin(geometry A geometry B radius)
bull ST_Equals(geometry A geometry B)
bull ST_Intersects(geometry A geometry B)
bull ST_Overlaps(geometry A geometry B)
bull ST_Touches(geometry A geometry B)
bull ST_Within(geometry A geometry B)
Analysis with GeoNode DB spatial joins
The SQL JOIN operator allows us to add fields to a table by performing spatial tests
For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to
We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Direct access to the GeoNode DB
bull We can create reports like a data dictionary report
Direct access to the GeoNode DB
bull We can view the data records and filter them
Direct access to the GeoNode DB
bull We can execute SQL queries to select insert or delete records
We can apply operators to perform calculations like average count standard deviation summation etc
Direct access to the GeoNode DB
bull Exercise ndash Use simple SQL queries to extract columns from some layers
SELECT some_columns
FROM some_data_source
WHERE some_condition
Remember column names in UPPERCASE must use QUOTES ldquoLIKETHISrdquo
ndash Use Sum() and Count() operators with SELECT to aggregate results
For instance find out how many different polygons make each District and what is their total areas
Direct access to the GeoNode DB
There are also spatial operators that perform calculations on geometries
bull ST_GeometryType(geometry) returns the type of the geometry
bull ST_SRID(geometry) returns the spatial reference identifier number of the geometry
bull ST_X(point) returns the X ordinate
bull ST_Y(point) returns the Y ordinate
bull ST_Length(linestring) returns the length of the linestring
bull ST_StartPoint(geometry) returns the first coordinate as a point
bull ST_EndPoint(geometry) returns the last coordinate as a point
bull ST_NPoints(geometry) returns the number of coordinates in the linestring
bull ST_Area(geometry) returns the area of the polygons
bull ST_NRings(geometry) returns the number of rings (usually 1 more if there are holes)
bull ST_ExteriorRing(polygon) returns the outer ring as a linestring
bull ST_InteriorRingN(polygon integer) returns a specified interior ring as a linestring
bull ST_Perimeter(geometry) returns the length of all the rings
Direct access to the GeoNode DB
And operators that generate a text description of the geometries in different formats or vice versa
bull ST_AsText(geometry) returns WKT text
bull ST_AsEWKT(geometry) returns EWKT text
bull ST_GeomFromWKB(bytea) returns geometry
bull ST_AsBinary(geometry) returns WKB bytea
bull ST_AsEWKB(geometry) returns EWKB bytea
bull ST_GeomFromGML(text) returns geometry
bull ST_AsGML(geometry) returns GML text
bull ST_GeomFromKML(text) returns geometry
bull ST_AsKML(geometry) returns KML text
bull ST_AsGeoJSON(geometry) returns JSON text
bull ST_AsSVG(geometry) returns SVG text
Analysis with GeoNode DB We can use the DB to perform more complex spatial analysis operations
Many lsquoGIS analysisrsquo tasks are based on a combination of geometric operations between features (intersection distance) and aggregation
PostGIS provides the following geometric operators
bull ST_Contains(geometry A geometry B)
bull ST_Crosses(geometry A geometry B)
bull ST_Disjoint(geometry A geometry B)
bull ST_Distance(geometry A geometry B)
bull ST_DWithin(geometry A geometry B radius)
bull ST_Equals(geometry A geometry B)
bull ST_Intersects(geometry A geometry B)
bull ST_Overlaps(geometry A geometry B)
bull ST_Touches(geometry A geometry B)
bull ST_Within(geometry A geometry B)
Analysis with GeoNode DB spatial joins
The SQL JOIN operator allows us to add fields to a table by performing spatial tests
For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to
We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Direct access to the GeoNode DB
bull We can view the data records and filter them
Direct access to the GeoNode DB
bull We can execute SQL queries to select insert or delete records
We can apply operators to perform calculations like average count standard deviation summation etc
Direct access to the GeoNode DB
bull Exercise ndash Use simple SQL queries to extract columns from some layers
SELECT some_columns
FROM some_data_source
WHERE some_condition
Remember column names in UPPERCASE must use QUOTES ldquoLIKETHISrdquo
ndash Use Sum() and Count() operators with SELECT to aggregate results
For instance find out how many different polygons make each District and what is their total areas
Direct access to the GeoNode DB
There are also spatial operators that perform calculations on geometries
bull ST_GeometryType(geometry) returns the type of the geometry
bull ST_SRID(geometry) returns the spatial reference identifier number of the geometry
bull ST_X(point) returns the X ordinate
bull ST_Y(point) returns the Y ordinate
bull ST_Length(linestring) returns the length of the linestring
bull ST_StartPoint(geometry) returns the first coordinate as a point
bull ST_EndPoint(geometry) returns the last coordinate as a point
bull ST_NPoints(geometry) returns the number of coordinates in the linestring
bull ST_Area(geometry) returns the area of the polygons
bull ST_NRings(geometry) returns the number of rings (usually 1 more if there are holes)
bull ST_ExteriorRing(polygon) returns the outer ring as a linestring
bull ST_InteriorRingN(polygon integer) returns a specified interior ring as a linestring
bull ST_Perimeter(geometry) returns the length of all the rings
Direct access to the GeoNode DB
And operators that generate a text description of the geometries in different formats or vice versa
bull ST_AsText(geometry) returns WKT text
bull ST_AsEWKT(geometry) returns EWKT text
bull ST_GeomFromWKB(bytea) returns geometry
bull ST_AsBinary(geometry) returns WKB bytea
bull ST_AsEWKB(geometry) returns EWKB bytea
bull ST_GeomFromGML(text) returns geometry
bull ST_AsGML(geometry) returns GML text
bull ST_GeomFromKML(text) returns geometry
bull ST_AsKML(geometry) returns KML text
bull ST_AsGeoJSON(geometry) returns JSON text
bull ST_AsSVG(geometry) returns SVG text
Analysis with GeoNode DB We can use the DB to perform more complex spatial analysis operations
Many lsquoGIS analysisrsquo tasks are based on a combination of geometric operations between features (intersection distance) and aggregation
PostGIS provides the following geometric operators
bull ST_Contains(geometry A geometry B)
bull ST_Crosses(geometry A geometry B)
bull ST_Disjoint(geometry A geometry B)
bull ST_Distance(geometry A geometry B)
bull ST_DWithin(geometry A geometry B radius)
bull ST_Equals(geometry A geometry B)
bull ST_Intersects(geometry A geometry B)
bull ST_Overlaps(geometry A geometry B)
bull ST_Touches(geometry A geometry B)
bull ST_Within(geometry A geometry B)
Analysis with GeoNode DB spatial joins
The SQL JOIN operator allows us to add fields to a table by performing spatial tests
For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to
We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Direct access to the GeoNode DB
bull We can execute SQL queries to select insert or delete records
We can apply operators to perform calculations like average count standard deviation summation etc
Direct access to the GeoNode DB
bull Exercise ndash Use simple SQL queries to extract columns from some layers
SELECT some_columns
FROM some_data_source
WHERE some_condition
Remember column names in UPPERCASE must use QUOTES ldquoLIKETHISrdquo
ndash Use Sum() and Count() operators with SELECT to aggregate results
For instance find out how many different polygons make each District and what is their total areas
Direct access to the GeoNode DB
There are also spatial operators that perform calculations on geometries
bull ST_GeometryType(geometry) returns the type of the geometry
bull ST_SRID(geometry) returns the spatial reference identifier number of the geometry
bull ST_X(point) returns the X ordinate
bull ST_Y(point) returns the Y ordinate
bull ST_Length(linestring) returns the length of the linestring
bull ST_StartPoint(geometry) returns the first coordinate as a point
bull ST_EndPoint(geometry) returns the last coordinate as a point
bull ST_NPoints(geometry) returns the number of coordinates in the linestring
bull ST_Area(geometry) returns the area of the polygons
bull ST_NRings(geometry) returns the number of rings (usually 1 more if there are holes)
bull ST_ExteriorRing(polygon) returns the outer ring as a linestring
bull ST_InteriorRingN(polygon integer) returns a specified interior ring as a linestring
bull ST_Perimeter(geometry) returns the length of all the rings
Direct access to the GeoNode DB
And operators that generate a text description of the geometries in different formats or vice versa
bull ST_AsText(geometry) returns WKT text
bull ST_AsEWKT(geometry) returns EWKT text
bull ST_GeomFromWKB(bytea) returns geometry
bull ST_AsBinary(geometry) returns WKB bytea
bull ST_AsEWKB(geometry) returns EWKB bytea
bull ST_GeomFromGML(text) returns geometry
bull ST_AsGML(geometry) returns GML text
bull ST_GeomFromKML(text) returns geometry
bull ST_AsKML(geometry) returns KML text
bull ST_AsGeoJSON(geometry) returns JSON text
bull ST_AsSVG(geometry) returns SVG text
Analysis with GeoNode DB We can use the DB to perform more complex spatial analysis operations
Many lsquoGIS analysisrsquo tasks are based on a combination of geometric operations between features (intersection distance) and aggregation
PostGIS provides the following geometric operators
bull ST_Contains(geometry A geometry B)
bull ST_Crosses(geometry A geometry B)
bull ST_Disjoint(geometry A geometry B)
bull ST_Distance(geometry A geometry B)
bull ST_DWithin(geometry A geometry B radius)
bull ST_Equals(geometry A geometry B)
bull ST_Intersects(geometry A geometry B)
bull ST_Overlaps(geometry A geometry B)
bull ST_Touches(geometry A geometry B)
bull ST_Within(geometry A geometry B)
Analysis with GeoNode DB spatial joins
The SQL JOIN operator allows us to add fields to a table by performing spatial tests
For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to
We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Direct access to the GeoNode DB
bull Exercise ndash Use simple SQL queries to extract columns from some layers
SELECT some_columns
FROM some_data_source
WHERE some_condition
Remember column names in UPPERCASE must use QUOTES ldquoLIKETHISrdquo
ndash Use Sum() and Count() operators with SELECT to aggregate results
For instance find out how many different polygons make each District and what is their total areas
Direct access to the GeoNode DB
There are also spatial operators that perform calculations on geometries
bull ST_GeometryType(geometry) returns the type of the geometry
bull ST_SRID(geometry) returns the spatial reference identifier number of the geometry
bull ST_X(point) returns the X ordinate
bull ST_Y(point) returns the Y ordinate
bull ST_Length(linestring) returns the length of the linestring
bull ST_StartPoint(geometry) returns the first coordinate as a point
bull ST_EndPoint(geometry) returns the last coordinate as a point
bull ST_NPoints(geometry) returns the number of coordinates in the linestring
bull ST_Area(geometry) returns the area of the polygons
bull ST_NRings(geometry) returns the number of rings (usually 1 more if there are holes)
bull ST_ExteriorRing(polygon) returns the outer ring as a linestring
bull ST_InteriorRingN(polygon integer) returns a specified interior ring as a linestring
bull ST_Perimeter(geometry) returns the length of all the rings
Direct access to the GeoNode DB
And operators that generate a text description of the geometries in different formats or vice versa
bull ST_AsText(geometry) returns WKT text
bull ST_AsEWKT(geometry) returns EWKT text
bull ST_GeomFromWKB(bytea) returns geometry
bull ST_AsBinary(geometry) returns WKB bytea
bull ST_AsEWKB(geometry) returns EWKB bytea
bull ST_GeomFromGML(text) returns geometry
bull ST_AsGML(geometry) returns GML text
bull ST_GeomFromKML(text) returns geometry
bull ST_AsKML(geometry) returns KML text
bull ST_AsGeoJSON(geometry) returns JSON text
bull ST_AsSVG(geometry) returns SVG text
Analysis with GeoNode DB We can use the DB to perform more complex spatial analysis operations
Many lsquoGIS analysisrsquo tasks are based on a combination of geometric operations between features (intersection distance) and aggregation
PostGIS provides the following geometric operators
bull ST_Contains(geometry A geometry B)
bull ST_Crosses(geometry A geometry B)
bull ST_Disjoint(geometry A geometry B)
bull ST_Distance(geometry A geometry B)
bull ST_DWithin(geometry A geometry B radius)
bull ST_Equals(geometry A geometry B)
bull ST_Intersects(geometry A geometry B)
bull ST_Overlaps(geometry A geometry B)
bull ST_Touches(geometry A geometry B)
bull ST_Within(geometry A geometry B)
Analysis with GeoNode DB spatial joins
The SQL JOIN operator allows us to add fields to a table by performing spatial tests
For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to
We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Direct access to the GeoNode DB
There are also spatial operators that perform calculations on geometries
bull ST_GeometryType(geometry) returns the type of the geometry
bull ST_SRID(geometry) returns the spatial reference identifier number of the geometry
bull ST_X(point) returns the X ordinate
bull ST_Y(point) returns the Y ordinate
bull ST_Length(linestring) returns the length of the linestring
bull ST_StartPoint(geometry) returns the first coordinate as a point
bull ST_EndPoint(geometry) returns the last coordinate as a point
bull ST_NPoints(geometry) returns the number of coordinates in the linestring
bull ST_Area(geometry) returns the area of the polygons
bull ST_NRings(geometry) returns the number of rings (usually 1 more if there are holes)
bull ST_ExteriorRing(polygon) returns the outer ring as a linestring
bull ST_InteriorRingN(polygon integer) returns a specified interior ring as a linestring
bull ST_Perimeter(geometry) returns the length of all the rings
Direct access to the GeoNode DB
And operators that generate a text description of the geometries in different formats or vice versa
bull ST_AsText(geometry) returns WKT text
bull ST_AsEWKT(geometry) returns EWKT text
bull ST_GeomFromWKB(bytea) returns geometry
bull ST_AsBinary(geometry) returns WKB bytea
bull ST_AsEWKB(geometry) returns EWKB bytea
bull ST_GeomFromGML(text) returns geometry
bull ST_AsGML(geometry) returns GML text
bull ST_GeomFromKML(text) returns geometry
bull ST_AsKML(geometry) returns KML text
bull ST_AsGeoJSON(geometry) returns JSON text
bull ST_AsSVG(geometry) returns SVG text
Analysis with GeoNode DB We can use the DB to perform more complex spatial analysis operations
Many lsquoGIS analysisrsquo tasks are based on a combination of geometric operations between features (intersection distance) and aggregation
PostGIS provides the following geometric operators
bull ST_Contains(geometry A geometry B)
bull ST_Crosses(geometry A geometry B)
bull ST_Disjoint(geometry A geometry B)
bull ST_Distance(geometry A geometry B)
bull ST_DWithin(geometry A geometry B radius)
bull ST_Equals(geometry A geometry B)
bull ST_Intersects(geometry A geometry B)
bull ST_Overlaps(geometry A geometry B)
bull ST_Touches(geometry A geometry B)
bull ST_Within(geometry A geometry B)
Analysis with GeoNode DB spatial joins
The SQL JOIN operator allows us to add fields to a table by performing spatial tests
For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to
We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Direct access to the GeoNode DB
And operators that generate a text description of the geometries in different formats or vice versa
bull ST_AsText(geometry) returns WKT text
bull ST_AsEWKT(geometry) returns EWKT text
bull ST_GeomFromWKB(bytea) returns geometry
bull ST_AsBinary(geometry) returns WKB bytea
bull ST_AsEWKB(geometry) returns EWKB bytea
bull ST_GeomFromGML(text) returns geometry
bull ST_AsGML(geometry) returns GML text
bull ST_GeomFromKML(text) returns geometry
bull ST_AsKML(geometry) returns KML text
bull ST_AsGeoJSON(geometry) returns JSON text
bull ST_AsSVG(geometry) returns SVG text
Analysis with GeoNode DB We can use the DB to perform more complex spatial analysis operations
Many lsquoGIS analysisrsquo tasks are based on a combination of geometric operations between features (intersection distance) and aggregation
PostGIS provides the following geometric operators
bull ST_Contains(geometry A geometry B)
bull ST_Crosses(geometry A geometry B)
bull ST_Disjoint(geometry A geometry B)
bull ST_Distance(geometry A geometry B)
bull ST_DWithin(geometry A geometry B radius)
bull ST_Equals(geometry A geometry B)
bull ST_Intersects(geometry A geometry B)
bull ST_Overlaps(geometry A geometry B)
bull ST_Touches(geometry A geometry B)
bull ST_Within(geometry A geometry B)
Analysis with GeoNode DB spatial joins
The SQL JOIN operator allows us to add fields to a table by performing spatial tests
For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to
We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Analysis with GeoNode DB We can use the DB to perform more complex spatial analysis operations
Many lsquoGIS analysisrsquo tasks are based on a combination of geometric operations between features (intersection distance) and aggregation
PostGIS provides the following geometric operators
bull ST_Contains(geometry A geometry B)
bull ST_Crosses(geometry A geometry B)
bull ST_Disjoint(geometry A geometry B)
bull ST_Distance(geometry A geometry B)
bull ST_DWithin(geometry A geometry B radius)
bull ST_Equals(geometry A geometry B)
bull ST_Intersects(geometry A geometry B)
bull ST_Overlaps(geometry A geometry B)
bull ST_Touches(geometry A geometry B)
bull ST_Within(geometry A geometry B)
Analysis with GeoNode DB spatial joins
The SQL JOIN operator allows us to add fields to a table by performing spatial tests
For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to
We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Analysis with GeoNode DB spatial joins
The SQL JOIN operator allows us to add fields to a table by performing spatial tests
For instance the lsquocitrus_farmingrsquo table contains locations of citrus farms but it does not say directly to which district they belong to
We can join spatially with the table lsquodistrictsrsquo by using the lsquoST_Containsrsquo test to find out the district for each farm
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Analysis with GeoNode DB spatial joins
We can also create new attributes by aggregating the information of all the features in the joined dataset
Use JOIN + GROUP BY and an aggregation operator (Avg Sumhellip)
For instance we want to calculate an aggregated biodiversity indicator for each biological corridor by using values from the biodiversity grid dataset
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
GIS Analysis with GeoNode DB
bull Exercise ndash Use spatial joins to add attributes to point layers (for instance in
which district or geological zone is an industry)
ndash Find out which biological corridors are crossed by a certain river
ndash You can get more ideas from
httpworkshopsopengeoorgpostgis-introjoins_exerciseshtml
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Saving your results
bull We can export the result of a SQL Query to a text file which can be loaded into a spreadsheet
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
bull We can also use the results of a query to create a new table
bull Or create a view which gets updated when the original table changes This is how we can easily add more columns with calculated values
CREATE TABLE table_name AS
SELECT hellip FROM hellip WHERE hellip
ALTER TABLE table_name ADD PRIMARY KEY (fid)
CREATE VIEW view_name AS
SELECT hellip FROM hellip WHERE hellip
Saving your results
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
bull If we want to create a SPATIAL table that we can reuse in GeoNode we also need to register the table as spatial
SELECT
Populate_Geometry_Columns(lsquotable_namersquoregclass)
Saving your results
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
bull Exercise ndash Create a spatial table by copying an existing one and adding some
columns
ndash Create the new layer in GeoNode from the lsquopostgres_importsrsquo store
ndash Run the lsquoupdatelayersrsquo command
ndash Display the new layer in a map and use the new column to style it
CREATE TABLE districts_area AS
SELECT fid the_geom ST_Area(the_geom)
FROM districts
-- Define a primary key
ALTER TABLE districts_area ADD PRIMARY KEY (fid)
-- Spatially enable it
SELECT
Populate_Geometry_Columns(districts_arearegclass)
Saving your results
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Projecting data with GeoNode DB
bull If we upload a PRJ file with our data Geonode will assign a SRID (Spatial Reference ID) in the PostGIS database Take a look at the geometry_columns table This ID is also called the EPSG code a standard way to refer to Spatial References See httpspatialreferenceorg
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Projecting data with GeoNode DB bull Another internal PostGIS table (spatial_ref_sys) describes in text format
each SRID
bull We can see there two standard text representations of each Spatial Reference (srtext is the same description used in prj files)
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Projecting data with GeoNode DB
bull Using different projections for different layers in a map should not be a problem since ndash Web servers project the data to the SRID the client requests
ndash GIS clients can project data to the view SRID on-the-fly
bull However sometimes it is convenient for efficiency to project data to a different SRID or we must do it because the server or GIS does not perform correctly the conversion (for instance when doing some operation between layers of different SRID)
bull For example imagine we want to change from UTM 16N on NAD27 datum to use a different datum like WGS84 (differences in datum often account for annoying displacements between layers in maps)
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Projecting data with GeoNode DB
bull Transforming geometries to a different spatial reference in PostGIS SQL is as simple as
bull Wersquoll see we can do the same as easily in a GIS
SELECT ST_Transform(the_geom DestinationSRID)
FROM original_table
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Projecting data with GeoNode DB
bull With a query we can find all SRIDs based on UTM zone 16N
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Projecting data with GeoNode DB
bull We can now transform the Belize base map data from UTM16N-NAD27 (26716) to UTM16N-WGS84 (32616)
bull In this case differences are about 2 meters
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Direct access to the GeoNode DB
bull To learn more about working with PostGIS
httpworkshopsopengeoorgpostgis-intro
httppostgisrefractionsnetdocumentation
manual-14ch04html
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Direct access to GeoNode from GIS
bull Desktop or Web GIS tools include powerful display and analysis functionality that we can use with our GeoNode
bull From a GIS we can connect to GeoNodersquos
ndash PostGIS DB
ndash Map Service (WMS) published by GeoServer
ndash Feature Service (WFS) and Coverage Service (WCS) as published by GeoServer
ndash Catalog Service (CSW) as published by Geonetwork but in practice this is not well solved yet
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Direct access to GeoNode from GIS
GeoServer
GeoNetwork
PostGIS DB
GeoNode
GIS
CSW
WFS WFS WCS
SLD
(Editing) WFS-T
Web front end
(Editing geoprocessing)
File download
File upload
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Direct access to GeoNode from GIS
bull For the workshop we will use Quantum GIS an open-source GIS available in Linux and Windows
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Access to Map Service
bull To connect to the GeoNode WMS we use the URL httpmygeonodeURLgeoserverwms Click on lsquoNewrsquo to create a new connection
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Access to Map Service
bull Click on lsquoConnectrsquo bull The layers published in the GeoNode map service will be
displayed (even layers that are not added to a map) bull Select one or more layers (belize_land_degradation) bull Click on lsquoAddrsquo and then lsquoClosersquo
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Access to Map Service
bull The selected layer will be displayed using images from the map server
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Access to Map Service
bull We can identify features in the map (internally uses GetFeatureInfo request)
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Access to Map Service from ArcGIS
bull It is straightforward to add a WMS server to ArcGIS by using the lsquoAdd Datarsquo tool
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Access to Map Service from ArcGIS
bull After adding the WMS service to the table of contents in ArcMap we can select specific layers to draw
bull However the Identify feature does not seem to work well
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Access to Feature Service bull By accessing WFS services we can access data with full geometry
and attribute information as if it were on a local file or database bull Because data might come in a different spatial reference in some
GIS we need to make sure the on-the-fly projection is enabled bull In qGIS we use the FileProject Properties dialog and for future
projects in the SettingsOptions dialog
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Access to Feature Service
bull To connect to the GeoNode WFS we use the URL httpmygeonodeURLgeoserverwfs
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Access to Feature Service
bull Click on lsquoConnectrsquo
bull The layers published in the GeoNode feature service will be displayed
bull Select one layer (belize_land_degradation)
bull Click on lsquoApplyrsquo and then lsquoClosersquo
Note that we didnrsquot change the Coordinate Reference System from the original
which is different from the current qGIS project view
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Access to Feature Service
bull The selected layer will be displayed using features downloaded from the server
bull We can style them and use in GIS analysis
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Access to Feature Service
bull We can also edit the WFS feature layer
bull Use the lsquoLayerToggle Editingrsquo command to enable editing
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Access to Feature Service
bull Use edit tools (try Node Tool or Reshape in qGIS) and the table of attributes (select the feature to edit) to change the geometry and field values
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Access to Feature Service
bull We can also edit attribute values in the table view and add or remove columns
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Access to Feature Service
bull If the GIS or Web editor supports WFS-T you can save changes directly to the server
bull Otherwise you can save the results to a file or to the PostGIS database
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Access to Map and Feature Services
bull Exercise
Create a map in qGIS by combining several WMS and WFS layers from GeoServer (you can style the WFS but not the WMS)
Save the project
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Access to Feature Service from ArcGIS
bull It is necessary to have the Data Interoperability extension installed (not a free add-on)
bull From ArcCatalog or the Catalog tab in ArcMap add a new Interoperability Connection select the WFS format and enter the GeoNode WFS URL
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Access to Feature Service from ArcGIS
bull Click the lsquoParametersrsquo button and open the Feature Types selector
bull A list of the layers from the WFS server should appear () and you can select to import one or more
() Does not seem to work with the Geoserver WFS try the URL showed here
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Access to Feature Service from ArcGIS
bull Once the interoperability connection to the WFS server is created you can add data from the connection to ArcMap
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Access to Feature Service from ArcGIS
bull You can also use it in geoprocessing (just drag the layer from the legend to the input field for a feature class
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Access to PostGIS DB from GIS
bull Desktop GIS systems can also connect directly to the spatial DB and use it as a data source
bull We create a connection just like we did from pgAdmin
bull We need to use the password in etcgeonodelocal_settingspy
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Access to PostGIS DB from GIS bull Once the connection is added click lsquoConnectrsquo to get the list of
available layers bull Select one layer and click lsquoAddrsquo You can repeat this for other layers
and click lsquoClosersquo when you are done bull We can also specify a definition query for each layer with lsquoBuild
queryrsquo (see next slide)
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Access to PostGIS DB from GIS
bull With the query we select only features complying with some condition
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Editing PostGIS DB from GIS
bull We can toggle editing and use the same tools as with WFS or other feature layers
bull This time we will be able to Save Edits in the same DB table or create a new one All services based on this layer will be affected
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Editing PostGIS DB from GIS
bull Exercise
Upload in qGIS some feature layers from the PostGIS DB
Edit them
Save the results to the DB
Go to GeoNode and see that the layers and maps reflect the changes
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Projecting data in GIS
In addition to displaying data in a different CRS (on-the-fly projection) GIS provide easy ways to save a new version of the data with the geometry transformed to a different CRS
In qGIS use the lsquoSave Asrsquo command for vector data and the lsquoRasterProjectionsWarprsquo for raster data
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Projecting data in GIS
Feature and raster projection is a common feature in other GIS like ArcGIS
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Projecting data in GIS datum shift
bull A common problem is that when the datum is changed in the projection some side shift may be introduced This is called lsquodatum shiftrsquo
bull The reason behind this problem is that the datum is only an ellipsoidal approximation to the Earthrsquos lsquozero levelrsquo shape which actually changes in different locations
bull As a first approximation a combination of translation and rotations -7 parameters ()- can reduce the error to a few meters
bull For yet better precision datum transformations need a shift grid specification provided for each territory usually a specific country or state often in NTv2 format (gsb file)
() See httpearth-infongamilGandGcoordsysonlinedatum CountryNorthAmericaTablehtmlNASL
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Projecting data in GIS datum shift
bull Once you have the grid shift file in ArcGIS you can define a custom transformation
bull This will later be selected when projecting data
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Projecting data in GIS datum shift
bull Most open-source software tools use the proj4 library which also supports shift grids
bull The grid file has to be installed in the library configuration folder and the definition of the CRS updated to specify this grid (the same CRS can have multiple grids and one is selected depending on the location of the geometries to transform)
bull qGIS as well as PostGIS keep the proj4 configuration in a database table
More details in httpprojmaptoolsorggen_parmshtml
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Projecting data in GIS datum shift bull For NAD27 to NAD83 (very close to WGS84) there is also the option to use
the NADCON utility from the USA National Geodetic Survey ()
bull A shapefile to shapefile conversion tool can be found here httpwwwcscnoaagovproductsshapenadconShapeNADCONexeZIP
bull However the tool operates on latitudelongitude values only so the process to convert UTM Zone 16N from one datum to another would be
26716 (UTM 16N + NAD27)
4267 (GCS NAD27)
4152 (GCS NAD83)
26916 (UTM 16N + NAD83)
Project Project
NADCON
() However after testing it seems the NADCON tool does not include the grids for the Caribbean region
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Accessing PostGIS DB from ArcGIS
bull ArcGIS support PostGIS spatial databases via ArcSDE
bull It is necessary to install the ArcSDE PostgreSQL AddOn
bull In theory this will allow us to access the GeoNode spatial data from ArcGIS
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Access to Catalog Service from GIS
bull The qGIS CSW Client plugin has been installed for the workshop
bull Use the lsquoWebCSW Clientrsquo menu command to open the dialog
bull Create a new connection to the Geonode URL
httpgeonodeURLgeonetworksrvencsw
bull Click on lsquoServer Inforsquo to verify the connection
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Access to Catalog Service from GIS
bull Now we can search for metadata records in the catalog
bull However we can not do more than read the list of records because GeoNode does not publish the WMS access information in the metadata
In the administration module we will see how to add the WMS linkage in the metadata
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Access to Catalog Service from ArcGIS
bull ESRI provides a free CSW Client add-on for ArcGIS downloadable from
httpsourceforgenetprojectsgeoportalfilesApplicationsCSWClientsv122
bull New catalog connections can be created using the Geonetwork profiles
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Access to Catalog Service from ArcGIS
bull However these profiles are not completely correct since they fail to get all records and do not allow to add the found WMS services to ArcMap (the linkage is not correctly detected)
bull For instance the CSW client found some records from the FAO catalog but none from the GeoNode catalog
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Using SLD styles from GIS
bull Many times we have already created styles (legends symbology) in our GIS projects and would like to use the same ones in GeoNode
bull The idea is that we can export SLDs from the GIS and upload them in GeoNode but in practice we will find a lot of issues of incompatibility between SLDs
bull For instance letrsquos create a style for a layer (Aquaculture) in qGIS ndash Add the SHP file
ndash Go to layer properties with the right-click menu
ndash Select the ldquoCategorizedrdquo type
ndash Create a color ramp
ndash Classify
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Using SLD styles from GIS
bull Now use the lsquoSave Stylersquo button in the qGIS Layer Properties dialog to create a SLD file
bull Open the sld file in a text editor like gedit
bull Now go to GeoServerrsquos style editor and inspect the default SLD created for the same feature layer You will see a lot of differences
bull For starters the Version number is not the same (100 in GeoServer 110 from qGIS) so probably some features are not supported
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Using SLD styles from GIS bull The GeoServer SLD uses a different
namespace some new properties (Description for rules) or different ones (SvgParameter instead of CSSParameter)
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Using SLD styles from GIS
bull Try to reconcile these differences (in an external XML-aware editor like lsquogeditrsquo is easier) and use the lsquoValidatersquo button in the GeoServer editor to verify the syntax
bull Even so you may still find that the style does not work correctly in GeoNode (avoid empty lines in the XMLhellip)
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Using SLD styles from GIS
bull Exercise
Open the GeoServer style editor for the elevation data
Create a different elevation color ramp for the same node in qGIS
Copy the rules as exported from qGIS into the GeoServer SLD code to modify the ramp
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Using SLD styles from GIS
bull For ArcGIS the following options are recommended by GeoNode ndash ArcMap2SLD (Free Open Source)
httpappsgeoinformfh-mainzdearcmap2sldArcMap2SLDConverter_Enghtm
ndash Arc2Earth (Free Community Edition)
httpwwwarc2earthcomcommunityedition
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Processing data in GIS
bull Data can be processed ndash Before uploading to GeoNode (data preparation) by using shapefiles
or original DB
ndash After uploading to GeoNode (data exploitation) by connecting to the PostGIS DB or WFS or by downloading the data
bull Some typical processes are
ndash Importing data point with X-Y values in a table
ndash Georeferencing data
ndash Limit the data to an area of interest
ndash Table and spatial joins
ndash Cleaning topology
bull Most of these can be done directly with PostGIS and SQL using spatial operators
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Importing points from XY table
bull In qGIS use the ldquoAdd Delimited Text Layerrdquo httpqgisspatialthoughtscom201201importing-spreadsheets-or-csv-files-tohtml
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Importing points from XY table
bull In ArcGIS use the Add XY Data command httpsupportesricomesknowledgebasetecharticlesdetail27589
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Limiting the data area
bull Some geoprocessing systems let you use Area of Interest options but if you have to do it manuallyhellip
bull Letrsquos say you need to work only on those parts of the lsquobiol_corridorsrsquo polygon layer that lie within the lsquoOrange Walkrsquo district
bull You can select the Orange Walk district (ViewSelect) and then use qGIS lsquoVectorGeoprocessingIntersectrsquo tool
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Limiting the data area
bull A new layer with the clipped features is now created
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Georeferencing data table join
bull If we have plain tabular data and a separate dataset with the matching geometries we perform a table join based on a common field (typically some kind of unique identifier for the features like a normalized name or code)
bull In qGIS open the table as dbf file using the lsquoAdd Vector Layerrsquo tool A table will show up in the table of contents
bull For instance we load the lsquocorridorDatadbfrsquo file with additional information about our biological corridors
Note This tool can be used to convert from Excel to DBF httpwwwwhitetowncomxls2dbf
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
bull Then we open the Layer Properties for the biological corridors layer and use the Joins tab to join with the lsquocorridorDatarsquo table resulting in additional columns added to the attribute table of the vector layer
Georeferencing data table join
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
bull Exercise
Join two shapefiles by attribute
Load the first shapefile as usual
Load the DBF from the second shapefile
Join the tables by using a common attribute as explained
Georeferencing data table join
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Raster georeferencing
bull Another common data input comes from rasters which have been scanned or photographs with no spatial reference
bull We need to assign them coordinates which allow us to superimpose the raster on a map and maybe use it for analysis
bull In qGIS use the PluginsManage dialog to enable the Georeferencer plugin based on the GDAL library for raster data
bull Then select the Georeferencer tool in the raster menu
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Raster georeferencing
bull We upload a raster to the Georeferencer which displays it in its own window
bull Then we need to create at least 4 control points to define the correspondence between the raster and the reference data
Open the ldquoBelize_islandspngrdquo image assigning the EPSG 26716 SRS
Then click on a distinctive point like a coast feature (you can use the Georeferencer navigation tools to get closer and pan around)
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Raster georeferencing
When you click on the image a dialog will appear to enter the reference coordinates for that point
Click on the lsquoFrom map canvasrsquo button to use the Belize base map or other layers as reference by clicking on it The coordinate input fields will get values and you can click lsquoOkrsquo
Enter at least other 3 reference points in the same way better if they are spread in different directions
You can remove points from the GPS table later on
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Raster georeferencing
bull Once you have at least 4 points click on ldquoStart geoferencingrdquo and choose the transformation options
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Raster georeferencing
bull A new GeoTIFF file with the CRS and coordinate information will be generated
bull You can check the georeferencing correctness in the map view and upload the GeoTIFF in GeoNode if the result is correct
More details in httpcoursesumassedunrc297sPDFsLab_Georeferencing_with_QGISpdf
httpglaikitorg20110327image-georeferencing-with-qgis
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Raster georeferencing
bull Exercise
Capture a view in Google Maps or any other map source
Save the capture to an image file
Georeference the image and upload it in GeoNode
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Using GPS data
bull GPS-enabled devices are becoming a quick and easy way to input data in GIS and SDIs
bull The GPS eXchange Format (GPX) is an open format that can be downloaded or uploaded fromo GPS-enabled devices
bull A GPX file contains points (waypoints) tracks (lines) and routes (future reference points and turns) using latitudelongitude coordinates on the WGS84 datum
bull It is widely supported in many systems and software including the Open Street Map project
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Using GPS data
bull qGIS provides a convenient plugin to import GPX files It can also be used to download or upload the files to a device
bull First enable the plugin in the PluginsManage dialog then go to the lsquoVectorGPSGPS Toolsrsquo command to pop up the import dialog
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Using GPS data
bull Exercise
Import some of the sample GPX files which may contain only points or tracks
Do not forget to enable the lsquoon the fly projectionrsquo
Use the OpenLayers plugin to set a nice background map (you may need to reset the project CRS to 26716 after this)
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Using GPS data
bull Because GPX data comes in WGS84 datum it may be affected by the datum shift issues discussed before
bull Once imported you can process GPX data as any other feature and save it to a shape file and upload in GeoNode
bull As we will see mobile data can be an important source for lsquocrowdsourcedrsquo data acquisition as in the OpenStreetMap project
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Topology
bull A very common problem with GIS data is the existence of topological errors (features intersect that should not features do not connect that shouldhellip)
bull These problems affect display and analysis results that some times may turn invalid or incorrect
bull qGIS provides some tools to
detect problems and change the offending geometries
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Topology
bull Before or after we upload a vector dataset to GeoNode we can detect topology problems
bull For instance use the lsquoCheck geometry validityrsquo tool on the lsquoroads_2010rsquo PostGIS layer
bull It will detect some cases of line self-intersection and duplicated nodes
bull These can be corrected by editing
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Topology
bull We can do the same for polygon data (eg land_degradation) where we find more serious problems
bull However it is very costly to fix these problems on a massive scale so we need automated tools for repairing topology
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Topology with qGISGRASS
bull Some GIS systems like the old ArcInfo and GRASS are internally built to always be topologically correct but this restriction has been relaxed in more lsquoCAD-likersquo GIS
bull qGIS includes an interface to GRASS that can be used to clean up topological defects in vector data
(Enable the GRASS plugin in lsquoPluginsManagersquo)
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Topology with qGISGRASS
bull GRASS uses a hierarchical folder structure starting with a folder (GRASS database or gisdbase) that we must create before using the plugin
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Topology with qGISGRASS
bull After creating manually the GRASS database folder use the ldquoNew mapsetrdquo command to launch the wizard
bull Select the location of the GRASS folder and then name a new Location assigning the CRS and extent
bull Then create a new Mapset that will be you personal workspace
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Topology with qGISGRASS
bull After loading in qGIS the vector layer you want to clean up
bull Use the lsquoPluginsGRASSOpen GRASS Toolsrsquo command to pop up the Tools dialog and look for the lsquoImport loaded vectorrsquo tool
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Topology with qGISGRASS
bull Select the lsquoland_degradationrsquo layer for clean up (has to be visible in the legend)
bull Write a name for the lsquocleaned uprsquo version
bull Run the tool and wait for the lsquoView outputrsquo button to get enabled (it may take a while)
bull Then click on this button to see in qGIS the imported vector data
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
bull Some problems may have already been fixed by the import
bull For instance intersecting polygons rings are separated into non-intersecting rings
Before
After
Topology with qGISGRASS
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
bull Once imported we can use additional GRASS topology clean up tools
Topology with qGISGRASS
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
bull Exercise
Create missing intersection points for a line dataset ndash Import the Roads_2010 shape to GRASS
ndash Display cleaned Roads in qGIS
ndash Run the vcleanbreak tool on the imported version
ndash Display the result in qGIS and Save As a new shapefile
ndash Toggle editing for both the original and the lsquobrokenrsquo version to see that the intersection point has been added
Topology with qGISGRASS
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
Reference documentation
httpqgisspatialthoughtscom201201
setting-up-working-grass-environment-inhtml
httpdocumentationqgisorghtmlituser_manual
grass_integrationgrass_integrationhtml
GRASS has many other analysis tools worth trying
It also has tools to upload GRASS data directly into PostGIS
Topology with qGISGRASS
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation
References bull OpenGeo education center
httpworkshopsopengeoorg
bull Introduction to GIS using qGIS
httpwwwbaruchcunyedugeoportalpracticum
bull Quantum GIS documentation
httpwwwqgisorgendocumentationhtml
bull PostGIS documentation from its makers Refractions Research
httppostgisrefractionsnetdocumentation