introduction to gdal/ogr - ocw.un-ihe.org · pdf file3 1 introduction during these exercises...

20
1 Introduction to GDAL/OGR Johannes van der Kwast UNESCO-IHE Institute for Water Education E-mail: [email protected] Version 3.1, September 2015 OpenCourseWare

Upload: vuonghuong

Post on 30-Mar-2018

230 views

Category:

Documents


4 download

TRANSCRIPT

Page 1: Introduction to GDAL/OGR - ocw.un-ihe.org · PDF file3 1 Introduction During these exercises you will get familiar with the basics of the Geodata Abstraction Library (GDAL). The first

1

Introduction to GDAL/OGR

Johannes van der Kwast

UNESCO-IHE Institute for Water Education

E-mail: [email protected]

Version 3.1, September 2015

OpenCourseWare

Page 2: Introduction to GDAL/OGR - ocw.un-ihe.org · PDF file3 1 Introduction During these exercises you will get familiar with the basics of the Geodata Abstraction Library (GDAL). The first

2

Contents 1 Introduction ............................................................................................................................................. 3

1.1 Learning objectives .......................................................................................................................... 3

2 Prerequisites ............................................................................................................................................ 3

2.1 Installation ....................................................................................................................................... 3

3 Using the command line .......................................................................................................................... 4

3.1 Opening a command prompt ........................................................................................................... 4

3.2 Viewing the contents of a folder ...................................................................................................... 5

3.3 Changing directories ........................................................................................................................ 6

3.4 Creating files .................................................................................................................................... 8

3.5 Copy, move, rename and delete files ............................................................................................ 11

3.6 Command history ........................................................................................................................... 12

4 GDAL - The Geospatial Data Abstraction Library ................................................................................... 13

5 Retrieving information from GIS data .................................................................................................... 14

5.1 Raster data ..................................................................................................................................... 14

5.2 Vector data ..................................................................................................................................... 14

6 Reprojecting files ................................................................................................................................... 15

6.1 Introduction ................................................................................................................................... 15

6.2 Raster data ..................................................................................................................................... 15

6.3 Vector data ..................................................................................................................................... 15

7 Change raster properties ....................................................................................................................... 16

7.1 Resize an image .............................................................................................................................. 16

7.2 Change raster format ..................................................................................................................... 16

8 Spatial queries of vectors ....................................................................................................................... 17

9 Convert Comma Separated Files ............................................................................................................ 18

10 Batch conversion .................................................................................................................................... 20

Page 3: Introduction to GDAL/OGR - ocw.un-ihe.org · PDF file3 1 Introduction During these exercises you will get familiar with the basics of the Geodata Abstraction Library (GDAL). The first

3

1 Introduction During these exercises you will get familiar with the basics of the Geodata Abstraction Library (GDAL).

The first part of these exercises introduce you to the DOS command prompt. If you are already familiar

with DOS commands, you can skip chapter 3 and start immediately with chapter 4.

These exercises are provided as OpenCourseWare and come without support. For support, please

register to our related short courses that can be found at http://www.unesco-ihe.org/shortcourses.

1.1 Learning objectives After these exercises you will be able to:

• Use the DOS prompt

• Understand files and directories and their relative and absolute paths

• Retrieve information (metadata) from raster and vector data

• Convert raster and vector data formats

• Reproject raster and vector data formats

• Change raster properties

• Perform spatial queries on vector layers

• Convert comma separated files

• Convert spatial data to Google KML.

2 Prerequisites

2.1 Installation

For these exercises GDAL needs to be installed, preferably using the OSGEO4W distribution package.

The GDAL website: http://www.gdal.org

The OSGEO4W website: http://trac.osgeo.org/osgeo4w/

Page 4: Introduction to GDAL/OGR - ocw.un-ihe.org · PDF file3 1 Introduction During these exercises you will get familiar with the basics of the Geodata Abstraction Library (GDAL). The first

4

3 Using the command line During the excersises we are going to use the Command Prompt. If you already know how to use the the

Command Prompt (aka DOS prompt), you can skip this section.

3.1 Opening a command prompt To open a command prompt you can go do:

Start → All Programs → OSGeo4W →OSGeo4W Shell

Now you will see the Command Prompt on your screen:

Page 5: Introduction to GDAL/OGR - ocw.un-ihe.org · PDF file3 1 Introduction During these exercises you will get familiar with the basics of the Geodata Abstraction Library (GDAL). The first

5

C:\> is called the command prompt. It shows your current working directory (or folder). In this case it

means that you are at drive C (which is called the root). You cannot delete or edit the command

prompt. You'll have to type the commands after the > of the command prompt.

How does your command prompt look like? Give drive name, directory and (sub)directories. It

may differ from the example in the screenshot.

3.2 Viewing the contents of a folder In this section, you will view the contents of a directory by using the dir command. The dir

command stands for "directory."

Type the following command after the prompt: dir

After typing a command press the <Enter> key to execute.

You will see something similar to this:

This is called a directory list. A directory list is a list of all the files and subdirectories that a directory

contains. If you type dir /? you will see all options of the dir command. For example, if you type

dir /w you will see the directory listing in columns.

Page 6: Introduction to GDAL/OGR - ocw.un-ihe.org · PDF file3 1 Introduction During these exercises you will get familiar with the basics of the Geodata Abstraction Library (GDAL). The first

6

3.3 Changing directories

Look at the list on your screen. All the names that have <DIR> beside them are directories (or folders).

The others are files. The size of the files is also displayed. You can see a list of the files in another directory

by changing to that directory, and then using the dir command again.

Type the following commands:

md John

md Peter

These commands create subdirectories called John and Peter. Check if the directory has been created

by listing the directories.

Type:

dir

Also try:

dir /AD

that only lists the directories

Now to go to the newly created directory "John" by typing

cd John

cd means "change directory". You will see that the command prompt changes to C:\...\John>

Page 7: Introduction to GDAL/OGR - ocw.un-ihe.org · PDF file3 1 Introduction During these exercises you will get familiar with the basics of the Geodata Abstraction Library (GDAL). The first

7

DOS is not case sensitive (it interprets upper and lowercase in a similar way), so you could also have

typed cd john.

List the contents of the directory John by typing

dir

Now you'll see that the directory is not empty. There are to directories, namely . and ..

Actually, the directory is empty, but it also shows the relative paths:

. means "current directory"

.. means "mother directory (or one directory up)"

Type:

cd ..

Check the command prompt: we're back in the directory where we started the exercise. Now

type :

cd .

What is the result of this command? Explain why.

We can go back to directory John using the relative path: cd

.\John

Note that this is the same as cd John.

Page 8: Introduction to GDAL/OGR - ocw.un-ihe.org · PDF file3 1 Introduction During these exercises you will get familiar with the basics of the Geodata Abstraction Library (GDAL). The first

8

Or the absolute path (in my case): cd C:\Users\jkw\John

If we want to go to the root (drive C) we can do it using the absolute path:

cd \

or the relative path:

cd ..\..\..

Go back to the directory called John. You can choose if you want to use the relative or absolute

path.

Type cd ..\Peter

Look at the prompt and explain what this command did.

Both ways of changing directories are equivalent. Why bother than? Well, if you move your files that are

organized in directories and subdirecties to another drive (e.g. drive D), than your absolute paths will not

work, while your relative paths will still refer to the same locations. This is specifically important when

writing scripts, which we'll do later.

Now we're going to delete the directory called Peter. First move one directory up, type:

cd ..

than type:

rd Peter

rd means "remove directory".

List the directory to check if Peter has been removed.

If you want to change to another drive, e.g. drive D, you use this

command:

d:

Now go back to c by typing:

c:

You'll be back in the last directory accessed on the C drive.

Move back to the directory where you created John before continuing with the next exercise.

3.4 Creating files Of course you can create files using your Windows software, but since you're learning the command

prompt you'll learn how to create a simple ASCII text file.

Page 9: Introduction to GDAL/OGR - ocw.un-ihe.org · PDF file3 1 Introduction During these exercises you will get familiar with the basics of the Geodata Abstraction Library (GDAL). The first

9

Type:

copy con listdirectories.bat

Type:

dir /AD

Type

<CTRL>-<Z>

(keep the control button pressed and type z)

Your screen looks like this:

Check if the file is in the directory list. What is the size of the file? Now display the contents of the file on the screen. Type: type listdirectories.bat

Now type:

listdirectories

Congratulations! You have created your first batch file. Batch files are scripts that can be used to

execute commands in batch. In this case the file executed dir /AD

Because batch files always have the file extension .bat, the computer knows that this is a batch file

and will execute the commands in the file. We'll come back to this later.

Let's create another file. Type:

dir > list.txt

This command will not show the result of the dir command on the screen, but saves it to an ASCII

file called list.txt. So we can use > after a command to save its results to a file instead of

printing it to the screen.

Check the contents of the file:

type list.txt

Page 10: Introduction to GDAL/OGR - ocw.un-ihe.org · PDF file3 1 Introduction During these exercises you will get familiar with the basics of the Geodata Abstraction Library (GDAL). The first

10

If the list is larger than the Command Prompt screen, you have to scroll. With large text files it is

easier to use:

type list.txt | more

The result of list.txt is given to the more command which displays the results in pages as big as your

window. Press <ENTER> to see the next line. Press <SPACE BAR> to see the next page. Press

<CTRL-C> to stop. You can use this last key combination to stop any command if it isn't doing what

you like, it crashed or it takes too long.

Type:

more

Nothing happens, because the more command expects input from another command. So you can

wait forever. In this case you can stop the execution of the more command by using <CTRL-C>.

Now try this:

type listdirectories.bat >> list.txt

Display the result:

type list.txt | more

What happened?

In summary:

• > saves the result of a command to a new file. If the file on the right hand of > already exists,

it will be overwritten.

• >> appends the result of a command to an existing ASCII file.

• | uses the result of the command on the left hand side in the command on the right hand side

of the |. This is called a pipe.

• Use <CTRL-C> to stop the execution of a command.

We can also use these operators to create so called lock files. These are used in scripting. The script will

check if a certain file exists. If it exists, the program will wait, if it is removed, the program continues.

Therefore, these files can be empty. You can create a lockfile with: type NUL > lockfile.txt

Page 11: Introduction to GDAL/OGR - ocw.un-ihe.org · PDF file3 1 Introduction During these exercises you will get familiar with the basics of the Geodata Abstraction Library (GDAL). The first

11

3.5 Copy, move, rename and delete files Now we can copy the file list.txt to a new file by typing:

copy list.txt newlist.txt

We can also copy list.txt to subdirectory John:

copy list.txt john\newlist.txt

Now we move newlist.txt to the directory John:

move newlist.txt John

Go to the subdirectory John and check the directory listing.

Because it is a bit confusing to have a copy of list.txt in subdirectory John, we're going to rename

it:

rename list.txt listjohn.txt

Check the directory listing again.

Save the directory listing to a file called dirjohn.lst

Now I'll introduce so called wildcards.

Type dir

Type

dir *.txt

dir *john.*

dir *.??t

dir *.?xt

Explain the function of * and ?

We can also append textfiles:

copy dirjohn.lst+listjohn.txt listappend.txt

Because we made copies, we want to remove duplicate file. Type:

del newlist.txt

You can now remove all files by typing:

(Make sure you are in the right folder!!)

del *.* (or simply: del .)

Page 12: Introduction to GDAL/OGR - ocw.un-ihe.org · PDF file3 1 Introduction During these exercises you will get familiar with the basics of the Geodata Abstraction Library (GDAL). The first

12

Now remove the directory John.

3.6 Command history Sometimes you often use the same command. There are several tricks to type them more efficiently.

Type <F3>

This repeats the last command used.

Clear the command prompt and press the right arrow button several times. You can see that the

characters of the previous command are repeated. If you press the up and down buttons, you

can browse through the previously used command.

When you use <TAB> while typing a path or a filename, it will try to automatically complete it.

With the doskey command we can do even more. Type

doskey /h

This prints all the commands you typed during this session to the screen.

If you close the command prompt, the command history will be lost.

Save the command history to a text file using one of the commands previously learned.

In this way you can edit the command history file in e.g. notepad. If you remove all the wrong commands

and you save the file with the .bat extension, you can execute all of the commands in batch.

Try this for a few of the commands you have learned so far.

You can close a command prompt either by clicking on the cross in the corner, typing exit and pressing

enter, or choosing Close when right clicking the Command Prompt icon on the task bar.

Page 13: Introduction to GDAL/OGR - ocw.un-ihe.org · PDF file3 1 Introduction During these exercises you will get familiar with the basics of the Geodata Abstraction Library (GDAL). The first

13

4 GDAL - The Geospatial Data Abstraction Library GDAL is a translator library for raster geospatial data formats that is released under an X/MIT style Open

Source license by the Open Source Geospatial Foundation. As a library, it presents a single abstract data

model to the calling application for all supported formats. It also comes with a variety of useful

commandline utilities for data translation and processing. See for more info: http://www.gdal.org

Download the GDAL using the OSGeo4W installer: http://trac.osgeo.org/osgeo4w/

Download the dataset for this exercise and save it to your harddrive e.g. D:\gdalExercises

The folder contains freely available GIS data:

roads.shp: road map from OpenStreetMap (http://openstreetmap.org)

srtm_37_02.tif: tile of a Digital Elevation Model (DEM) from the Shuttle Radar

Topography Mission (SRTM) (http://www2.jpl.nasa.gov/srtm/)

gem_2011_gn1.shp: borders of Dutch communities, freely available from CBS (Statistics

Netherlands) and Kadaster (http://www.cbs.nl).

Before continuing, make sure that you are in the right folder.

Open the OSGeo4W Shell (Start → All Programs → OSGeo4W → OSGeo4W Shell)

Change directory to

D:\gdalExercises

Page 14: Introduction to GDAL/OGR - ocw.un-ihe.org · PDF file3 1 Introduction During these exercises you will get familiar with the basics of the Geodata Abstraction Library (GDAL). The first

14

5 Retrieving information from GIS data

5.1 Raster data One of the easiest and most useful commands in GDAL is gdalinfo. When given an image as an

argument, it retrieves and prints all relevant information that is known about the file. This is especially

useful if the image contains additional tag data, as is the case with TIF files. When working with satellite

imagery, this is an extremely useful way of keeping track of the images location in long./lat. coordinates

as well as the image projection.

Execute the following command:

gdalinfo srtm_37_02.tif

What is the size of the image?

What is the coordinate system? What is the EPSG code?

5.2 Vector data Sometimes similar information is needed from a vector image, for this there is ogrinfo.

Execute the following commands:

ogrinfo -al roads.shp | more

ogrinfo -al gem_2011_gn1.shp | more

What are the coordinate systems of these shapefiles?

Lookup the EPSG codes of both shapefiles at http://spatialreference.org. You'll need this later.

There exists a useful online tool to convert .prj format (as included by ESRI files) to EPSG code:

http://prj2epsg.org

Page 15: Introduction to GDAL/OGR - ocw.un-ihe.org · PDF file3 1 Introduction During these exercises you will get familiar with the basics of the Geodata Abstraction Library (GDAL). The first

15

6 Reprojecting files

6.1 Introduction In this exercise we want to make a map of the community of Delft with the main roads and the relief.

Because the datasets are in different formats we have to reproject them to a common coordinate

system. Here we reproject all datasets to the Dutch Amersfoort/RD New projection.

6.2 Raster data GDAL has the capability to change a raster coordinate system using the following syntax:

gdalwarp -t_srs EPSG:... <input> <output>

The -t_srs argument specifies the target coordinate system. If the source coordinate system is

unknown it must be specified with the -s_srs argument. EPSG:... specifies the EPSG code of the

projection.

We are now going to reproject a Digital Elevation Model (DEM) acquired by the Shuttle Radar

Topography Mission (SRTM). You can use this website to download DEM's for your area of interest:

http://srtm.csi.cgiar.org/index.asp

In order to reproject the DEM from WGS-84 lat/lon to Amersfoort/RD New we use this command:

gdalwarp -t_srs EPSG:XXXXX srtm_37_02.tif dem_rd.tif

Replace XXXXX with the proper EPSG code for Amersfoort/RD New (see one of your previous

answers using ogrinfo).

Execute the command and visualize the result in QGIS.

6.3 Vector data For vector data again ogr is used to convert the OpenStreetMaps road data to the Amersfoort/RD New

projection.

Execute:

ogr2ogr -t_srs EPSG:XXXXX roadsreprojected.shp roads.shp

Replace XXXXX with the proper EPSG code.

Note that with ogr the output filename is typed before the input filename! With gdal and most other

tools this is different.

Page 16: Introduction to GDAL/OGR - ocw.un-ihe.org · PDF file3 1 Introduction During these exercises you will get familiar with the basics of the Geodata Abstraction Library (GDAL). The first

16

7 Change raster properties

7.1 Resize an image The gdal_translate command gives us several options to modify raster images.

gdal_translate can be used simply to change the size of an image using the -outsize

parameter, which takes two integer values as the xsize and ysize respectively, or two percentage

values to scale the image. The syntax is:

gdal_translate -outsize newxsize newysize inputFile outputFile

Let's apply it to our DEM:

gdal_translate -outsize 15% 15% dem_rd.tif resized.tif

Visualize the original image and the resized image in QGIS.

Explain what happened.

7.2 Change raster format gdal_translate's primary function is to change between image formats. Resizing and changing the

image format can also be combined into one step by using both the -outsize and -of parameters.

The basic syntax is:

gdal_translate -of FORMAT inputFile outputFile

All supported formats can be found here: http://gdal.org/formats_list.html

Now we are going to convert the DEM from geoTiff to PCRaster format. PCRaster is open source

software for spatial dynamic modelling and has its own GIS format (http://pcraster.geo.uu.nl).

Execute:

gdal_translate -of PCRaster -ot Float64 dem_rd.tif dem.map

Visualize the result in QGIS

The -ot argument is needed to specifiy that it is continuous data, so PCRaster will interpret the map as

a scalar (continuous) data layer. Use -ot Int32 to convert to integer (thematic) maps. Use

-ot Byte to convert to a boolean PCRaster map.

Page 17: Introduction to GDAL/OGR - ocw.un-ihe.org · PDF file3 1 Introduction During these exercises you will get familiar with the basics of the Geodata Abstraction Library (GDAL). The first

17

8 Spatial queries of vectors For our map of Delft we want to do the following GIS analysis:

Select the community of Delft from the community map and save it into a new shapefile;

Intersect the community boundaries of Delft with the road map of the Netherlands.

We can use a spatial query to select a feature from a vector map.

What is the attribute in the community map containing the names of the communities? You can

use either ogrinfo or QGIS to answer this question.

Execute the following command:

ogr2ogr -f "ESRI Shapefile" -where GM_NAAM='Delft'

-a_srs EPSG:28992 delft.shp gem_2011_gn1.shp

This will save the feature with GM_NAAM Delft to a new file called delft.shp. The argument -

a_srs EPSG:28992 is used to assign the Amersfoort/RD New projection to the output file. The

argument -f defines the output format.

Now open in QGIS the reprojected DEM (dem_rd.tif), the reprojected road map

(roadsreprojected.shp) and the community of Delft (delft.shp).

Page 18: Introduction to GDAL/OGR - ocw.un-ihe.org · PDF file3 1 Introduction During these exercises you will get familiar with the basics of the Geodata Abstraction Library (GDAL). The first

18

9 Convert Comma Separated Files Sometimes you want to reproject coordinates in an ASCII file, e.g. which has been saved in a

spreadsheet program. Here we will convert the coordinates in a comma separated ASCII file

(locations.csv) to a new ASCII file (locations_reprojected.csv).

View the contents of locations.csv using notepad

First, we have to create a virtual data source by creating an XML control file. On your windows

computer open notepad and type the XML code in the screenshot below. Use indentations of

three spaces.

Save the file as "locations.vrt" in the gdalExercises folder.

Some explanation about the XML file:

<OGRVRTLayer name="locations"> should correspond with the

<SrcDataSource>locations.csv</SrcDataSource>

<GeometryField encoding="PointFromColumns" x="lon" y="lat"/>

indicates the columns with the coordinates that you want to convert.

Execute the following command:

ogr2ogr -t_srs EPSG:28992 -f "CSV" locations_reprojected

locations.vrt -lco GEOMETRY=AS_XY

In this example locations.csv with lat/lon WGS-84 coordinates is converted to

locations_projected.csv with Amersfoort/RD New projection. Note that the file is saved in the

folder with the same name as the output file.

Use notepad to check locations_reprojected.csv. What is saved in each column?

In the same way we can convert the comma separated file to a shapefile. Execute:

ogr2ogr -f "ESRI Shapefile" locations.shp locations.vrt

Visualize the shapefile in QGIS by plotting the locations over the DEM, road map and Delft

community border. Make a nice map.

We can also convert the locations to Google Earth KML format. Type:

ogr2ogr -f KML locations.kml locations.vrt locations

Page 19: Introduction to GDAL/OGR - ocw.un-ihe.org · PDF file3 1 Introduction During these exercises you will get familiar with the basics of the Geodata Abstraction Library (GDAL). The first

19

Note that you don't need to specify an output projection (-t_srs), because for KML this is

always WGS 84 (EPSG:4326).

Visualize the results in Google Earth in Windows. If you don't have Google Earth yet, you can

download it from: http://www.google.com/earth/index.html

Double click the file and it opens in Google Earth.

What are the objects in our CSV file?

Page 20: Introduction to GDAL/OGR - ocw.un-ihe.org · PDF file3 1 Introduction During these exercises you will get familiar with the basics of the Geodata Abstraction Library (GDAL). The first

20

10 Batch conversion Desktop GIS programmes are very useful for GIS operations, but are hard to use if we have to repeat the

same task for many GIS layers. Then scripting can be a solution.

Here we have an example dataset from a land-use model of Dublin. The data are in IDRISI raster format,

with one layer for each year between 1990 and 2030. Our task is to convert all layers to .tif format.

Unzip landuse.zip to a folder called "landuse".

Open the command prompt and "cd" to this folder.

Make a batch file with the following commands:

Try to understand the code. This is a for loop that loops over all *.rst files in the folder. %%f is the

variable that contains the filename of each file. With echo we can print something to the screen.

Here we print %%~nf , which is the part of the filename before the dot that separates it from the

extension. Then we use the gdal_translate command with output format GeoTiff. At the end of

the line we add the .tif extension to the filename.

Now you can execute the batch file and check the results.