from pandas to geopandas - 地理資料處理與分析

From Pandas to GeoPandas

倪鈵斯@PYCON TAIWAN 2016

Painted by Shih Jiang-Zhu.

誤入叢林的⼩小⽩白兔

ABOUT ME

倪鈵斯（nispc） National Taiwan University of Science and Technology Major in Information Management


ABOUT ME

Development Intern DSP.IM (DSP智庫驅動) 2015~


Hacking Skills


Computer Science

Pandas GIS


ABOUT ME

台科⼤大資管系 2014~

DSP智庫驅動 2015~


ABOUT ME

台科⼤大資管系 2014~

DSP智庫驅動 2015~

Hacking Skills

Pandas GIS

From These Placees, I Learned


ABOUT ME

Pandas GIS

GeoPandas

台灣最常⽤用的座標參考系統

The most Common CRS in Taiwan

Geographic coordinate system • WGS84（Lat/Lon）

Projected Coordinate System • TWD67 ( TM2 )、TWD97 ( TM2 )

The image above using Mercator Projection, but TM2 using Transverse Mercator ProjectionNote:

以直觀（但不太正確）的角度解釋TM2的台北座標

Using TM2 to Locate Taipei City

Taipei City x = 304803.322190434 m y = 2767518.970508755 m

X

Y

(0, 0)

分辨⼿手上的資料是TM⼆二度分帶（TWD67 or 97）或經緯度座標

Distinguish which CRS of your Data

Before processing your data, you should know the CRS of your dara. You can do the following if you are not sure:

TM2 or Lat/Lon？

• If the numbers is bigger your data must be TM2.

TWD97( TM2 ) or TWD67( TM2 )？

• Normally, It’s TWD97 data, unless you get that earlier than 1999 ( 921 Jiji earthquake ).

台灣常⽤用的座標參考系統

More About Common CRS Used in Taiwan

• http://www.sunriver.com.tw/grid_tm2.htm

A website explaining the details of common CRS Used in Taiwan

http://www.sunriver.com.tw/grid_tm2.htm

Shapefi le的結構

The Structure of Shapefile

• Shapefile save numerical data and text data in dBase(.dbf) file.

• Shapefile save shapes and positions in .shp file.

• and .shx save index, .prj save projection information.

讓我們快速的匯入台灣村⾥里界圖！

Let’s Quickly Import Our Data

%pylab inlinepylab.rcParams['figure.figsize'] = (20.0, 20.0)import geopandas as gpdimport shapely

villages_shp = gpd.read_file('./台灣⾥里界圖_TWD97/Village_NLSC_121_1050219.shp')

Import the geoPandas, and setting ipython notebook to plot map more beautifully.

Import a shapefile.

Let’s Quickly Import Our Data讓我們快速的匯入台灣村⾥里界圖！

See the data

COUNTY_ID C_Name OBJECTI

D PRO_ID Substitute TOWN_ID T_Name … geometry

0 64 ⾼高雄市 8573 64 中興⾥里 6401000 旗津區 …POLYGON

((177607.7583971694

1 10007 彰化縣 8578 10 頂潭⾥里 1000712 ⽥田中鎮 …POLYGON

((207765.5412078464

2641446.58074142 10007 彰化縣 8579 10 中潭⾥里 1000712 ⽥田中鎮 …

POLYGON ((207195.1317646

012 2641217.4845214

3 10007 彰化縣 8580 10 ⻯⿓龍潭⾥里 1000712 ⽥田中鎮 …POLYGON

((206697.0792388434

2641761.28365324 10007 彰化縣 8581 10 ⼤大社⾥里 1000712 ⽥田中鎮 …

POLYGON ((206697.0792388

434 2641761.2836532

villages_shp.head()

讓我們快速的匯入台灣村⾥里界圖！

Let’s Quickly Import Our Data

villages_shp.plot(colormap='OrRd')

… and plot it!

geometry

楠梓區 POLYGON ((180500.0468580014 2511840.500121991,...

左營區 POLYGON ((179578.8300897071 2507758.486656731,...

三⺠民區 POLYGON ((182669.6859151478 2504196.741750024,...

⿎鼓⼭山區 POLYGON ((175673.2319138724 2502706.179941533,...

新興區 POLYGON ((178357.0004510713 2502407.254186806,...

前⾦金區 POLYGON ((177708.4723173666 2502176.189247195,...

鹽埕區 POLYGON ((177056.8290152341 2502272.220627455,...

苓雅區 POLYGON ((180854.1866929225 2501717.513408468,...

前鎮區 POLYGON ((180493.3661862851 2496473.692563967,...

旗津區 POLYGON ((178725.3208101905 2495444.779886731,...

⼩小港區 POLYGON ((184614.1874942174 2490602.750132099,...

GeoPandas如何儲存形狀

How GeoPandas Storage The Shapes

geometry

楠梓區 POLYGON ((180500.0468580014 2511840.500121991,...

左營區 POLYGON ((179578.8300897071 2507758.486656731,...

三⺠民區 POLYGON ((182669.6859151478 2504196.741750024,...

⿎鼓⼭山區 POLYGON ((175673.2319138724 2502706.179941533,...

新興區 POLYGON ((178357.0004510713 2502407.254186806,...

前⾦金區 POLYGON ((177708.4723173666 2502176.189247195,...

鹽埕區 POLYGON ((177056.8290152341 2502272.220627455,...

苓雅區 POLYGON ((180854.1866929225 2501717.513408468,...

前鎮區 POLYGON ((180493.3661862851 2496473.692563967,...

旗津區 POLYGON ((178725.3208101905 2495444.779886731,...

⼩小港區 POLYGON ((184614.1874942174 2490602.750132099,...

How GeoPandas Storage The ShapesGeoPandas如何儲存形狀

All of the shapes and positions are save in shapely.geometry objects

However, GIS not Only Deal with Spatial Data,

But Also Attribute Data.

」

「

結合數值資料並且做資料視覺化

Combine Numerical Data and Visualization

geometry

楠梓區 POLYGON ((180500.0468580014 25...

左營區 POLYGON ((179578.8300897071 25...

三⺠民區 POLYGON ((182669.6859151478 25...

⿎鼓⼭山區 POLYGON ((175673.2319138724 25...

新興區 POLYGON ((178357.0004510713 25...

前⾦金區 POLYGON ((177708.4723173666 25...

鹽埕區 POLYGON ((177056.8290152341 25...

苓雅區 POLYGON ((180854.1866929225 25...

前鎮區 POLYGON ((180493.3661862851 24...

旗津區 POLYGON ((178725.3208101905 24...

⼩小港區 POLYGON ((184614.1874942174 24...GeoSeries

結合數值資料並且做資料視覺化

Combine Numerical Data and Visualization

⼈人⼝口⼈人⼝口密度 geometry

楠梓區

左營區

三⺠民區

⿎鼓⼭山區

新興區

前⾦金區

鹽埕區

苓雅區

前鎮區

旗津區

⼩小港區

census density

Series GeoSeries

？

To Combine Numerical Data and Visualization,

We Should Preprocess Our Data

」

「

使⽤用⼩小資料練習Pandas及GeoPandas

Practice Pandas and GeoPandas with Small Data!

villages_shp = gpd.read_file( ‘./台灣⾥里界圖_TWD97/Village_NLSC_121_1050219.shp)

Import a shapefile:

census_villages = pd.read_csv( 'Data/104年12⽉月⼈人⼝口數.csv', skiprows=[0])

Load census data:

COUNTY_ID C_Name OBJECTI

D PRO_ID Substitute TOWN_ID T_Name … geometry

0 64 ⾼高雄市 8573 64 中興⾥里 6401000 旗津區 …POLYGON

((177607.7583971694

2497521.1105418

1 10007 彰化縣 8578 10 頂潭⾥里 1000712 ⽥田中鎮 …POLYGON

((207765.5412078464

2641446.5807414

villages_shp.head(2)

census_villages.head(2)

統計年⽉月區域別村⾥里⼾戶數⼈人⼝口數

⼈人⼝口數-男

⼈人⼝口數-⼥女

0歲-男 0歲-⼥女 …

0 10412 新北市板橋區

留侯⾥里 691 1649 786 863 5 8 …

1 10412 新北市板橋區流芳⾥里 638 1564 732 832 3 3 …

have some problems.

census_villages['city'] = census_villages[‘區域別’].copy(deep=True)

census_villages['city'] =census_villages['city'].apply(lambda t: t[:3])

census_villages['district'] =census_villages['區域別'].copy(deep=True)

census_villages['district'] =census_villages['district'].apply(lambda t: t[3:])

統計年⽉月

區域別村⾥里⼾戶數⼈人⼝口數⼈人⼝口數-男 … city district


留侯⾥里 691 1649 786 … 新北市板橋區


流芳⾥里 638 1564 732 … 新北市板橋區


⾚赤松⾥里 318 838 410 … 新北市板橋區

demo_villages = ['惠楠⾥里', '惠⺠民⾥里', '惠豐⾥里', '莒光⾥里', '合群⾥里', '永清⾥里', '橋頭⾥里', ‘橋南⾥里']

kaohsiung_villages_shp =villages_shp[villages_shp['C_Name']=='⾼高雄市']

demo_villages_shp = kaohsiung_villages_shp[ kaohsiung_villages_shp['V_Name'].apply( lambda t: True if t in demo_villages else False )]

Select Some Villages for Demo

, and Leave the Columns which We Needed.kaohsiung_villages_shp = kaohsiung_villages_shp[

['C_Name', 'T_Name', 'V_Name', ‘geometry’]]

C_Name T_Name V_Name geometry

16 ⾼高雄市左營區永清⾥里 POLYGON ((177943.1675527846 25...

17 ⾼高雄市左營區莒光⾥里 POLYGON ((177698.0080210482 25...

2675 ⾼高雄市左營區合群⾥里 POLYGON ((177309.4301372209 25...

2705 ⾼高雄市楠梓區惠楠⾥里 POLYGON ((181185.0309484294 25...

2706 ⾼高雄市楠梓區惠⺠民⾥里 POLYGON ((180715.7155030172 25...

2707 ⾼高雄市楠梓區惠豐⾥里 POLYGON ((179848.5000471332 25...

3208 ⾼高雄市橋頭區橋頭⾥里 POLYGON ((179476.0936987746 25...

3209 ⾼高雄市橋頭區橋南⾥里 POLYGON ((179887.6875365854 25...

demo_villages_shp

Select Some Villages for Demo

demo_villages = ['惠楠⾥里', '惠⺠民⾥里', '惠豐⾥里', '莒光⾥里', '合群⾥里', '永清⾥里', '橋頭⾥里', ‘橋南⾥里']

kaohsiung_census_villages = census_villages[census_villages['city'] == '⾼高雄市']

demo_census_villages = kaohsiung_census_villages[ kaohsiung_census_villages['村⾥里'].apply( lambda t: True if t in demo_villages else False )]

, and Leave the Columns which We Needed.demo_census_villages = demo_census_villages[ ['city', 'district', '村⾥里', '⼈人⼝口數']]

city district 村⾥里⼈人⼝口數

3430 ⾼高雄市左營區永清⾥里 1288

3431 ⾼高雄市左營區莒光⾥里 2024

3433 ⾼高雄市左營區合群⾥里 2656

3463 ⾼高雄市楠梓區惠楠⾥里 3533

3464 ⾼高雄市楠梓區惠⺠民⾥里 4514

3465 ⾼高雄市楠梓區惠豐⾥里 5407

4020 ⾼高雄市橋頭區橋頭⾥里 2550

4021 ⾼高雄市橋頭區橋南⾥里 1299

demo_census_villages

比較處理空間資料及屬性資料—Group By（1）

Dealing with Spatial and Attribute Data Simultaneously



















⾼高雄市左營區

⾼高雄市楠梓區

⾼高雄市橋頭區

demo_villages_shp_group =\ demo_villages_shp.groupby(by=['C_Name', 'T_Name'])

C_Name T_Name geometry

0 ⾼高雄市左營區POLYGON ((177943.1675527846

2510441.206417892,...

1 ⾼高雄市楠梓區POLYGON ((181185.0309484294

2514217.982513821,...

2 ⾼高雄市橋頭區POLYGON ((179887.6875365854

2517777.250005289,...

demo_district_shp = gpd.GeoDataFrame( demo_villages_shp_group[‘geometry’] .agg(lambda t: shapely.ops.unary_union(t)))

aggregation function










Dealing with Spatial and Attribute Data Simultaneously比較處理空間資料及屬性資料—Group By（2）










demo_census_district_group =\ demo_census_villages.groupby(by=['city', 'district'])

⾼高雄市左營區

⾼高雄市楠梓區

⾼高雄市橋頭區

city district ⼈人⼝口數

0 ⾼高雄市左營區

1288

2024

2656

1 ⾼高雄市楠梓區

3533

4514

5407

2 ⾼高雄市橋頭區

2550

1299

demo_census_district_group = \ demo_census_district_group.sum()

aggregation function


0 ⾼高雄市左營區

1288

2024

2656

1 ⾼高雄市楠梓區

3533

4514

5407

2 ⾼高雄市橋頭區

2550

1299

demo_census_district_group = \ demo_census_district_group.sum()

SUM ( )SUM ( )SUM ( )

= 5968

= 13454

=3849aggregation function


0 ⾼高雄市左營區 POLYGON ((177943.1675527846 2510441.206417892,...

1 ⾼高雄市楠梓區POLYGON ((181185.0309484294

2514217.982513821,...

2 ⾼高雄市橋頭區POLYGON ((179887.6875365854

2517777.250005289,...


0 ⾼高雄市左營區 5968

1 ⾼高雄市楠梓區 13454

2 ⾼高雄市橋頭區 3849

Dealing with Spatial and Attribute Data Simultaneously ( Join by Attribute)

比較處理空間資料及屬性資料—Join by Attribute


0 ⾼高雄市左營區POLYGON

((177943.1675527846 2510441.206417892,...

1 ⾼高雄市楠梓區POLYGON

((181185.0309484294 2514217.982513821,...

2 ⾼高雄市橋頭區POLYGON

((179887.6875365854 2517777.250005289,...


0 ⾼高雄市左營區 5968

1 ⾼高雄市楠梓區 13454

2 ⾼高雄市橋頭區 3849

demo_census_district_shp = demo_district_shp.merge( demo_census_district, left_on=('C_Name', 'T_Name'), right_on=('city', 'district'))

製作區域密度圖

Making a Choropleth Map

demo_census_district_shp.plot(column='⼈人⼝口數', scheme='QUANTILES', k=3, cmap=‘OrRd'

)

… and plot it!

https://github.com/geopandas/geopandas/blob/master/examples/choropleths.ipynb

Fake Mayor

This is imagined此為講解GIS應⽤用之虛構情境

市⻑⾧長的煩惱

柯P照⽚片： ETtoday 背景圖⽚片：聯合影⾳音 / 鄭清源

https://youtu.be/T1NkqxkjQrw

https://video.udn.com/news/419058

環域分析

Buffer Analysis

https://docs.qgis.org/2.8/en/docs/gentle_gis_introduction/vector_spatial_analysis_buffers.html

疊圖分析

Overlay Analysis

Intersection Union

DifferenceSymmetrical Difference

https://docs.qgis.org/2.8/en/docs/gentle_gis_introduction/vector_spatial_analysis_buffers.html

忠孝橋引道問題

Zhongxiao Bridge’s Problem

Approach road to Zhongxiao Bridge

Select the Approach road

Calculate the zone of influence

Find the Villages which be influenced

Buffer IntersectionFine “忠孝重慶” int. and “忠孝環河" int.

roads = gpd.read_file('./Taiwan Road’)

匯入台灣道路資料

Import Roads’ Data

roads = roads.to_crs(epsg=3826)

Import Taiwan’s Roads:

To Calculate, We Should Project Our GeoDataframe to TWD97（Epsg code: 3826）

選取台北的道路

Select Roads in Taipei

taipei_shp = \ villages_shp[villages_shp[‘C_Name’]=='臺北市']

taipei_roads = \ roads[roads.intersects(taipei_shp.unary_union)]

選出忠孝橋引道

Select “Approach road to Zhongxiao Bridge”

重慶路 = taipei_roads[ taipei_roads['name'].str.contains('重慶')==True]

忠孝路 = taipei_roads[ taipei_roads['name'].str.contains('忠孝')==True]

環河路 = taipei_roads[ taipei_roads['name'].str.contains('環河')==True]

ax = 重慶路.plot(linewidth=3, color="Orange")忠孝路.plot(linewidth=3, ax=ax, color="Blue")環河路.plot(linewidth=3, ax=ax, color="Green")

忠孝路

環河路

重慶路

Fine “忠孝重慶” int. and “忠孝環河" int.

忠孝重慶int = 忠孝路.unary_union.intersection( 重慶路.unary_union).centroid 忠孝環河int = 忠孝路.unary_union.intersection( 環河路.unary_union).centroid

忠孝重慶int.x , 忠孝重慶int.y

We Get the Easternmost Position of “Approach road to Zhongxiao Bridge”

忠孝環河int.x , 忠孝環河int.y

… and the Easternmost Position of “Approach road to Zhongxiao Bridge”

Ouput: (301790.13325580396, 2771103.9320118735)

Output: (301073.18135184597, 2771299.2108038645)

忠孝路_x = 忠孝路.geometry.centroid.apply( lambda t: t.x)

忠孝橋引道 = 忠孝路[ (忠孝路_x < 忠孝重慶int.x)& (忠孝環河int.x < 忠孝路_x)]

畫出影響範圍

Calculate the Zone of Influence

忠孝橋引道buffer = \ gpd.GeoSeries(忠孝橋引道.unary_union.buffer(500)) ax = 忠孝橋引道buffer.plot()忠孝橋引道.plot(ax=ax, linewidth=4)

找出被影響的⾥里

Find the Villages which be Influenced

influences_villages_shp = villages_shp[ villages_shp .intersects(忠孝橋引道buffer.unary_union)==True]

COUNTY_ID C_Name … V_Name geometry

2365 63 臺北市 … 光復⾥里POLYGON ((301794.5933154966

2771355.22341227, ...

2366 63 臺北市 … 黎明⾥里POLYGON ((302365.136612736

2771239.723670578, ...

2374 63 臺北市 … ⽟玉泉⾥里POLYGON ((301272.2482771481

2771805.173890756,...

2375 63 臺北市 … 建明⾥里POLYGON ((302012.1867108274

2771553.461402568,...

...共13⾥里

ax = influences_villages_shp.plot()忠孝橋引道buffer.plot(ax=ax)忠孝橋引道.plot(ax=ax, linewidth=4)

找出各⾥里實際於影響區內的區域

Find the Influenced Part in the Villages

influences_villages_shp_clip = \ influences_villages_shp.copy() influences_villages_shp_clip['geometry'] = \ influences_villages_shp_clip.intersection( 忠孝橋引道buffer.unary_union )

ax = influences_villages_shp.plot(color=‘White')

influences_villages_shp[( influences_villages_shp_clip['geometry'].area / influences_villages_shp['geometry'].area ) > 0.8].plot(ax=ax, color=‘Orange')

忠孝橋引道buffer.plot(ax=ax)忠孝橋引道.plot(ax=ax, linewidth=4)

ax = influences_villages_shp.plot(color=‘White')

influences_villages_shp[( influences_villages_shp_clip['geometry'].area / influences_villages_shp['geometry'].area ) > 0.5].plot(ax=ax, color=‘Orange')

忠孝橋引道buffer.plot(ax=ax)忠孝橋引道.plot(ax=ax, linewidth=4)

DONEC QUIS NUNC

`

台灣的⼯工程師真的很棒

圖⽚片來源：⾃自由時報 / 廖振輝此為講解GIS應⽤用之虛構情境This is imagined

遺珠之憾

Pearls of regret

There are two subjects are interesting, too.

• Spatial join

• Folium

from pandas to geopandas - 地理資料處理與分析

Data & Analytics