Design, Development, & Deployment of Wisconsin DNR

Map Services

Jeff Shaw, John Laedlein, Jerry SullivanWisconsin Department of Natural Resources

Bureau of Technology ServicesGIS Services Section

Wisconsin Land Information Association - 2011 Annual Conference February 16-18, Madison, WI

Design, Development & Deployment of Wisconsin DNR Map ServicesPresentation Content

This presentation shares experiences using ArcGIS Server 9.3.1 and 10 to build cached basemap services within an applications development framework.

Initially, four types of cached services are being rolled out: terrain services (NED hillshades), imagery services (Landsat ETM+, NAIP 2008); feature basemap services (ESRI Streetmap, DNR, DOT layers); and a public lands layers service (WDNR, PADUS CBI, ESRI).

This talk will explore guiding concepts, technical design issues, map rendering and labeling, caching considerations and tips, and integration within an overall applications framework.

AGS Map Service Design & Caching Practices

Jeff Shaw

DNR, GIS

Services

Cached Map Service Structure

L1: 1:4,500,000C..0 C..1 C..2 C..3

R..0

R..1

R..2

R..3

Cache Folder Structure (Exploded format)

L0: 1:9,000,000C00000000 C00000001

R00000000

R00000001

DNR Tiling Scheme

Coordinate System:

Wisconsin Transverse Mercator (WTM)Image Format:

PNG32 or jpeg, 256x256

Scales:

16 levels* ( 1:8,110,080 to 1:990 )

*

Origin

*Based on scales useful for mapping (WGS-based services can’t do that, because their scale varies with latitude)

4959901,9803,9607,92015,840 1” = .25 mi (4” = 1 mi)23,760 *31,680 1”= .5 mi47,520 *63,360 1”= 1 mi95,040 *126,720 1” = 2 mi253,440506,880 1” = 8 mi1,013,7602,027,520 1” = 32 mi4,055,0406,082,560*8,110,080

Why Use a Custom Tiling Scheme?

Issues with WGS 84 Web Mercator-based tiling schemeScale changes with latitude (also distorts area, preserves shape)

7% difference in scale from N. Wisconsin to S. Wisconsin31% difference from equator to central WisconsinWGS 1:72K scale is actually 1:53K in S. WI and 1:49K in N. WIServices projected to WTM shrink 31% - text becomes too smallNOTE - Google & Bing don’t use ellipsoid and have .33% scale distortion in Y direction “since used only for map display, not displaying coordinates”

Finer-grained scales desiredPotential desktop use

ArcMap assumes fixed scale when measuring !!! (v9.3.1) – will be 30% off w/WGSHundreds of layers stored in WTM Coordinate System

Projection on-the-fly performance – Threshold TBDNote –

You can

edit WTM

data w/in a web app displayed in WGS 84 provided you use the Geometry service to project measurements..

Which Scales to Use?

WGS at Equator WGS at WI Lats Optimal WGS DNR Scales* Mapping units9,244,667.36 6,536,967 7,168,000 8,110,080 1 inch = 128.0 miles

5,376,000 6,082,560 1 inch = 96.00 miles4,622,333.68 3,268,483 3,584,000 4,055,040 1 inch = 64.0 miles2,311,166.84 1,634,242 1,792,000 2,027,520 1 inch = 32.0 miles1,155,583.42 817,121 896,000 1,013,760 1 inch = 16.0 miles

577,791.71 408,560 448,000 506,880 1 inch = 8.0 miles288,895.85 204,280 224,000 253,440 1 inch = 4.0 miles144,447.93 102,140 112,000 126,720 1 inch = 2.0 miles72,223.96 51,070 56,000 63,360 1 inch = 1.0 miles36,111.98 25,535 28,000 31,680 1 inch = .50 miles18,055.99 12,768 14,000 15,840 1 inch = .25 miles9,028.00 6,384 7,000 7,920 1 inch = .125 miles4,514.00 3,192 3,500 3,960 1 inch = .063 miles2,257.00 1,596 1,750 1,980 1 inch = .031 miles1,128.50 798 875 990 1 inch = .016 miles

564.25 399 438 495 1 inch = .008 miles

If you might adopt WGS, and want to avoid designing around entirely new scale thresholds…

*not including 1:95K, 1:47K & 1:23K

The cached images you might think would be displayed in ArcMap when projected, aren’t

Modifying a Tiling Schemefor existing services

Adding new scale levels New larger scales, not a problemNew intermediate scales

Rename levelsAdd cached files for new levelsUpdate service config file and restart serverNeed to clear cached files on client – or create a parallel service that points at the same cache folder

Changing origin, image size, or coordinate system –

then need to regenerate the cache

Map Services

Cached map services:Detailed Feature Basemap ”Dark” – lines & labelsPublic Lands Basemap – polygon fillImage Basemap – Hillshaded air photos & satellite imagery

Detailed Feature Basemap ”Light” – lines & labelsBasic Terrain Basemap ”Light” – tan hillshade

Basic Feature Basemap ”Light” – for use with light background

Dynamic map services:Query Layers (PLSS, MCDs, GNIS, etc)Air photos at 1:495 [ Future ??? ]

Future Cached Services:Topo Map - simulates scanned topo-quads Basic Feature Basemap “Dark”Color Infra-Red (IR) Image BasemapFalse Color Satellite

Move to and or have some services in WGS 84 Web Mercator-based scheme

Reference basemaps

Mashup basemaps

Basemap Series

“Light” detailed basemap grp “Dark” detailed basemap group

Ref

eren

ce b

asem

aps

Mas

hup

Bas

emap

“Light” simple basemap “Dark” simple basemap

[ Future ]

Basemap Series

“Light” Colored Basemap Group

Detailed “Dark” Basemap Group

Detailed Basemap Group

Detailed Basemap Group

Shaded Relief

Public Lands

Surfacewater Flow Direction

Map Design Tips

Text halos Light text with dark halo for dark backgroundDark text with light halo for light background

Typical Maplex settings:Repeat line labels & now polygons also (v10)Abbreviation Dictionary

MSD-based anti-aliasing helps a lot!Can use semi-transparent symbols (v10)Labels on top

avoids placement conflicts & they don’t get covered up

Shared Cache Folder

All cache folders stored on the same Storage Area Network (SAN)Dev, UAT and Production services all share the same configuration file and cached filesAllows R&D, Dev, UAT, Prod and workstations to all read and update the same cache folders.

Need less storage spaceMore horsepower for cachingNo need to copy cached files

\\ags_cache\agsoutput

\arcgiscache\MyCachedMapService

\Layersconf.xml\_alllayers

Shared Map Resource Folder

Store map resources on a file shareMultiple SOMS serving parallel services can use the same map resourceMakes promoting changes easier

\\MyFileShare\Intranet

\Caching_workstation\Development\UAT\Prod_staging\Production

\Cached_MapMy_Resource.MSDMy_Resource.MXD

\Dynamic_Map\Public

\...

Staging Map Service

Used to generate the cached filesExists on same disk drive as the published service

\arcgisoutput\Arcgiscache

\MyCachedService\MyCachedService-stage

Workflow1.

Populate the "staging”

service levels2.

Move (drag) old levels folders into “trash”

folder3.

Drag levels from “staging”

service into the published one4.

Delete the “trash”

files when convenient

You never need to copy files –

would take 6 days for a statewide image service (exploded format). ~6 hrs for compact format –

still too long for production

Proxy Map Resource

One “Staging” MXD/MSD used to generate the cacheOver 100 themes to cover all scales in “detailed” basemapUse whatever name is useful to map author

(ex. 24k dissolved DNR_MGS_LNDS, 150K-300K, no labels)

Layers don’t need to be kept on-lineCan change/replace layers with less impact on service layer id referencesTechnically, don’t need any layers – high availability

A different, “Proxy” MSD used for the published serviceFewer layers, less for AGS to manageCan use an MSD, even if a MXD needed for cachingBetter, simpler legend & user-friendly layer namesSelect fields turned off or user-friendly alias assignedUsed for identify/find tasksIf using cache-on-demand at large scale(s), design for that

Cache File Management Tips

Turn off disk indexing

Turn off Windows Explorer thumbnail db (can create locks)

Don’t use ArcMap to evaluate quality (v9.3.1)(often resamples even when it shouldn’t)

Use RoboCopy to copy files, DOS command line to delete files (restarts automatically after network disconnection)

Cache Management

Log these per scale (level) per caching job:# of Rows (folders)# of Tiles (files)Total sizeTotal size on diskTotal Duration

Calculate size and processing time per image

then use to estimate processing time and disk storage space for entire service.

Cached to 1:1,980 scale• Image: 4 days, 125 GB• Feature: 5 days, 55 GB

Problems/Bugs

Status bar is not reliable – rely on your logIncomplete tiles – run the caching process twice (v 10)

To track number of tiles – count files to determine completeness

Repeating labels can crash process (with AGS 10, SP 1)

Might generate incomplete images – don’t assume that if it draws in ArcMap (even MSD) that it will work in AGS.

Layer locking issues are resolved in ArcGIS 10!(Use custom config parameter)

Sequential Layer ID References

DNR Service Registry - sample entry<EN_Detailed_Features_DkBk>

<ID>EN_Detailed_Features_DkBk</ID><TYPE>STATIC</TYPE><URL>http://apwmad1a0229/ArcGIS/rest/services/DW_Map_Cached/EN_Detailed_Feature_Basem

ap_DkBk_WTM_Ext/MapServer</URL><SOAPURL>http://apwmad1a0229/ArcGIS/services/DW_Map_Cached/EN_Basic_Feature_Basema

p_DkBk_WTM_Ext/MapServer</SOAPURL>

<DNROFFICE id="2" shape="POINT"/><MCD id="7" shape="POLYGON"/><OPENWATER id="28" shape="POLYGON"/><COUNTY id="30" shape="POLYGON"/>

</EN_Detailed_Features_DkBk>

Issues

Outdated files cached on client (need to clear client cache occasionally)

We might try:•

Saving a browser cookie that stores the date accessed•

Then, have application compare service update date with last accessed date. Prompt if necessary.

■

Slow Cache-on-Demand (v9.3.1)Conveying Positional Accuracy - disclaimersAcknowledgements for layers used in public services

DNR

Image Basemap

Caches John Laedlein

–

DNR, GIS

Services

•

External Image Basemap

Cache•

Internal Image Basemap

Cache

•

Future…

External Image Basemap Cache

•

Currently available in Production•

Presented in Public DNR-Managed Lands application

•

Landsat

ETM+ Imagery•

10-meter NED Hillshade

•

2008 NAIP Orthos

External Image Basemap Cache

Landsat,10m NED, 2008 NAIP•

Downloaded from USGS GloVis

site

(Bands 7, 4 & 2)•

14-meter resolution, extent WI & adjacent states

•

Cached at small scales•

Rendered as false natural color

•

Data available on DNR

FTP site (~900mb) ftp://dnrftp01.wi.gov/geodata

1:6,082,560

1:6,082,560

1:2,027,520

1:2,027,520

External Image Basemap Cache

Landsat,10m NED, 2008 NAIP•

Acquired by DNR/FR in July 2009

•

Extent buffered WI•

Cached at intermediate to large scales as backdrop to semi-

transparent imagery•

Data available on DNR

FTP site

(~130mb) ftp://dnrftp01.wi.gov/geodata

1:506,880

1:506,880

External Image Basemap Cache

Landsat,10m NED, 2008 NAIP•

Public domain imagery

•

1-meter, leaf-on, natural color•

Extent USGS Q-Q-Quads covering WI

•

Cached at intermediate to large scales

•

Data available from WisconsinView site http://www.wisconsinview.org/

1:253,440

1:63,360

1:15,840

1:3,960

1:1,980

Internal Image Basemap Cache

•

Landsat

ETM+ and 10m NED Hillshade

at small scales

•

Most current “high-resolution”

orthos at intermediate to large scales

•

Image metadata issues

Internal Image Basemap Cache

Most current available high-res orthos

•

Mostly leaf-off •

Mostly includes orthos

restricted to

internal DNR

use•

Mostly a mix of 18-inch, 1-foot & 6-

inch orthos•

Cached at intermediate to large scales, mainly from MrSID

county

mosaics

Internal Image Basemap Cache

Conditioning of imagery for caching•

Need to minimize MrSID

“map collar”

artifacts •

2008 NAIP orthos

used where no

high-res

imagery available•

Edited County boundary feature class included for metadata purposes

1:506,880

1:23,760

1:1,980

1:990

Internal Image Basemap Cache

Minimize MrSID “map collar” artifacts

•

Attempt to render map collars transparent

•

Adjust drawing order in staging mxd•

“Despeckle”

black collars

•

“Patch”

most serious areas of white collar overdrawing

Future Image Basemap Caches

•

2010 NAIP Orthos•

2010 WROC 18-inch statewide public domain product

•

Topo

Basemap–

Including DRGs

& elevation contours

–

Alternatives being evaluated

•

Color Infrared Basemap–

Including Landsat

ETM+ CIR

–

2010 NAIP CIR–

DNR/FR CIR

imagery

Choosing Vector Data for Caching: Hydrography, Roads,

Public LandsCaching vector basemaps requires evaluating many alternative geodata sources …

and often adding themes many, many times to the Staging.mxd,

to set rendering and labelling properties appropriate to the chosen caching scales.

Choosing Vector Data for Caching: Hydrography, Roads,

Public LandsHydrography – Basically, 2 choices WDNR 1:24K Hydro, or USGS National Hydrography Dataset (NHD) 2.0

An overriding consideration is completeness of Stream Order, to facilitate generalization;

And completeness of Flow Direction, for large scale views.

NHD 2.0 is still not very ready …

Blue – Stream level coded

Pink – No stream level coded !

Choosing Vector Data for Caching: Hydrography, Roads,

Public LandsRoads required more mixing: •

Interstate & US Highways (3)

•

Major Roads (4) •

DOT WISLR County Roads (5)

•

Other Major Roads (6) •

Local Streets (7)

•

Additional Street Classes (8)

Choosing Vector Data for Caching: Hydrography, Roads,

Public Lands•

WDNR Managed Lands – many views

•

US Forest Service – from WI office•

US Fish & Wildlife Service – cadastral

•

National Park Service – gateway•

Other Federal Lands –

•

Tribal Lands – TIGER ? PAD ? GLIFWC ?•

County Forests – WDNR or PAD ?

•

Protected Areas Database – CBI (not USGS)

•

County & Municipal Parks – ESRI

Choosing Vector Data for Caching: Hydrography, Roads,

Public Lands•

Public Lands (0)

•

County Forests - 80K out (1) •

State Natural Areas - non-sensitive (2)

•

DNR Managed Lands (3) –

DNR Managed Lands (68K-150K) (4)

–

DNR Managed Lands (150K - 500K) (5) –

DNR Managed Lands (dissolved) - solid - 500K-2M (6)

–

DNR Managed Lands (dissolved) - solid - 2M Out (7)

Choosing Vector Data for Caching: Hydrography, Roads,

Public Lands•

Local Parks (8) –

Local Parks (1:150-800k) (9) –

Local Parks (40K-150k) (10) –

Local Parks (40k in) (11) •

Federal Lands (12) –

Native American Lands - PAD - all scales (13) –

National Forest Boundaries - 150K in - USFS (14) –

Nat’l Forest Boundaries - 150K - 1.3M USFS (15) –

National Forest Boundaries - 1.3M-3M ESRI (16) –

Natl. Forest Boundaries - 3M out Streetmap (17) –

Other Federal Lands – PADUS-CBI - 150K in (18) –

Other Federal Lands – PADUS-CBI - 150K out (19)

Design, Development, & Deployment of Wisconsin DNR

Map Services

Jeff Shaw, John Laedlein, Jerry SullivanWisconsin Department of Natural Resources

Bureau of Technology ServicesGIS

Services Section

Wisconsin Land Information Association - 2011 Annual Conference February 16-18, Madison, WI