harnessing the power of python in arcgis using the conda ...€¦ · 2016 esri developer...
TRANSCRIPT
HarnessingthePowerofPythoninArcGISUsingtheCondaDistributionShaunWalbridge
MarkJanikas
TingLee
https://github.com/scw/conda-devsummit-2016-talk
HandoutPDF
HighQualityPDF(2MB)
Conda
CondaBrandnew:thoughtitwasmoreimportanttoshowittoyouthantofocusontellingyouaboutitTimetodaytodiscussyourneedsandwhatwemightdotosolveyourproblems
WhyPython?Accessiblefornew-comers,andthe
Extensivepackagecollection(56thousandon ),broaduser-baseStronggluelanguageusedtobindtogethermanyenvironments,bothopensourceandcommercialOpensourcewithliberallicense—dowhatyouwant
mosttaughtfirstlanguageinUSuniversites
PyPI
PackageManagementforPython
Whynotpip,wheels,virtualenvs?Don’thandletheharderproblemofsystemdependencies,consideredoutofscopebyPythonpackagers—doesitendupinsite-packages?Packagedevs:OnOSXandLinux,‘easy’togetthedeps!Useasystempackagemanager(e.g.apt,brew,yum)andtheincludedcompiler(e.g.clang,gcc).It’sstillnoteasytomakereproduciblebuilds,andwhataboutWindows?
WhataboutWindows?WeareparticularlystuckonWindowswhichlacksbroadlyusedpackagemanagement
NuGetisgreat,butnotasystem-levelpackagemanagerIfmanagingapplications,try
OnlydevshaveaCcompilerontheirmachineTheessentialmodeliscompilersforfew,runtimesforall
Packagemanagementishard!(ExceptonJavaScript–universalcompilersarealeg-up)
Chocolatey
WhataboutWindows?WeareparticularlystuckonWindowswhichlacksbroadlyusedpackagemanagement
NuGetisgreat,butnotasystem-levelpackagemanagerIfmanagingapplications,try
OnlydevshaveaCcompilerontheirmachineTheessentialmodeliscompilersforfew,runtimesforall
Packagemanagementishard!(ExceptonJavaScript–universalcompilersarealeg-up)
EnterConda
Chocolatey
WhyConda?
ScientificPythoncommunityidentifiedthattherewasagapnotbeingaddressedbythecorePythoninfrastructure,limitingtheirabilitytogetpackagesintothehandsofusers
Industrystandardbuiltbypeoplewhocareaboutthisspace–ContinuumAnalytics
WhyConda?
Itsolvesahardproblem:
Handlesdependenciesformanylanguages(C,C++,RandofcoursePython)
BuiltforPythonfirst,butitreallysolvesamuchbroaderinfrastructuralissue.
Conda
CondaCross-platform:simplydeveloprecipesforbuildingandinstallingsoftwareonLinux,OSXandWindows.Allittakes:ameta.yaml,andabuildrecipe.Opensource(BSD):Esriisusingit,youcanuseitinyourownprojectsforothercontexts
CondaWhatcanitinstall?Notjustscientificpackages.Itcanhelpwith:
GUItoolkits(PyQt,TKinter)C++Libraries(Boost)IDEs(Spyder,Juptyer)
See foracomprehensivesetofbuildrecipes.EverythingfromapplicationstocompilerstoPythonmodules,hundredsofmaintainedrecipesacrossmanyproblemdomains.
conda-recipes
CondaEnvironments:CanisolateaPythonenvironment,flexiblymakechangeswithotaffectinginstalledsoftware.Requirements–includeexplicitstateinformation,notjustthepackagename.Namesaren’tenough!AlsohandlesplatformsandJupyternotebooks
HowDoesitWork?Condapackagescancomefromavarietyoflocations:
Ondisk(file://)PublicrepositorieshostedonAnacondaCloudPublicrepositoriesself-hostedPrivaterepositoriesPaidprivaterepositories
CondaBasicsCommandlineinterface
Willshowwhatwe’reworkingontomakethiseasier,especiallyfornon-developers
CondaCheatsheet
CondaBasicsTostart:
AcollectionofpackagesandPythoninstalliscalledanenvironmentorenv,thebuildingblockformanagingPythonwithCondaCanhavemultipleenvironmentsandseamlesslyswitchbetweenthem
conda --help
CondaBasicsActivatingenvironments,acoupleways:
UsetheshortcutsManuallyactivatetheenvironment:
cd /d C:\ArcGIS\bin\Python\Scripts activate arcgispro-py3
CondaBasicsOnceyou’reinanenvironmentgetdetailswithinfo:
Condainfoisthestartingpoint–ittellsyouthestateoftheenvironment.
conda info
CondaBasicsconda info
Current conda install:
platform : win-64 conda version : 4.0.4 conda-build version : not installed python version : 3.5.1.final.0 requests version : 2.9.1 root environment : C:\ArcGIS\bin\Python (writable) default environment : C:\ArcGIS\bin\Python\envs\arcgispro-py3 envs directories : C:\ArcGIS\bin\Python\envs package cache : C:\ArcGIS\bin\Python\pkgs channel URLs : https://conda.anaconda.org/esri/win-64/ https://conda.anaconda.org/esri/noarch/ https://repo.continuum.io/pkgs/free/win-64/ https://repo.continuum.io/pkgs/free/noarch/ config file : C:\ArcGIS\bin\Python\.condarc
CondaBasicsconda list
# packages in environment at C:\ArcGIS\bin\Python\envs\arcgispro-py3:#arcgispro 1.0 0 esrimatplotlib 1.4.3 np19py34_0 defaultsnose 1.3.7 py34_0 defaultsnumpy 1.9.3 py34_0e [arcgispro] esripandas 0.17.1 np19py34_0 esripip 8.0.3 py34_0 defaultspyparsing 2.0.3 py34_0 defaultspypdf2 1.25.1 py_0 esripython 3.4.4 2 defaultspython-dateutil 2.4.2 py34_0 defaultspytz 2015.7 py34_0 defaultsscipy 0.16.1 np19py34_0e [arcgispro] esrisetuptools 20.1.1 py34_0 defaultssix 1.10.0 py34_0 defaultssympy 0.7.6.1 py34_0 defaultswheel 0.29.0 py34_0 defaultsxlrd 0.9.4 py34_0 defaultsxlwt 1.0.0 py34_0 defaults
CondaBasicsCreatingnewenvironments:
Afewdifferentways.Canmanuallyspecifythedependencies:
Canalsouseafilewhichincludesallthedependencies:
Thesecancontainexplcitinformationaboutchannels,toensurethatthenewenvironmentpreciselymatchestherequirements.
conda create --name my_env python=3.4 numpy flask dask
conda create --name my_env --file my_sweet_depends.txt
Condavs…Name Means Will
Ship?
Conda Thecommanditself ✓Miniconda AminimumsetofPythonpackagesto
buildandrunConda.✓
Anaconda Adistribution200+packagesbuiltwithConda
AnacondaServer
Hostthefullinfrastructureinternally
CondaDemo
DeeperDive
CondaBehindFirewallHow’sitwork?
Lockitdown:Don’tusenetwork
Canvettheinstallation
Willworkoutoftheboxwithdefaultpackageswithoutanynetworkconnectivity
.condarcModifydefaultswithasimplesimpleYAMLfileforconfigurationCanbeupdatedwithconda config,justlikeusinggit configtoupdatethedefaultconfiguration
Adetailedexample.condarc
CreatingpackagesStraightforward:
Ametadatadocument(meta.yaml)specifyingthecontentsanddependenciesAbuildcommand(bld.bat,build.sh)specifyinghowtobuild
Creatingpackagesmeta.yaml:
package: name: pypdf2 version: "1.25.1"
source: fn: PyPDF2-1.25.1.tar.gz url: https://pypi.python.org/packages/source/P/PyPDF2/PyPDF2-1.25.1.tar.gz md5: ee5e5b01d00b120805e5049e56c6fd7c
requirements: run: - python
Creatingpackagesbld.bat:
"%PYTHON%" setup.py install
MultiplePythonsCurrently:
Platform Pythonversion
Desktop Python2.7.x(2.7.10)
Pro Python3.4.x(3.4.3)
MultiplePythonsUpgradecode?
Doitalready!Youcansupport2+3withoutthatmuchworkIfyouhitanissue,it’sprobablybecauseyoudon’tunderstandUnicodeyet– ,PragmaticUnicode,or,HowdoIstopthepain?
PythonmigrationforArcGISPro
WatchthisPyContalk
MultiplePythonsUpgradecode?
Doitalready!Youcansupport2+3withoutthatmuchworkIfyouhitanissue,it’sprobablybecauseyoudon’tunderstandUnicodeyet– ,PragmaticUnicode,or,HowdoIstopthepain?
But…thiscanbecostly.Formanyorganizations,asignificantburden,evenifthelanguagechangesarerelativelysmall.
PythonmigrationforArcGISPro
WatchthisPyContalk
MultiplePythonswithConda
WithConda,wecansupportmultipleplatforms:
Py2.7,3.4,3.5inPro1.3
Createanewenvironment,targetadifferentPython,userscannowusethatwiththePy2code
Stillneedtochangearcpy.mappingtoarcpy.mpwhenmovingfromDesktoptoPro,butnoPythonlanguagelevelchangesneeded.
ChallengesHavetomakesureyou’rerunningtherightPython(whathappens
whenyoutypepythonatthecommandline?)WewillmakethiseasyaspossibleIt’llbeeasytotellinappIsolatedinstallationfixesavarietyofissues
Requiressomeusereducationoverthe“onlyonePythononthebox”model
WhatDoIGetOutoftheBox?
CondacommandandaCondarootPythoninstallNewmodules(e.g.requests)CondaenvironmentwithalloftheArcGISProdependenciesasCondapackages
HowcanIusethis?WealreadyshipyoutheSciPystack–powerfulandoutofthebox,canusetoday(Proand10.4)Canstartusingcondatoday.Minicondaisfullystand-alone,won’taffectyourglobalPython(unlessyoutellitto)Packageyourwork:thisisanopportunitytodistributeit,possiblyincludingcommercialsideaswell.
WhereCanIRunThis?
ArcGISPro1.3(Release:2016UC)WillbethePythoninstall.UIforinteraction
Future:TakeadvantageofmorefeaturesIntegrationwithplatform
fromfutureimport*EffectivelymanagecomplexsoftwaredependencieswithConda.Thousandsofpackagesexisttoday,canintegrateitintoyour
organization’sneeds.
Resources
ResourcesCondaRecipes
Anaconda.org
CondaCheatsheet
Closing
ThanksEsriCondaTeam:
ContinuumAnalyticsforcreatingandopensourcingConda
RateThisSessioniOS,Android:Feedbackfromwithintheapp
WindowsPhone,don’tuseasmartphone?:Cuniformtabletsaccepted(sorry!limitation).
RateThisSessioniOS,Android:Feedbackfromwithintheapp
WindowsPhone,don’tuseasmartphone?:Cuniformtabletsaccepted(sorry!limitation).
WindowsPhone,ornosmartphone?Cuneiformtabletsaccepted.