scala e xchange. fredik ekholdt. introducing adept
DESCRIPTION
Dependency management should be something nobody should need to relate to, yet a surprising amount of time is spent on this aspect which should simply work. Adept is a new dependency management system in Scala, here to change the way it is done on the JVM. This talk will be about what Adept is, why you should care and how you can help.TRANSCRIPT
Dependency managers“a dependency manager automates the process ofusing and upgrading software libraries in a
consistent manner.”Finds the libraries you need, based on the libraries you want
A note on dependencymanagement
It does not have any intrinsic valueBut saves some time/effort
It is not interesting (boring)But sometime takes time (especially debugging)
Useful because it shares a decision you madeʹI want lib A version 2ʹ
We should update our libraries more often!So it should be easy to do
Introducing Adept“a person who is skilled or proficient at
something.”A dependency managerReliableFastKnows what works with whatOnly a dependency manager, nothing elsePlatform independent
State = identity(time,value)Identity: the library and the artifacts to be used (jars in junit,spring, ...)Value: the version to use (4.11, 3.2.5, ...)Time: the library as it is ʹright nowʹ (lib A 5.0‑SNAPSHOTtoday VS tomorrow)State: the version of a library at a given time
ReliabilitySame result every timeOn the JVM, dependency managers are sort of reliableWill state (the versions of the libraries at any given time)never change?
Versioned meta‑dataMeta‑data: defines the data (dependencies, names, ...)We want:Immutable meta‑dataThat is easy to changeAtomically change the entire meta‑data
In other words: Git
Distributed repositoriesDownload all meta‑data for each repositoryBenefitsYou are in control of changes (pull)You can go back in time (checkout)SpeedGit is supringsly good at compressingMaven central: 150 mbs of compressed data (once)
No round trips to fetch meta‑dataNo ʹinvalidʹ meta‑data cacheParellel download of artifacts
Artifacts and meta‑dataArtifacts: represents the actual files used (jars)When you want some libraries.....the dependency manager looks up the meta‑data
Dependency manager finds the libraries you need......based on the meta‑data
The artifacts you must download are therefore......defined in the meta‑data ‑ RIGHT?
Artifacts ‑ mark 1WRONG: Choosing a library in Ivy/MavenMeans a new lookup to some repository (or cache)Artifact = file that matches the same repository pattern (orcache)
Artifacts that can be chosen are:The ones supposed to be used based on some repositorymeta‑dataThe ones that happens to cachedA new artifact, but using cached meta‑dataA cached artifact, but using new meta‑data
Artifacts ‑ mark 2Cleanly seperated from meta‑data and repositoryMeta‑data points to the SHA‑256sArtifact lookup files: SHA‑256 and locationsThis means:New artifact means new meta‑dataNew meta‑data can safely use new/old artifact
Benfits:You are sure you got the right oneEasy to reason aboutVery easy to cacheSecure
SpeedWhen I say ʹa8dac15ʹ...... I do not care where it comes fromMeans I can:Use url to download jar from file server (nothing new)Check cache if jar is in the cache (nothting new)Use different urls to download the same jar (in pieces)Use bittorrent to download one jarUse bittorrent to download many jars...
State & AdeptSummary:You ask for the commit hash of...The repositories you need...And the libraries you requireWhich points a stable set of SHA‑256
No changes in state:For a given input, same artifacts every time
QED: Adept is reliable
Ivy/MavenAutomatically resolves to the ʹhighestʹ/ʹlatestʹ/... versionCompatiblity is not factored inThere is no... matrix
Users must know how libraries work together:Must specify version(s) (ranges) on how libraries arecompatibleOverride/exclude if there is a problem
Programs are good at thisDos not communicate intent
Example: sbt pluginsImagine if sbt:In addition to ʹversionʹ (0.12.0, 0.12.1, ...)Defines an extra attribute: ʹbinary‑versionʹ (0.12, ...)
A plugin (e.g. sbteclipse) that requires sbt specifies theʹbinary‑versionʹ
The user/build tool:I use/am sbt 0.12.0, what are the sbteclipse versions I canuse(Then chose the ʹbestʹ version)
sbteclipse version 1.0 requires sbt binary-version 0.12sbteclipse version 1.1 requires sbt binary-version 0.12 sbteclipse version 2.0 requires sbt binary-version 0.13
AttributesAdept allows resolution based on:versionbinary‑version
But also:Snapshot/prereleaseQA department ʹseal of approvalʹ...
Any attribute is a first‑class citizen
Adept valuesIvy/Maven: ʹversionʹ is the ʹvalueʹ of the libraryAdept: ʹversionʹ is a common attributeVariant == value in AdeptExample:sbt { version 0.12.0, binary‑version 0.12, qa‑tested false }sbt { version 0.12.0, binary‑version 0.12, qa‑tested true }
VariantsIdcom.typesafe.play/play
ArtifactsAttributesʹversion = 2.2.0ʹʹbinary‑version = 2.2ʹ...
DependenciesIdcom.typesafe.akka/akka‑actors
Constraintsʹbinary‑version = 2.2ʹ...
Adept resolution3 statesResolved (only one variant for each library)Under‑constrained (too many variants)Over‑constrained (conflicts, or non‑existing Id)
Naive algorithmGrab Ids of dependencies ......and all constraints for Id...for each resolved: repeat...
ResolvedVariants:
Dependencies: play 2.2.1 + akka 2.2.0 + scala‑library 2.10.3
scala-library {2.9.3, 2.10.2, 2.10.3}
akka 2.0.5 depends on scala 2.9.3akka 2.2.0 depends on scala 2.10akka 2.2.1 depends on scala 2.10
play 2.2.0 depends on {akka 2.2, scala 2.10}play 2.2.1 depends on {akka 2.2, scala 2.10}
Over‑constrainedVariants:
Dependencies: play 2.2.1 + akka 2.2.0 + scala‑library 2.9.3Or: foobar 2.2.1 + akka 2.2.0 + scala‑library 2.10.3
scala-library {2.9.3, 2.10.2, 2.10.3}
akka 2.0.5 depends on scala 2.9.3akka 2.2.0 depends on scala 2.10akka 2.2.1 depends on scala 2.10
play 2.2.0 depends on {akka 2.2, scala 2.10}play 2.2.1 depends on {akka 2.2, scala 2.10}
Under‑constrained???Variants:
Dependencies: akka 2.2.3 + scala‑library 2.10.3 + play (anyversion)Resolves: there is only exactly one possible play version: 2.2.1
scala-library {2.9.3, 2.10.2, 2.10.3}
akka 2.0.5 depends on scala 2.9.3akka 2.2.0 depends on scala 2.10akka 2.2.1 depends on scala 2.10
play 2.2.0 depends on {akka 2.2, scala 2.10}play 2.2.1 depends on {akka 2.2.3, scala 2.10} !!! ̂ !!!
Under‑constrainedVariants:
Dependencies: akka 2.2.1 + play 2.2.1 + any 2.10 scala‑library?Or: akka 2.2.1 + scala‑library 2.10.3 + any 2.2 play?
scala-library {2.9.3, 2.10.2, 2.10.3}
akka 2.0.5 depends on scala 2.9.3akka 2.2.0 depends on scala 2.10akka 2.2.1 depends on scala 2.10
play 2.2.0 depends on {akka 2.2, scala 2.10}play 2.2.1 depends on {akka 2.2, scala 2.10}
Architecture: ExtensionsAdept does not do much!Ideal: I declare only binary‑versions and commit,Build tools gives me the ʹhighestʹ versions compatible
Build tools needs Adept + some extensionsSmall core = minimal changesDifferent extensions, but consistent resolution
Extensions:Conventions/attribute names and meaningVersion, binary‑versionVersion ordering
OverridesExclusionsImports (Ivy/Maven/...)...
Overrides/exclusionsNeed to resolve ʹbenignʹ conflicts (Versions differ, but not binary‑versions)Creates a new variant with:Different dependenciesUnique ʹoverridesʹ/ʹexcludesʹ attributes
The social dependency manager?I can figure how my library is used
Other usecasesSnapshots #1
Use ʹprereleaseʹ/ʹsnapshotʹ/... attributeOverwrite variant + commit new versionUpdate meta‑data = new snapshotFlexible ......but you know exactly which artifact/meta‑data that wasused
Other usecasesSnapshots #2
CI & nightlies:We know which version we used:Safely update metadata after passing tests
User use same constraints (even when stable is released)But build tool can give warning
No SNAPSHOTs after a release, and......no failures because a ʹversionʹ has been removed
What worksResolutionʹEqualityʹ constraintbinary‑version 1.0 == 1.0
Extensions:Overrides, excludesVersion orderingIvy imports
Tests (test framework)
Help!Final design!Now is the time to set things straight
Handle libraries that defines the same interfaceAttribute types & comparable constraints? (types, !=, >, <, ...)Meta‑data is not (de)serializedArtifacts files are not (de)serializedMore tests/use cases
Help!!Git is not implementedNo CLI (useful for testing)No built tool plugins (we need: sbt, gradle, SBuild, ant,maven?)Resolution is concise (but could even more concise?)Resolution is fast (but could be faster?)
Ideal workflow1. Figure the library you want2. Choose only version(s) you want to be compatible with3. Repeat
Adepthub #1Online resolution using AdeptHosts meta‑dataWorkflow:1. Go to adepthub.com2. Search and add new (compatible) libraries3. Save (creates a perma link)4. User/plugin/build tool downloads one file with:
Perma link, if you want to updateLocations (urls) and some human readable meta‑datafor all artifacts
Adepthub benefitsFor you:Uses Adept and hosts Adept repositoriesSearch across all repositories using:Keywords on IdProject info: github url, author, company, ...
No need to download meta‑dataHowever, it is possible to do it if you want
Import (meta‑data enhancement)Resolution is done online......but can be consumed by maven, ivy, ...= no plugins required
Adepthub benefitsFor library authors:Easy to publishGithub interfacePublic & private repositoriesCan have nice stats:Who is using version X anyways? Now we know!
Trophies: show your awesomenessFor me:Easier to build pluginsRead file + download artifacts
Launches:When it is done, hopefully February/March
FeedbackIs search.maven.org good enough?Please show support by signing up for beta:http://adepthub.com/#beta
Interested?
(README links to: mailing list/spec)https://github.com/adept‑dm/adept