apache yetus: helping solve the last mile problem

Apache Yetus: Helping Solve the Last Mile Problem

Allen Wittenauer Effective Machines

Apache: Big Data North America 2016-05 ApacheCon North America 2016-05

! email: aw @ effectivemachines.com, aw @ apache.org ! github: aw-was-here ! gitlab: _a__w_ ! slideshare: allenwittenauer ! twitter: @_a__w_ ( 1 a 2 w 1 )

22016-05-12

https://www.flickr.com/photos/83633410@N07/7658225516/ 52016-05-12

! “How do these changes actually impact me?”

! “The method my code uses changed!”

! “The quality of this software is awful. Is it actually tested?”

62016-05-12

Apache Yetus is a collection of libraries and tools that enable contribution and release processes

for software projects.

Release Doc Maker

92016-05-12

releasedocmaker

! Input ! One or more JIRA projects ! Version, multiple versions, or a range of versions ! Optional ! Title !Project Name

! Linting

!Output ! Changes and Release Notes !Per version !Grouped by Type and Flags (Important, Incompatible)

! Markdown

112016-05-12

Audience Annotations for Java

What’s “safe” to use?

Compiler vs. Human

Audience

!Private

! Limited-Private

!Public

212016-05-12

Stability

!Stable

!Evolving

!Unstable

!Deprecated

222016-05-12

@[email protected] class GlobFilter implements PathFilter {...}

@InterfaceAudience.LimitedPrivate({"MapReduce"})@InterfaceStability.Unstablepublic class Progress {...}

import org.apache.yetus.audience.InterfaceAudience

import org.apache.yetus.audience.InterfaceStability

232016-05-12

JavaDoc/... Support

! ...audience.tools.ExcludePrivateAnnotationsStandardDoclet

! ...audience.tools.ExcludePrivateAnnotationsJDiffDoclet

! ...audience.tools.IncludePublicAnnotationsStandardDoclet

242016-05-12

Shelldocs

HADOOP-9902

## @description Print a message to stderr## @audience public## @stability stable## @replaceable no## @param stringfunction yetus_error{ echo "$*" 1>&2}

272016-05-12

smart-apply-patch

30

smart-apply-patch --plugins=github \ --github-user=apache --github-repo=orc 55

smart-apply-patch --plugins=jira HADOOP-1

smart-apply-patch https://my/cool/server/patch

Committer Mode (--committer)

!Only works with git format-patch created patches

!Always --commit!Always --signoff!Always --whitespace=fix

! TODO: Add more support for other patch formats (YETUS-158) ! TODO: Add --sign (YETUS-315)

312016-05-12

precommit (aka test-patch)

Driver

Personality Bug Systems Build Tools Tests

Language Support Static Analysis

Unit Tests

Test Log Processing

40

!Compiled Language Support – C/C++ – Java – Scala

! Interpreted Language Support – bash/ksh/sh: shellcheck – perl: Perl::Critic – Python: pylint – Ruby: rubocop, ruby-lint

Tool Support (part 1)

! Build Tools – ant – autoconf – cmake – gradle – maven – make – “nobuild”

! Bug Systems – Bugzilla (RO) – GitHub – JIRA

412016-05-12

! Unit Test Formats –ctest –Junit –TAP

! Runtime Support –Automation (e.g., Jenkins) –Docker –Multiple JVMs –Patch branches –Per-instance maven repos

Tool Support (part 2)

! Specialized Tests – Apache RAT –@author –Java checkstyle –Javadoc –Java findbugs –maven site –Scaladoc –whitespace –XML

422016-05-12

452016-05-12

test-patch --plugins=all \--github-user=apache --github-repo=tajo \--jira-re='^TAJO-[0-9]+$'--branch-default=master \--buildtool=cmake --basedir=tajo \TAJO-1

Personality

462016-05-12

#!/usr/bin/env bash

personality_plugins "all"

function personality_globals{ BUILDTOOL="maven"

PATCH_BRANCH_DEFAULT=masterJIRA_ISSUE_RE='^TAJO-[0-9]+$'GITHUB_REPO="apache/tajo"

}

Running

test-patch --personality=file --basedir=tajo TAJO-1

test-patch --personality=file --basedir=tajo \https://example.com/file-branch.patch

test-patch --personality=file --basedir=tajo /tmp/patchfile

smart-apply-patch --personality=file --basedir=tajo TAJO-1

472016-05-12

Bundled Sample Personalities

482016-05-12

qbt (quality build tool)

https://www.flickr.com/photos/83633410@N07/7658230838/ 522016-05-12

https://www.flickr.com/photos/83633410@N07/7658230838/

Administrivia

!Website: https://yetus.apache.org

!Mailing list: [email protected]

! Twitter: @ApacheYetus

!Current Release: 0.2.1 (2016-04-14)

!Yetus is an archaic synonym of the Cymbium genus of gastropods

! First ASF TLP with a ‘Y’

532016-05-12

apache yetus: helping solve the last mile problem

Software