CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution** All unlicensed or borrowed works retain their original licenses

SOTS v4State of the Stack May 20th, 2015

OpenStack Summit, Spring 2015

@randybias

With significant help from many Cloudscalers and EMCers. Thank you!

The Randy Bias• Built big clouds; production clouds

• An OpenStack Original

• part of launch in 2010, on Foundation Board since formation

• built some of the largest and earliest OpenStack clouds

• Top <insert number here> cloud/twitter/pioneer/visionary

• you pick…

2

What Winning Looks Like

3

Fastest Growing Open Src Community

4

COMPANIES

TOTAL DEVELOPERS AVERAGE MONTHLY CONTRIBUTORS

TOTAL CODE CONTRIBUTIONS

3,654 >600 125,000+

502TOP 10 COUNTRIES

27,398INDIVIDUAL MEMBERS

COUNTRIES

140+United States, India, China, United Kingdom, France, Russia, Canada, Germany, Japan, Australia

Now That’s Something

5

OpenStackCloudStackEucalyptusVMware vSphere

Source: trends.google.com

Active Contributors in The Community

6

Qingye Jiang (John) - Open Source IaaS Community Analysis CY15 - Q1http://www.qyjohn.net/?p=3801

Monthly Commit Volume

7

Qingye Jiang (John) - Open Source IaaS Community Analysis CY15 - Q1http://www.qyjohn.net/?p=3801

Accumulated Developers

8

Qingye Jiang (John) - Open Source IaaS Community Analysis CY15 - Q1http://www.qyjohn.net/?p=3801

9

Source: LinkedIn Groups via RedMonk analysis

AAWS Community Network Azure OpenStack

Why Did We Win?

10

Explosive Growth

What’s Also Very Dangerous?

11

Explosive Growth

Let’s Review

12

13

Infrastructure as a ServiceCompute

Nova Ironic

Magnum

NetworkNeutron (LBaaS)

(VPNaaS) (FWaaS)

StorageSwift

Cinder Manila

Cloud Management

Telemetry Ceilometer

Deployment Triple O

Orchestration Heat

Test Suites Tempest

Rally

Advanced Services (Consume IaaS)

Image Management: Glance

Data Processing

Sahara

Key Management

Barbican

DNS Management

Designate

Database Management

Trove

Message Queue Zaqar

Service Catalog Murano

Workflow Management

Mistral

Policy Management

Congress

Common/Shared: Identity: Keystone Common Libraries: Oslo

User/Admin

UI API CLIKilo

OpenStack Interdependence

14

– Me / Ako / Moi / Yo *

“OpenStack is at risk of collapsing under its own weight.”

15* http://www.cloudscaling.com/blog/openstack/the-future-of-openstack-is-now-2015/

Lots of Improvements• Product WG formed

• Create an aggregation point for longer term planning, bring user feedback into process, prioritization of blueprints, lobbying TC and PTLs for work queues, “funding” of key blueprints, etc.

• Integrated release & 6-month cycle reformed to “Big Tent” approach

• No more forced 6-month integration, more project autonomy, encouragement of 3rd party integration testing of drivers, tagging for release, etc.

16

Product Working Group

17

User Committee N+3 members: 3 selected by the board, the TC and an additional nominated representative. An additional N

members elected by the user community.

Enterprise

Focused teams to gather user requirements from segments and represent them

Telco / OPNFV

Application Ecosystem

Large Deployments

API Working Group

Working Groups to address a particular requirement set. These WGs should have a target set of deliverables and conclude when those are met. Maintenance should be a

function of the regular workflows.

Logging

Ops Tools

Monitoring

HPC

Product Working Group Gather requirements from both sets of WGs (Segment and Requirement Oriented) above in the form of user

stories, work with cross-project team to populate blueprints from user stories across projects, work to identify developers to help complete blueprints, communicate with project PTLs and core team to collect feedback on future directions, and compile this data into a multi-release roadmap that is publicly available. In summary,

facilitate a feedback loop between projects, user community, and working groups.

Multi-Release Roadmap

New

“Big Tent” Release Cycle Reform

18

Solving for “How do we allow for the additional projects in the future without breaking down?”

(Current) Tag Categories:

Release Team

Tag Descriptionintegrated-release Frozen tag, not given to new projects. Identifies projects that were integrated prior to Kilo.release: indepdent Projects with this tag “release as needed” and don’t have to coordinate with other projects.

release: at-6mo-cycle-end Projects that commit to being a part of a coordinated release every 6 months. They can still have intermediate releases independent of the 6 month cycle “final” release.

release: has-stable-branches Projects that have stable branches (from the last release in the cycle)release: managed Projects that agree to follow the processes/timelines outlined by the OpenStack Release Management Team

team: diverse-affiliation This tag shows that the developer team for the project is from a diverse set of organizations (1 < 50% and 2 < 80%). This is tested every 6 months.

Details at http://governance.openstack.org/reference/tags

(Current) Tags:

It’s not enough …

19

Complexity Kills

20

Enterprise Cloud Maturity

21

Most enterprises in the “thick middle” of maturity cycle

The “Thick Middle” Struggling

22

Docker is Simpler

.. And Adopted Faster:

3M downloads in H1’2014 100M downloads at end of 2014

23

Reality Check Time

24

Technology Adoption Curve 101

25

Technology Adoption Curve 101

25

We are roughly here!

Technology Adoption Curve 101

25

We are roughly here!

… and headed for here!

26

How Do We Know?• Growing skepticism from analysts, reporters, and pundits

• Growing dissatisfaction with certain aspects of OpenStack

• Lots of failures in the field, enough to be worrisome

• Peak OpenStack?

• 6K+ attendees, early signs of slow down in adoption?

• Decide for yourself; I could be calling it early

27

1

2

3

The Growing Skepticism

28

Linthicum believes that despite the fact that OpenStack has "the only game in town" for open source, the implementation hasn't met up to all of the hoopla since

its release. http://searchcloudcomputing.techtarget.com/podcast/OpenStack-talk-of-open-source-town-but-is-it-hype

OpenStack can run a fine private cloud, if you have lots of people to throw at the project and are willing to do lots of coding, according to Alan Waite, a research

director at Gartner.

OpenStack has the following drawbacks as a platform on which to build a private cloud*:  1 Difficulty of implementation 2 Shortage of skills available in the market 3 Conflicting or uncoordinated project governance 4 Weak spots in some projects 5 Integration with existing infrastructure *Recent Q1’2015 Gartner Report

OpenStack Self-Improvement Survey• Intention:

• determine if and where project dissatisfaction exists

• report back to provide perspective on where we need to change

• After 10 days:

• 65+ respondents w/ 30 months average time with OpenStack

• Survey: http://tinyurl.com/improve-openstack [ TAKE ME! ]

29

How Would Your Characterize Your Participation in OpenStack Land?

30

19%

16%

30%

35%

OpenStack DeveloperOpenStack Operator/AdministratorOpenStack End-User (MIA)Pundit, Analyst, Reporter, OpenStack Evangelist, or GroupieOther

Average Time Working With OpenStack: 32 months

What is Your MOST favorite OpenStack Project?

31

NovaSwiftHeat

KeystoneNeutron

IronicCinder

DesignateCeilometer

TroveBarbican

GlanceHorizonManila

OsloSaharaTripleOZaqarOther

0 4 8 12 16Responses

What is Your LEAST favorite OpenStack Project?

32

CeilometerNeutronTripleOCinder

HorizonOslo

GlanceKeystone

NovaHeat

IronicSahara

SwiftTroveZaqar

BarbicanDesignate

ManilaOther

0 4 8 12 16Responses

User Survey Feedback• Neutron:

• “Neutron is a lot more complex and harder to provide real HA” – Survey Respondent

• “Complexity, availability and scalability remain some of the concerns [ of the operators during the Operator Meetup in March ]” – User Survey Team

• Ceilometer: • “adoption has not been rising as quickly as expected … dozens of

comments related to stability and reliability, particularly at scale.” – User Survey Team

33

User Survey: Neutron

34

Well run technology organizations will often throw away or re-architect v1 and even v2 products.  Do you think this is a good practice?

35

19%

81%

Yes No

Why can’t we fix these?It’s been years now…

36

OpenStack Threats• Explosive growth drives complexity

• Continued complexity slows adoption

• Can’t simplify, kill, or re-architect “broken” projects

• Rigid technical governance model (still too centralized)

• Long term vision & product strategy doesn’t emerge (in process)

37

Towards Glory

38

Technology Adoption Curve 101

39

Technology Adoption Curve 101

39

Need to get here!!

Path to the Plateau of Productivity

40

Plan Item Objective

#1) Streamlining Governance Model empower projects, scale TC, focus Product WG, focus Board and Foundation on marketing and interoperability

#2) Allow Competition force poor projects to evolve or die, allow other projects, particularly non-Python to come under our “big tent”

#3) Conform to Well Known APIs don’t create new APIs in places where they exist (e.g. OAuth 2.0)

#4) Testable Reference Architecturesallow for vertical and horizontal-specific OpenStack

reference implementations and separate infrastructure from platform

#5) Ruthless Simplificationdownloadable “OpenStack Basic IaaS” should be 1-click

download and install to run a POC/trial on a simple stack (1 switch, 10 servers)

41

18 Categories (including retired), 252 Projects

The ASF Scales

Source: http://apache.org/foundation/governance/orgchart

To Manage This… You Need This.

Allow Competing Projects & Multiple Languages

• Competition is good; pretending our shit doesn’t stink is bad • Poor projects must die; survival of the fittest works • There is already leeway for this:

• “Where it makes sense, the project cooperates with existing projects rather than gratuitously competing or reinventing the wheel.”*

• i.e. Competitive projects are OK as long as they have good reason

• Python isn’t good for everything • A bigger tent means allowing non-Python projects • Swift is already experimenting with re-writing pieces in Go Language (golang)

42Source: http://governance.openstack.org/reference/new-projects-requirements.html

–Thierry Carrez, Chairman of the TC, Foundation Release Manager

“OpenStack is about community, common values, and a common governance model.”

43

Keystone API• Seriously … WTF?

• There are dozens of well known, documented, scalable, tested, standard APIs for authN/authZ

• OAuth1/2, SAML, Kerberos

• There is no excuse for creating something from whole cloth

• Google is secure as hell and they use OAuth 2.0

• You aren’t better at security than the Google team; sorry

• We don’t apply this standard to our community (completely new Nova API anyone?)

44

Example Reference Architectures

45

OpenStack Interop Standard RA “Key” Components RA Optional Components

Basic IaaS 1+ of Nova/Magnum/Ironic OAuth 2.0 Server (Keystone or other) Glance, Horizon

Advanced IaaSOpenStack Basic IaaS Cinder, Swift Neutron (or an alternative?) OAuth 2.0 Server (Keystone or other)

Glance, Horizon

OpenStack AppServices Zaqar, Trove, Designate, OAuth2.0 Horizon

OpenStack AppManagement Heat, Murano, Mistral, Horizon, OAuth2.0 Horizon

OpenStack for NFVBasic IaaS Pluggable SDN Controller w/ Neutron APIs

OpenStack Public Cloud Advanced IaaS + OpenStack App Svcs + OpenStack App Mgmt ec2api, gce-api, etc.

P2 tests

How it Might Work

46

Reference Architecture

Default Config Opts

Key + Optional Projects { Project 1

Project 2

Interoperability Test Suite

Defined “Capabilities” (previously “DefCore”)

RefStack

P1 tests

Capabilities Tests API Code

Owner(s): Infrastructure Team & Working Groups

TC, Board, Vertical/Horizontal Working Groups, Community, &

Foundation

PTLs & key committers/reviewers (more like Apache PMC??)

Unit Tests

We Are t3h Borg. You Will Be Assimilated.

47

“Maybe I’m an idiot, but I have no idea what anyone is talking about. What is it? It’s complete gibberish. It’s insane. When is this idiocy

going to stop?” Larry Ellison on Cloud

ComputerWorld, July 2000

"This 'telephone' has too many shortcomings to be seriously considered as a means of

communication. The device is inherently of no value to us.”, Western Union internal memo, 1876.

Decca Records rejected the Beatles, saying "guitar groups are on the way out" and "The Beatles have no future in show business,"

We Can Do It!• Interrelated, but not interdependent projects

• Testable reference architectures that are interoperable

• Streamline governance

• Survival of the fittest project and programming language

• OpenStack is not specific code or APIs, it’s:

• Community, common values, and common governance

48