lessons learned from xen (texas linux fest 2013)

How to (Almost) Kill a Successful Project and Bring It Back to Life Again

Russell Pavlicek

Xen Project Evangelist

Citrix Systems

Russell.Pavlicek@XenProject.org

Lessons Learned from the Xen Project

A guy with a lot of experience and a really big mouth

So Who is the Old, Fat Geek Up Front?

• Linux user since 1995; Linux desktop since 1997

• Linux advocate before I ever saw the software

• Early advocate in DEC, Compaq

• Former columnist for Infoworld, Processor

• Former panelist on The Linux Show

• Wrote book, Embracing Insanity: Open Source Software Development

• Speaker at 40+ conferences

• Currently Xen Project Evangelist employed by Citrix

About the Speaker...

• We will spend a couple minutes reviewing the project

• We will spend a few minutes considering its history

• But we will spend the bulk of the time considering lessons to take away

• We are not here for the project's history; we are here for your future

About This Talk

• The premier Open Source Hypervisor

• Powering some of the biggest Clouds in the industry (Amazon, Rackspace, Terremark)

• Celebrating its 10th Anniversary

• Now a Linux Foundation Collaborative Project

– Sponsoring organizations include: Google, AMD, Intel, Samsung, Cisco, Oracle, Amazon, Verizon and more

What is the Xen Project?

• The Xen Hypervisor, including ARM servers

– Type 1 (Bare Metal)

• Xen Cloud Platform & XAPI

– Cloud readiness out of the box

• Other Projects

– Mirage

– ARM Hypervisor for mobile devices

What Does the Xen Project Produce?

• It was the first industrial-strength Open Source Hypervisor

• It enjoyed a very high rate of adoption

• It employed excellent technology

• It had a FOSS-friendly corporation behind it

• And, yet, 2 years ago, it ran the risk of being abandoned by the FOSS community before its 10th birthday

The Xen Project Story (30 second version)

• The project, though viable, developed an inward focus

– Reach out to the rest of the Open Source community was limited

– Reach out to its users was minimal

– Code development continued, but the community became insulated and stagnated

– No one stepped up to contradict the rumor that Xen was dying technology, overcome by competitors

How Did This Happen?

• The project forgot the importance of working with its ecosystem

– Upstream projects (Linux Kernel, QEMU) were branched rather than engaged with patches

– The project decided that others in the ecosystem (i.e., the distributions) would have to carry the burden of maintaining and supporting those differences

– This went on too long, and the ecosystem got fed up

How Did This Happen? (continued)

• The corporation backing it (XenSource) was sold to a company with a long closed source software history (Citrix)

• The new corporation was interested in the technology, but had no particular interest in the project itself

How Did This Happen? (continued)

• It was not about malice

• It was not about fear

• It was about disconnection

– The project became disconnected from the FOSS community

– The project became disconnected from the users

– The new company became disconnected from the needs of the project, because, in part, the project never really explained what it needed from the company

Why Did This Happen?

• The prognosis was not good

• Xen Hypervisor had been overtaken by a commercial offering in IT mindshare

• Xen Project had been overshadowed by another Open Source Hypervisor in the community

• Distributions stopped facilitating Xen

• The FOSS Community began to forget Xen

About Two Years Ago: Battling for a Future

• About 2 years ago, a new direction was plotted

• Citrix decided it wanted to understand FOSS and reinvigorate the Xen community

• The company began to hire FOSS-savvy people to reconnect with the community and with users

• Brought about efforts to birth Apache CloudStack, OpenDaylight, and to move the Xen Project under the Linux Foundation

A Conscious Reversal in Direction

• Common themes heard at FOSS events:

– What is Xen?

– Xen is dead, right?

– Isn't Xen closed source?

– No one uses Xen anymore

Reality Two Years Ago: Xen Who?

• Linux kernel 3.0 contains all that Xen needs to exist by default

• Most Linux distributions are Xen-enabled

– CentOS has a project to give RHEL6 users a Xen option

• Xen Project now part of Linux Foundation

• Launch of a new user-friendlier website (XenProject.org)

Reality Today: Xen is Back!

So What Did We Learn?

• It is possible to die while you are winning

– Being first is not enough

– Great technology is not enough

– Having a FOSS-friendly corporation behind you is not enough

• A project must stay vibrant as an Open Source organism or it will perish

Lesson 1

• Disconnection from users can make you a “Dead Project Walking”

– You can be adding functionality, issuing new releases, and still be dying

– The connection between project and users is essential

– Focusing on software alone is not enough

• If you are not interacting with your users, you are at serious risk

Lesson 2

• Connecting with your developers != connecting with your users

– You need information sources for both users and developers

– If users have to dig through technical websites, wikis, etc. to answer simple questions, you are in trouble

– Even Linux kernel development – arguably one of the most insular projects – cannot thrive in a vacuum

Lesson 2 (continued)

• Never ignore your project's Open Source root structure

– Cut flowers are beautiful – until they die

– Living plants need their roots

– FOSS community is the root structure, and it must spread wide

– The project team cannot stand alone

• Pay attention to your partners in FOSS: libraries, kernel, packaging

Lesson 3

• Never ignore your support structure

– Xen needed cooperation from Distributions to be properly supported

– The relationship with the distributions was allowed to stagnate; it was not continuously cultivated

– When one distribution invested in another Open Source virtualization solution, other distributions were swayed

• Your distribution route can be critical to success

Lesson 4

• Having corporate backing is not enough

– The corporation has its own set of goals, and they rarely align exactly with the project's goals

– When the project and the company don't mesh, friction can occur

– This isn’t about good versus evil; business and projects are just two separate animals with different needs

Lesson 5

• Having a FOSS company backing you is no guarantee

– Even FOSS-centric companies can be sold

– Sometimes they are sold to other FOSS companies (e.g., JBoss, Gluster)

– Sometimes they are sold to closed source companies (e.g., MySQL, Xen, Cassatt)

• If a project won’t survive without FOSS company backing, consider options (e.g., Linux Foundation)

Lesson 6

• In FOSS, there is no such thing as autopilot

– Intent is critical

– If you are not planning to succeed, you are planning to fail

– Great software is not enough; you can have the best technical solution, but if a “big dog” starts throwing its weight around, you need to be able to respond

– If you're not looking at your whole ecosystem, you are inviting failure

Lesson 7

• If it ain't growing, it's dying

– If your project team is seeing no new blood over time, be concerned

– Open Source organisms must move and grow

– New folks are needed from time to time to add new ideas and keep focus on what users need

Lesson 8

• Know where your project could fit in the world and make a plan to get there

– Competition means you probably won't be best fit for every situation

– It may not be possible to have every feature your competitors have (especially if they have much corporate backing)

– Figure out who your users are, what they need, and what you need for them to use your code

Lesson 9

• Competition Increases Innovation

– Lack of competition can cause stagnation (consider Unix CDE)

– Competing technologies keep the ball moving forward continuously

– Xen's competition with KVM and VMware insured that new virtualization capabilities would keep flowing

– A competing project has to stay on top of its game or it won't make it

Lesson 10

• Major new features can keep your mindshare alive in the community

– Large advances (e.g., ARM and Mirage) generate attention from the FOSS ecosystem and the userbase

– If you aren’t making headway, your root and support structures may stop working to give you what you need

– Periodic advances keeps the project growing

Lesson 11

• Sometimes, perception really is reality

– You can have the best code in the world, but if no one cares, it’s useless

– If the rumor arises that you are dying, outmoded, or outdone by some other project, you must fight that perception

– The unchallenged lie will become fact for many people

Lesson 12

• In contrast, KVM managed perceptions well

– It could have looked like a purely Red Hat/IBM business play when Red Hat purchased Qumranet

– The relationship between Red Hat and the KVM project was well-defined & appropriate; no disconnect occurred

– FOSS community embraced KVM project

– Clearly, Red Hat and IBM are still focusing major business initiatives on KVM, but the community accepts that because it was done correctly

Lesson 12 (continued)

• Go to local FOSS events – Submit talks

• Get used to rejection and learn from it

• Talk to the track chairman

– “I don’t like speaking” – get over it; calculus was way harder than this

– Get an ORG booth for cheap

• Stock it with flyers, CDs, business cards, stickers

• Shoot your mouth off – Blogs

– A usable website

– Podcasts (TLLTS)

– Social Media

• Twitter, Facebook, Google+, LinkedIn

• More mouthing off – YouTube demos and tutorials

– Write for Linux.com LXer, LWN.net

• Get a “kick-*aas” mascot! – But our buddy the Xen panda is taken!

• Shout out and live, or shut up and die! – Passion is your ally

– Let it leak over everyone

– Don’t imitate the suits; do what fits you

Crash Course in Perception Management

• There's a new reality for FOSS projects: the corporate connection

– Projects used to be primarily volunteers working nights and weekends

– Today, corporations play a big role in development

• You need to have a good grip on what your corporate sponsors want from you, and what you need from them; disconnection can be fatal

Lesson 13

• Manage the relationship between business and project

– Prevent the loss of the project’s identity

– If the project appears “owned” by a business, the FOSS community might become suspicious and back away

– In this case, perception is as dangerous as reality

– If you forget what the project is, every else will too

– Project ecosystem will wither away; only the business remains

Lesson 13 (continued)

• Establish a symbiotic relationship

– Business provides user feedback, resources

– Project provides overall focus, goal, direction, labor

– Both sides need to color in the lines

– Otherwise, you get “fake Open Source:” the code is open, but there is no community, no support, no ecosystem

Lesson 13 (continued)

• Make sure your project addresses its entire ecosystem:

– Is the code good?

– Are you reaching out to your users?

– Is your development community active, engaged, and growing?

– Are reaching out to the FOSS community?

– Have you insured you have proper support (libraries, distros, kernels, etc.)?

Final Thoughts

• If you are in a relationship with a corporation, is that relationship healthy?

– Do you have freedom to do what you need to do?

– Are you getting user feedback to seed new growth in the project?

– Is your project's identity getting lost in the corporate identity? (if so, consider a foundation route)

• Whatever else, don't give up!

Final Thoughts (continued)

Russell.Pavlicek@XenProject.org

XenProject.org

Questions?