linaro and android kernel

Linaro and Android Kernel

Thanks to Matt Hsu and John Lee at 0xlabs,AzureWave and the other organizers

for being so kind as to invite me

Linaro and Android Kernel

1. What is Linaro2. Android kernel patches and Linaro

3. Angry Birds

Andy Green, TI Landing Team Lead, LinaroWith thanks to John Stultz at Linaro

Linaro and Android

1. What is Linaro2. Android kernel patches and Linaro3. ARM Wrestling with Angry Birds

Andy Green, TI Landing Team Lead, LinaroWith thanks to John Stultz at Linaro

Andy Green

Hardware, software and SoC design for 25 years; FPGA early adopter in 1989; wrote HTML5 libwebsockets

Worked as kernel maintainer at Openmoko with many people now at 0xlabs; architected txtr e-book reader

Texas Instruments Landing Team Leader at Linaro

Moving to Taiwan in August 2011 with my family

My wife is progressing learning 中國語言

I am stuck at 好, know a little にほんご though

Part 1: What is Linaro?

Structure

Paid for by its members

“Not for profit” - money is spent on engineers and engineering

“Core” members get a director on the board and representatives on the “Technical Steering Committee”

Three engineering sections:

Working groups

Platform engineering

Landing Teams

Working Groups

Specialist teams focused on doing difficult tasks inside their domain

Kernel – eg, pushing Device Tree, Thumb2 kernel

Toolchain – eg, advanced gcc work

Power Management – eg, idle state latency improvement, dynamic CPU multicore hotplug

Graphics / Multimedia – eg, OpenGLES 2 compiz, prepare upcoming ARM GPUs kernel frameworks

http://status.linaro.org

Landing Teams

Work closely with vendors

Three person team from Linaro works with vendor assignees

Cooperate to get stuff upstream and in Nicolas Pitre's linux-linaro tree

I lead the TI Landing Team, work with TI folks in Nice, France, and Bangalore, India

OMAP4 Panda and OMAP5 stuff shortly

Manpower

Started with engineers from Canonical last year

Contract Engineers

Employees

Member Assignees

Over 100 engineers now in total

Some big names like Nicolas Pitre

Linaro Goals

Agree common problems with the vendors and try to solve them directly, or be the basis of wider solutions

Some problems can't be solved by one vendor

Some problems might otherwise not be solved by volunteers

Some problems require a funded, stable, trusted organization to lead the way to the solution

Linaro Goals

Agree common problems with the vendors and try to solve them directly, or be the basis of wider solutions

Some problems can't be solved by one vendor

Some problems might otherwise not be solved by volunteers

Some problems require a funded, stable, trusted organization to lead the way to the solution

*

Part 2: Android Kernel patches & Linaro

Android becoming ubiquitous

All the SoC vendors are quite clear Android is very important for them

Many of the vendors' customers are quite clear Android is very important to them

Excepting a certain Finnish phone company recently for some reason

Linaro already engages with Android and has Android experts on board

Android is really “a Linux distro”

The whole Android “distro”

Linux Kernel

Android Kernel Patchset

Special minimal libc – Bionic

Other libraries

The Dalvik VM

Application Frameworks

Apps themselves

Just going to cover Android kernel

Linux Kernel

Android Kernel Patchset

Special minimal libc – Bionic

Other libraries

The Dalvik VM

Application Frameworks

Apps themselves

Android Kernel Patches Overview

All features that mainline does not offer, or improvements

Android Kernel Patches scope 1 / 6

Ashmem

Android Shared Memory, allows naming allocated memory and having it visible between multiple processes

/dev/ashmem device node tracks usage count on the named memory blocks

Under memory pressure, kernel can free blocks with no current users

Binder

RPC scheme like CORBA


Pmem

Allows userspace to map a given range of contiguous physical address

Used for userland talking to DSPs and similar that cannot tolerate scatter-gather

Logger

Android's kernel logging, supports four logging buffers “main”, “events”, “radio”, and “system”

Userland can read from the files down /dev/log/...


Early Suspend

Allows userland to coordinate a staged suspend action of four levels

From most awake to suspended, it first allows screen to be blanked, then stop access to the framebuffer, then turn off the framebuffer / crtc, and lastly stop all non-wake input devices

Wakelocks

Allow userland to veto either low power state (eg, stop CPU Clock) and / or entry to suspend; a few kernel places need to use this too – ie, covering pending timers in input device driver


Alarm Timer

Deal with tracking elapsed time while in suspend

Schedule wakeups from RTC alarm while CPU completely OFF; kernel already allows some of this

Low Memory Killer

More proactive and cooperative version of existing OOM Killer

Paranoid Network

Restricts certain network operations to processes with magic group IDs


RAM Console

Debugging helper similar to EARLY_PRINTK but buffering very early messages to RAM, good for silently dying boot

Apanic

Panic handler tries to write the panic dump to flash where it can be recovered the next boot

ADB Gadget Driver

USB Gadget driver enables debugger operations from host PC – also able to listen on a network interface


Timed GPIO

Adds a period capability to generic GPIO, allowing primitives like “set this high for 50ms”

“Other”

Small hacks and fixes for ARM, MMC, Bluetooth etc

Android Kernel Patches

~250 patches for 2.6.38, 3.3MBytes of patches

Relatively thin layer of improvements over mainline, adding features that are not otherwise supported

“Hey this OS is working great, let's send those patches upstream!”

Submitted Jan 2009

Lkml vs Android Patches threads

0

500

1000

AshmemAlarmLowMem

KillerBinder

emails

Use DBUS!

Use OOMKiller!

Lkml vs Android Patches threads

0

1000

2000

AshmemAlarmLowMem

KillerBinder

emails

wakelock

Lkml vs Google 2009 experience

Proposed patchsets: 13 Accepted: 0

Not Invented Here

Political, cultural and practical problems dealing with a large after-the-fact patchbomb

Google just wanted their patches accepted since they were shipping insane volumes already; stopped caring and got on with their job out of tree

As a compromise a copy of the Google patches (not proposed by Google itself) allowed in staging

Some folks tried to work with kernel process but the patches were removed from staging in 2.6.33

Lkml vs Google 2010 experience

Limited re-engagement with lkml

After two years of rejection and stalled progress there are real commits from @android.com in mainline

Over 100, > 50 from 2011

But these are not the 13 critical patchsets

Out-of-tree model works well enough

Wakelock impact on drivers makes practical difficulty working on drivers at both mainline and Android though

Everyone would be happier if they were all in mainline

Now Invented Here

runtime_pm apis added to kernel to do similar job to wakelock by Rafael Wysocki; design at least influenced by wakelock discussion

GregKH, who removed android stuff from staging, writes on his blog 23rd March that android drivers using this native implementation are now welcomehttp://www.kroah.com/log/linux/android-wakelock-solution.html

Competes with existing Google implementation, political as well as technical decision for Google; how to transition?

Unclear if runtime_pm is full wakelock replacement

Summary

Google felt their contribution should have been taken in mainline based on awesome Android volume success and high quality results there.

lkml felt targeted by a fait accompli that did not consider wider kernel architecture issues outside of phone case

If Google had included lkml in the loop to start with they probably get easier ride, but lkml does not follow their schedule

Linaro and the patchset

Controversy about some elements of the patches overshadowed good fixes and improvements

“Trivial Tree” established; fixes split out in topic branches and offered upstream by John Stultz from Linaro

Difficulties defending the patches as third-party; uplevelling all the topic branches while out-of-tree is significant effort

John also working on mainline Posix Alarm Timers that may cover Android Alarm Timer functionality

Linaro Android Platform

Upstream-focused ARM Android Community getting started

Linaro toolchain packages for Android

Cloud-based build services

ARM board farm for automated testing, validation and benchmarking

Integration point for member SoC platform code

Based on linux-linaro tree from Nicolas Pitre

Linaro Android Platform

Installable images and public code for low-cost Linaro member SoC boards

Combines latest public AOSP platform code with latest Linaro kernel and toolchain, plus member SoC platform code

Monthly releases

#linaro-android on Freenode IRC

*

Part 3: Angry Birds

Linus, Sep 1998

To Theodore Ts'o's complaints about lost patches...

“...Quite frankly, this particular discussion (and others before it) has justmade me irritable, and is ADDING pressure. Instead, I'd suggest that ifyou have a complaint about how I handle patches, you think about what Iend up having to deal with for five minutes.

Go away, people. Or at least don't Cc me any more. I'm not interested, I'mtaking a vacation, and I don't want to hear about it any more. In short,get the hell out of my mailbox.''

http://lkml.indiana.edu/hypermail/linux/kernel/9809.3/0850.html

Analysis

Symptom of overload due to success

This was before bitkeeper and later git, the amount of manual work was too high - “think about what I end up having to deal with”

With awesome tools like git, Linus was able to do much more with the same effort and “scaled well”

Able to keep up with the ecosystem and Linux has grown for another 12 years since then

Linus, June 2010

I got a bit frustrated with ten different ARM pulls per day at one point.

There's something wrong with ARM development. The amount of pure noise in the patches is incredibly annoying. Right now, ARM is already (despite me not reacting to some of the flood) 55% of all arch/ changes since 2.6.34, and it's all pointless churn in

arch/arm/configs/

arch/arm/mach-xyz

arch/arm/plat-blah

and at a certain point in the merge window I simply could not find it in me to care about it any more. http://lwn.net/Articles/392135/

Linus sees ARM arch overloading him

ARM architecture is growing strongly

Bit chaotic because outside the core ARM CPUs, it is very diverse indeed

Random peripheral unit macrocells get implemented alongside custom DSP, custom interconnect, custom IPC hardware

Same peripheral unit may be wired on different buses on different SoC, need “glue layer” like MUSB

Platform layer

RMK brings order to the chaos with “platform layer”

A platform is, eg, OMAP2+ for later TI chips, imx for Freescale, etc, combining commonality as much as possible and offering “just works” core code for chips

There are more specific mach- files for boards that use some or all the platform features plus extra support for other chips on the board

“pointless churn... arch/arm/plat... arch/arm/mach...” per-board config in “arch/arm/configs”

How the ARM SoC world works

ARM SoC vendors license CPU Core from ARM

Cheaper, quicker, more certain, better tool / OS ecosystem than proprietary... and patents

So how to differentiate?

Pile on specialized integrated peripheral units

Talk up special process sauce, eg, smartreflex biasing

Sell companion chips with special sauce (eg, OMAP4 vs twl6030 PMIC and twl6040 audio)Twl6030 and CPU talk on dedicated serial link for voltage scaling

Interversion changed files peak 1700

V2.6.12V2.6.14

V2.6.16V2.6.18

V2.6.20V2.6.22

V2.6.24V2.6.26

V2.6.28V2.6.30

V2.6.32V2.6.34

V2.6.36

0

200

400

600

800

1000

1200

1400

1600

1800

Files changed in arch/arm

files

Source: git diff using linus tree

Interversion diffstat peak 250kloc

V2.6.12V2.6.14

V2.6.16V2.6.18

V2.6.20V2.6.22

V2.6.24V2.6.26

V2.6.28V2.6.30

V2.6.32V2.6.34

V2.6.36

0

50000

100000

150000

200000

250000

300000

Lines -

Lines +

Source: git diff using linus tree

Why Linus is a bit stressed

500 pull / version endurance limit

Data Source: http://lwn.net/Articles/393694/

“2.6.26”“2.6.27”

“2.6.28”“2.6.29”

“2.6.30”“2.6.31”

“2.6.32”“2.6.33”

“2.6.34”“2.6.35”

0

500

1000

1500

2000

2500

Linus pulls

2.6.26 thru 2.6.35

Patches total

Patches direct

Pulls Total

Pulls in MW

version

Pu

lls /

Pa

tch

es

Interversion commit count stable at ~11K

Source: eg, git log v2.6.26..v2.6.27 --oneline | wc -l

v2.6.26 v2.6.27 v2.6.28 v2.6.29 v2.6.30 v2.6.31 v2.6.32 v2.6.33 v2.6.34 v2.6.35 v2.6.36 v2.6.370

2000

4000

6000

8000

10000

12000

14000

Commits

Gaah. Guys, this whole ARM thing is a f*cking pain in the ass.

You need to stop stepping on each others toes. There is no way that your changes to those crazy clock-data files should constantly result in those annoying conflicts, just because different people in different ARM trees do some masturbatory renaming of some random device. Seriously.That usb_musb_init() thing in arch/arm/mach-omap2/usb-musb.c also seems to be totally insane. I wonder what kind of insanity I'm missing just because I don't happen to see the merge conflicts, just because people were lucky enough to happen to not touch the same file within a few lines.

Somebody needs to get a grip in the ARM community. I do want to do these merges, just to see how screwed up things are, but guys, this is just ridiculous. The pure amount of crazy churn is annoying in itself, but when I then get these "independent" pull requests from four different people, and they touch the same files, that indicates that something is wrong.

Linus, March 2011

Linus, March 2011 (continued)

And stop the crazy renaming already! Just leave it off. Don't rename boards and drivers "just because", at least not when there clearly are clashes. There's no point. I'm not even talking about the file renames (which happened and can also make it "fun" to try to resolve the conflicts when somebody else then makes _other_ changes), but about the stupid "change human-readable names in board files just to annoy whoever needs to merge the crap".

Somebody in the ARM community really needs to step up and tell people to stop dicking around.

(I'm replying to the omap pull request, because that's the one I did last, but I don't know who to "blame". I don't care. It really doesn't matter. I realize thar ARM vendors do crazy shit and haven't figured out this whole "platform" thing yet, but you guys need to push back on the people sending you crap).

https://lkml.org/lkml/2011/3/17/492

Analysis

Overload handling merge conflicts between ARM arch trees

Renames and missed common code opportunities are “crap”

Asking for more management in ARM world

Subsequently made it clear he doesn't think Russell King (ARM Linux maintainer) is the problem

Big crisis for ARM world, boss guy is raging but individual maintainers unable to react alone

LOT of people dependent on flow to mainline...

Difficulty with MUSB – 1 / 5

Linus complained about MUSB, what is it, what's the problem?

MUSB is Mentor's USB OTG hardware IP

MUSB is a good case study of general issue

Mentor sell hardware design tools including macrocells

Using someone else's tested CPU core has advantages: same goes for someone else's tested USB OTG implementation

MUSB peripheral unit generic view – 2/5

Devil in the details – 3 / 5

Led to Generic driver + SoC glue – 4 / 5

MUSB Macrocell driver down drivers/usb/musb/

Also set of “glue layer” files for SoCs to take care of the messy and different implementation details, eg, blackfin.c, omap4230.c etc

For example, if power interrupts are coming from a different chip, must be set up and enabled by that driver, not MUSB driver, despite MUSB controls the interrupt

Seems to be “not liked” but hardware realities can mandate this kind of setup

Limits of discoverability – 5 / 5

AMBA bus used to hook up peripheral units on ARM SoC is more lightweight than PCI or USB

Probing memory with no unit leads to death by bus exception, so we have to know what we have already

Connectivity width is NOT a property the peripheral unit can declare, it's defined outside the unit itself

OMAP has hwmod “registry”, DT wants to claim that turf

Maybe Linus will end up improving or at least impacting future SoC composition visibility

Unable to react: business as usual

arch/arm arch/m68k arch/mips arch/x86 ?0

10

20

30

40

50

60

70

80

linux-next churn post-Linus rant

Percentage

arch

pe

rce

nta

ge

http://lists.infradead.org/pipermail/linux-arm-kernel/2011-April/048133.html

From dirstatPosted byRMK Apr 14

http://lists.infradead.org/pipermail/linux-arm-kernel/

Xilinx blocked from mainline

Russell King - ARM Linux linux at arm.linux.org.uk Mon Apr 18 13:18:57 EDT 2011

I'm very sorry for people with new platforms outstanding like John Linnwho are on the sharp end of this (who have platform code ready to go)but I see no solution at the moment. It really is the case that eitherI can say no to it, or I can put it in my tree and Linus will say no toit. So putting it in my tree *doesn't* help.

Will we ever be able to put John's code in the kernel? Honestly, I haveno idea.


You are going on a dietLinus Torvalds torvalds at linux-foundation.org Mon Apr 18 12:23:51 EDT 2011

Hint for anybody on the arm list: look at the dirstat that rmk posted,and if your "arch/arm/{mach,plat}-xyzzy" shows up a lot, it's quitepossible that I won't be pulling your tree unless the reason it showsup a lot is because it has a lot of code removed.

People need to realize that the endless amounts of new pointlessplatform code is a problem, and since my only recourse is to say "ifyou don't seem to try to make an effort to fix it, I won't pull fromyou", that is what I'll eventually be doing.

Exactly when I reach that point, I don't know.


Linaro working to form a solution

Linaro has “good offices” and good people that can help with a solution for all ARM SoC vendors

Eg, Thomas Gleixner, Arnd Bergmann, Nicolas Pitre

Well placed to help form a solution

Linaro in contact with all the main participants these last weeks

*

Thanks for listening

http://status.linaro.org

linaro and android kernel

Technology