the open hardware kit - lu

Requirements Common platform for machine instrumentation White Rabbit Open Hardware

The Open Hardware Kit

Javier Serrano

CERN, Geneva, Switzerland

12 May 2011

1 / 57


1 Requirements2 Common platform for machine instrumentation

IntroductionThe FMC standardCERN’s implementation

3 White RabbitIntroductionTechnology overview

Precision Time Protocol (IEEE1588)Synchronous EthernetPhase trackingWhite Rabbit Switch

ApplicationsWR in CERN’s BE-CO-HT Hardware Kit

PlanningCurrent statusDevelopment plans for 2011

4 Open Hardware

2 / 57


Outline1 Requirements2 Common platform for machine instrumentation






4 Open Hardware3 / 57


Desirable HW features of a distributed control system1/2

The good things of custom HWFunction is exactly what you need.Can change easily if you find a bug. Or have it changed!Peer review. Potential for really good designs.Not tied to a single company (you never know).

The good things of commercial HWDesigned, built and tested by someone else (resourcegain).Immediately available.Hardware proven by many users in many differentapplications.

4 / 57


Desirable HW features of a distributed control system2/2

ModularRe-use components easily.Have different people in an organization do what they dobest.

Interconnect!Allows to build distributed systems easily.Based on communication standards.Good sync capabilities. Transparent common notion oftime.

5 / 57


Background

CERN’s BE Controls group supports a kit of standardhardware modules.Support includes stocks management, help in debuggingand low level software:

Linux Device Drivers.C/C++ libraries with usage examples.Test programs for drivers and libraries.

With the injectors renovation project, supported platformswill include PCIe in addition to VME64x.A carrier/mezzanine strategy has been adopted.

7 / 57


Advantages of the carrier/mezzanine approach

Re-useOne mezzanine can be used in VME64x, PCIe or other carriers.

ReactivityNo need to place and route a complex FPGA PCB for everynew user need.

Rational split of workControls can design the carrier, Instrumentation an ADCmezzanine, RF a DDS one, etc.

8 / 57


Outline

Courtesy of VITA: http://www.vita.com/fmc.html

9 / 57


Connectors

Ball Grid Array (BGA) characterized for high bandwidthapplications.Low Pin Count (LPC) and High Pin Count (HPC) variantswith 160 and 400 contacts respectively.

10 / 57


Physical Dimensions

Small dimensions for thermal reasons.Keep all digital circuitry in the carrier.

11 / 57


Agnosticism

Pin function, sense – input or output – and electricalstandard are defined at FPGA configuration time.Carrier reads FMC identity through an I2C serial bus andconfigures the FPGA accordingly.

12 / 57


Carrier design

VMEInterface

PowerSupplies

EthernetInterface

ClockGeneration

DDRMemory

SystemFPGA

EEPROMMemory

ApplicationFPGA

ZBTMemory

FMCConnectors

To ALL the modules

13 / 57


Ongoing developments1/2

WR-enabled carriersVME with two single-width (one double-width) slots.PCIe with one single-width slot.Under discussion: PXIe (and maybe PXI).

14 / 57


Ongoing developments2/2

MezzaninesFour-channel 100 MS/s 14-bit ADC with oscilloscope-typeanalog front end.Fine delay generator.Time-to-Digital Converter (TDC).Under discussion:

Eight-channel 100 kS/s 16-bit sampler.Simple parallel digital I/O.Four-channel 250 MS/s DAC.One-channel 24-bit 2 MS/s sampler.

15 / 57


A critique of FMCWhat we can say after 1 year of experience

ProsIt’s a standard! Somebody thought hard about mechanics,dissipation, connectors...The only agnostic standard for FPGA mezzanines.Auto-discovery of card type very useful.

ConsReal estate a potential problem in some applications.Little consideration given to Carrier-to-mezzanine clocks.

Overall a good choice in our opinion.

16 / 57


Use cases

White RabbitSwitch

SFP

Carrier FMCADC

White RabbitNetwork

Distributed oscilloscope

SFP

CarrierFMCDAC

PLANT

Distributed feedback system

SFP

Carrier FMCADC

SFP

Carrier FMCADC

White RabbitNetwork

White RabbitNetwork

17 / 57


Gateware and softwareSome initial ideas

GatewareInternal bus is Wishbone-based (open standard with IPcores available).Try to automate repetitive code through scripts.Auto-discovery of Wishbone cores by Linux kernel wouldbe nice.

Software (very preliminary ideas!)Define Wishbone as a bus in Linux.Write Linux modules and interconnect them by a driverrepresenting the whole board.Integration into official kernel desirable.

18 / 57


Part 1 Summary

The first agnostic standard to interface mezzanines andFPGAs.CERN’s BE-CO group will adopt it to improve support ofhardware and reduce maintenance costs.Combined with Open Hardware paradigm andcollaborations, it can reduce duplication and improvedesign quality.

OutlookLaunch calls for tender for some designs before June 2011.Start using commercial open hardware before the end of2011.

19 / 57


What’s in a name?

Oh dear! Oh dear! I shall be too late!

21 / 57


Development model

Developed in the frame of CERN’s renovation and GSI’sFAIR projects.Open source design done in collaboration with industry.Commercial production and support.

22 / 57


What is White Rabbit?

23 / 57


What is White Rabbit?

An extension to Ethernet which provides:Synchronous mode (Sync-E) - common clock for physicallayer in entire network, allowing for precise time andfrequency transfer.Deterministic routing latency - a guarantee that packettransmission delay between two stations will never exceeda certain boundary.

24 / 57


Design goals

ScalabilityUp to 2000 nodes.

Range10 km fiber links.

Accuracy and precision1 ns time synchronization accuracy, 20 ps jitter.

25 / 57


Technologies used in White Rabbit

Sub-nanosecond synchonization in WR is achieved by usingthe following three technologies together:

Precision Time Protocol (IEEE1588).Synchronous Ethernet.DMTD phase tracking.

26 / 57


Network topology

27 / 57


PTP Protocol (IEEE1588)

PTPSynchronizes local clock with the master clock by measuringand compensating the delay introduced by the link.

Packet timestampingLink delay is measured by exchanging packets with precisehardware transmit/receipt timestamps.

28 / 57


PTP Protocol (IEEE1588)

Master

time scale

Slave

time scale

t1

t4

t2

t3

Having values of t1...t4, slave can:calculate one-way link delay:δms = (t4−t1)−(t3−t2)

2

syntonize its clock rate with themaster by tracking the value oft2 − t1compute clock offset:offset = t2 − (t1 + δms)

29 / 57


Disadvantages of traditional PTP

All nodes have free-running oscillators.Frequency drift has to be continously compensated,causing lots of network traffic.That doesn’t go well with determinism...

30 / 57


Synchronous Ethernet

Common clock for the entire networkAll network nodes use the same physical layer clock,generated by the System Timing Master.Clock is encoded in the Ethernet carrier and recovered bythe receiver chip (PHY).PTP is used only for compensating clock offset.Having the same clock frequency everywhere enablesphase detector technology as the means of measuringtime.

31 / 57


Synchronous Ethernet

32 / 57


Phase tracking

Plain PTPPTP alone is not enough if we want very good accuracy,because of the granularity of the timestamps.

SolutionMeasure the phase shift between transmit and receive clock onthe master side, taking the advantage of Synchronous Ethernet.

33 / 57


Phase tracking

Master Slave

Transmitter

TransmitterReceiver

with CDR PLL

Receiverwith CDR PLL

DMTDphase

detector

DMTD-basedphase shifting

PLL

reference clock uncompensated clock

recovered clock

recovered in-phase clock

fm

Link(up to 10 km single mode fiber)

clock loopback

Monitor phase of bounced-back clock continuously.Phase-locked loop in the slave follows the phase changesmeasured by the master.

34 / 57


White Rabbit Switch

Central element of WR network.Fully custom design, designed from scratch.10 1000Base-LX ports, capable of driving 10 km of SMfiber.200 ps synchronization accuracy.

35 / 57


White Rabbit Switch

Designed in microTCA MCH (Management Carrier Hub)format.Multi-PCB design: base board with main big FPGA andCPU and Clocking Mezzanine, which handles the timing.Can work in standalone mode (without a microTCA crate)via mini-backplane.

36 / 57


Switch block diagram - main part

resets

main/failsafe flash select

Sensors

DMTDCLK_MAIN

CMI interface

REFCLK_MAIN

(from clocking mezzanine)

(to clocking mezzanine)

REFCLK_DPx (for downlink PHYs)

FPGA_REFCLK

ZBT_CLK

IPMB-A,B (2x)

To FPGA

IPMB-L (8x)

uTCA-specific

From FPGA

From FPGA

From CPU

Main FPGAAltera EP3C120

AT91SAM9263CPU

512 kBZBT SSRAM

2x uplink PHYs

8x

do

wn

link

PH

Ys

8x SMI links

Front Panel32-bit bus

with IRQs/DMAEBI1

GPIO

GPIO/SSP0

GPIOGPIOSPI1UART

JTAG

Config

EBI064 MBSDRAM

10/100MbpsPHY + magnetics

8 MB r/oDataFlash

(failsafe partition)

256 MB r/wNAND Flash

(main partition)

32

8

SPI0

MII

Atmega128watchdog &IPMB hub

Externalmanagementport

Ext PPS inSMA

Ext PPS outSMA

Serialconsole

LEDs

Leveltranslator

Clock MezzanineInteface (CMI)

CPU interfaceinternal network

controller

WR

sw

itch

Switch Managementinterface (SMI)

1:10 LVTTLfanout

to b

ackp

lan

e I

/Fse

ctio

n

(from clocking mezzanine)

32

System FPGA handles all packet processing.CPU implements PTP stack and management functions(SNMP, Spanning Tree).

37 / 57


Possible applications of White Rabbit

Large-scale

data acquisition

systems

Clock & trigger

distribution

Robust

event delivery

Precise

time tagging

38 / 57


WR in CERN’s BE-CO-HT Hardware Kit

FPGA

FM

C c

on

ne

cto

r

Memory

PowerWR port

ADC

DAC

Time-to-Digtal

Fine delay

Carrier boardFMCs

CERN’s BE-CO-HT FMC-based Hardware Kit:FMCs (FPGA Mezzanine Cards) with ADCs, DACs, TDCs,fine delays, digital I/O.Carrier boards in PCI-Express and VME64x formats (aµTCA carrier also exists in the OHR).All carriers are equipped with a White Rabbit port.

39 / 57


Ethernet Clock distribution a.k.a. Distributed DDS

WR Network

Phasedetector

digitalfilter

DDSsynthesizer DDS

synthesizer

input clock

in-phasereconstructed clock

DDS control words

Encoder Receiver

DDS control

DDS control

time & frequency reference

time & frequency reference

Distributed Direct Digital SynthesisReplaces dozens of cables with a single fiber.Works over big distances without degrading signal quality.Can provide various clocks (RF of many rings and linacs)with a single, standard link.

40 / 57


Distributed oscilloscope

WR NetworkFront-end cards Operator's PC

WR Switch WR Switch

WR Switch

Master

ADC

TDC

DDS

Analog signals

Triggers

Sampling clocks

Common clock in the entire network: no skew betweenADCs.Ability to sample with different clocks via Distributed DDS.External triggers can be time tagged with a TDC and usedto reconstruct the original time base in the operator’s PC.

41 / 57


WR Switch development status

Switch hardwareWorking and debugged V2 hardware prototype.Tested on 10-km fiber links.Interoperates with standard Ethernet gear.

Switch softwareDone the Hardware Abstraction Layer and PTP daemon.Sub-nanosecond accuracy over PTP has been achieved.Verified interoperability with other PTP devices on ISPCS2010 Plug Fest.

42 / 57


Already achieved...

According to ISPCS Plug Fest results ...... White Rabbit is the most accurate PTP implementation

in the world!

43 / 57


Foreseen milestones

WR SwitchFull basic functionality of HDL and software alreadyachieved, code cleanup foreseen during Q2 2011.V3 prototype: Q3 2011.Commercial product: Q1 2012.

WR EcosystemFMC Carriers: VME and PCIe prototypes done, moving torev 2 Q2 2011.WR timing node in VME and PCIe: commercially availableQ2 2012.Mezzanines: Full set of cards available Q4 2011.

44 / 57


Part 2 Summary

A data link fulfilling all our needs in synchronization anddeterminism.A successful collaboration including institutes andcompanies.Full system commercially available mid-2012.

For more information, visithttp://www.ohwr.org/projects/white-rabbit/wiki

45 / 57

http://www.ohwr.org/projects/white-rabbit/wiki


Open Hardware: our definition

Publish everything needed to review

Specifications, discussions, schematics and layouts in somehuman-readable format, HDL, etc. Publish universally, noNDAs.

Publish everything needed to modify

Schematics and PCB layout files for your favorite EDA tool.Unfortunately the best ones are neither free nor free. . .

Publish everything needed to produceManufacturing files, bill of materials, etc.

47 / 57


Advantages

Peer reviewGet your design reviewed by experts all around the world,including companies!

Design re-use

How many people are designing a 100 MS/s ADCindependently, making the same – or different – mistakes?

Healthier relationship with companiesNo vendor-locked situations. Companies selected solely on thebasis of technical excellence, good support and price.

48 / 57


Getting organized

“It’s all about interfaces”, Bob DalesioYour piece of HW can speak to others if you can agree on a setof interfaces. Examples (currently in OHR) include Ethernet,VME, PCIe, FMC. . .

Design compromisesThe price to pay for sharing (and saving time and money) is tochoose sub-optimal technical solutions from time to time. Wedid not choose to write an OS more suited to our needs thanLinux, did we?

49 / 57


Role of companies

Design partnersPay a company specialized in a given topic to design a specificcard with/for you.

Commercial partnersBuy the cards you designed from a company that will take thecharge of manufacturing, testing, managing stocks andproviding support.

50 / 57


Some business model examples for commercialpartners

IBM-styleBecome part of a larger OH team, fully respecting OH practice.Sell full systems based on OH kit.

Red Hat-styleSell manufacture, test and support of individual boards alongwith a guarantee. Participate in design if needed.

Oracle-styleSupport OH kit and build a closed solution on top with addedvalue.

51 / 57


LicensingA quick landscape tour

Hardware is not like softwareCopyright protects the expression of an idea, not the ideaitself.For a schematic (and even HDL), GPL is easily bypassed.

OptionsOHL (persistent). If you take my design and use it, youpromise not to sue me for patent infringement.BOHL (persistent). Design files are not released.MIT/BSD (non-persistent). Do what you like, don’t blameme in case of problems.CERN OHL. See next slide.

52 / 57

http://www.tapr.org/ohl.html

http://www.balloonboard.org/docs/Balloon_License_0v2.pdf

http://www.opensource.org/licenses/mit-license.php

http://www.linfo.org/bsdlicense.html

http://ohwr.org/cernohl


LicensingCERN’s choices

LGPL for HDLIt’s very easy to turn a “used in” into a “connected to”situation in HDL, so GPL would not help.We do want to be informed and profit if our cores areimproved.

CERN OHL for PCBsPersistent.Requires that manufactures inform designers of dates andquantities of production→ helps measure impact.

53 / 57


What about free riders?

Free riders are finePeople and companies who take open designs and do notcontribute anything back do not pose a problem to us.

But what about mean free riders?If somebody takes OH and uses it to build a closed solution fora profit, that is fine as well, but we would not be clients.

54 / 57


Mad patent disease and patent trollsSee http://www.eetimes.com/showArticle.jhtml;?articleID=216600017

“In this climate, many fear being charged with willfully infringingpatents or omitting prior art in patent applications, a chargeknown as inequitable conduct. So Intel and other companieshave put strict procedures in place to control which patents itsengineers can read.”

Opening up your designs does make you more vulnerable tothis disease.

55 / 57

http://www.eetimes.com/showArticle.jhtml;?articleID=216600017


Open Hardware Repository: http://www.ohwr.org

A very useful toolA web-based collaborative tool for electronics designers.

Made itself of open softwareRedmine for wiki and task/issue management.Sympa mailing list manager.SVN/GIT for version management (integrated in Redmine).

Other possible usesTraceability for Technology Transfer departments.Prove prior art with UTC time stamps in SVN, GIT, wiki...

56 / 57

http://www.ohwr.org


Conclusions

Open Hardware looks like a good idea so far

We can get the best of the custom and COTS worlds.We are learning a lot, even electronics! ;)Definitely more fun than closed HW.

First commercial open hardware available before the end ofthe year. Stay tuned!

57 / 57

the open hardware kit - lu

Documents