lcls llrf distributed control system

1 Dayle [email protected]

1LCLS LLRF Distributed Control System LCLS LLRF Distributed Control System

EPICS Collaboration Meeting, Vancouver 1 May 2009

LCLS LLRF Distributed Control System

Dayle KotturiControls Department

SLAC National Accelerator Lab





OutlineScope

Global Overview

General stability requirements

Principal motivator

Solutions

Throughput measurement

Conclusions

Additional resources




Scope

The low level RF controls system consists of RF phase and amplitude controls at these locations:

LaserGunL0-A (a.k.a. L0-1)L0-B (a.k.a. L0-2)L0 Transverse cavityL1-SL1-XL2 – using 2 klystrons to control avg phase/ampl of L2L3 Transverse cavityL3 - here is a bit different (lots of klystrons!)




LLRF Global Overview





For LCLS, the general RF stability requirements are: 0.1 deg phase and 0.1% amplitude in L0 and L1 for S band.




Principal motivator

Placing the digitizers next to the low noise RF components eliminates transmission of low noise analog signals outside the chassis.




I and Q Demo-dulator

CPU

FIFOs

ADC

slow

DAC

slow

Thermocouple system

EVR

VME Crate

CPU

laser

L0-AL0-B

L1-SL1-XT Cav

gun

Beam-based longitudinal

fast feedback gigabit

ethernet

Controls gigabit ethernet (interface to MCC)

Eth recvr

Private ethernet8 kBytes at 120 Hz

PAD

ADC

I and Q Modulator

DAC

FIFOs

1 trigger for 4

channels of 1k

samples


Solution 2: Multiple VME crates with in-house modules

476 M

Hz R

F Refe

rence

cloc

k dist

ribute

d to a

ll 30 s

ector

s in t

he Li

nac a

nd be

yond

RF Reference/4 = 119 MHzstabilized to 50 fs jitter

RF Reference*6 = 2856 MHzstabilized to 50 fs jitter

Controller with

ethernet

Controller with

ethernet

Local trigger

Possibly combined into one module

Slow adjustments to allow rotation of the

reference phase

ADC

fast

Other waveformsFast, but not 119

MHz. 59.5 MHz ok

Global longitudinal beam-based

feedback VME crate

L2: in sector 24, there are 3 stations to adjust in order to accurately control phase and amplitude for long, beam-based fast feedback

PAC

Sector 25 T Cav (new 4/2005)

RF Phase and Amplitude correction at 120 Hz for:laser, gun, L0-A, L0-B, L1-S, L1-X, T cav, L2 and S25 Tcav

10' accelerator

IQ Modulator: a phase shifter

and an attenuator

1 kW 1 kW

100 mW

Solid State Sub Booster

Klystron

SLED cavity

60 MW

HPRF240 MW

60 MW

1 kW

All except laser RF

100 mW

119 MHz Laser

Oscillator

Amps

GunNB: For the gun, SLED

cavity is shorted out

119 MHz120 Hz

UV

photodiode

photodiode

1 trigger to travel up to ½ sector

away




CPU

DAC (slow)Updates

@120 Hz;stable 20 µs before

beam arrives and able to distinguish beamcodes

CPU


feedback VME crate

IQ Modulator (driving sub-

booster klystron in sectors 29)

RF Phase and Amplitude correction at 120 Hz for: L3

Beam-based longitudinal fast feedback gigabit

ethernet


476

MHz

RF

Refe

renc

e clo

ck d

istrib

uted

to a

ll 30

secto

rs in

the

Linac

and

bey

ond

500 W



klystron

10' accelerator

1 kW 60 MW

HPRF240 MW

60 MW

SLED cavity

500 W

klystron

10' accelerator

1 kW60 MW

HPRF240 MW

60 MW

SLED cavity

Sub Booster Klystron Sub Booster Klystron

100 mW 100 mW



1 kW1 kW

8 copies8 copies




Eth recvr

EVR

VME Crate at S20 running

longitudinal, beam-based

feedback.

CPU

PAC

Gun

PADPADPAD

SPAC

RF Dist’n

Laser

PAC

L0-B

PADPAD

PAC

L1-S

PADPAD

PACL0-Tcav

PADPAD

PACL0-A

PADPAD

SPAC

PAD

PAD PAC

Beam Phase Monitor

PAD

SPACSPACSPAC

SPACPAD

Fast PACs: Slow PACs (SPACs): PADs: VME crates:

Eth recvr

EVR

CPU

PAC

L24-1

PACL24-2

PACL24-3

PACTcav L24-8

SPAC

S29

SPAC

S30

PADPAD

RF phase and amplitude correction and global feedback at 120 Hz for LCLS LINAC

PAC

L1-X

PADPADPAD

PAD



feedback.

PAC

Key:

Indicates may be needed

86191

128212

4221

S20 S24 Total

The maybe is included in counts below

Indicates located in RF HutOtherwise at Klystron





CPU

FIFOs

DAC

slow

EVR

PAD

ADC

FPGADAC

1 trigger

Coldfire CPU

running RTEMS

and EPICS

Coldfire CPU

running RTEMS

and EPICS

PAC

DAC

slow

Example of generic LLRF control system instance

VMETemperature monitors Temperature monitors

1 trigger

IQ Modulator gives phase

and amplitude control

2 channels of 1k

samples

4 channels of 1k

samples




Phase/Amplitude Detector -> VME

Coldfire

LLRF PAD (Phase and amplitude detector)

ADCChannel

1

ADCChannel

0

ADCChannel

3

ADCChannel

2

LLRF VME(Handles phase and amplitude

conversions, feedback calculations and setting of PAC)

mvme6100

LCLSCA network

MCC workstation running lclshome GUI

Change PAD channels’ size and offsets

Set PDES, ADES and control RF feedback

Use secondary ethernet interface to send 4 channels of PAD data via UDP broadcast via private

switch. Time required: 100 nsec

- Receive I and Q averages- Convert to 4 channels of Phase and amplitude- Compute weighted averages- Trigger BSA - Do feedback calculation- Send I and Q adjustment to PAC

EVR timing trigger

Update GUIs with actual phases amplitudes, feedback

corrections, etc, every 2 seconds

Update GUIs with PAD

waveforms, temperature,

I and Q actuals every

2 seconds

Private gigabit switch




OS, BSP and EPICS versions

PAD: rtems4.9.1m68k uC5282epics-R3.14.10

VME: rtems4.9.1powerpc beatnik (mvme5500/mvme6100)epics-R3.14.8.2

PAC: rtems4.9.1 m68k uC5282epics-R3.14.10




PAD waveform readout




PAD IOC Stats




VME Feedback Calculation




VME IOC Stats




PAC waveform control




Throughput Time: PAD->VME->PAC

1

2

3

4

5

6

7

8

910




Throughput steps: PAD->VME->PAC

Event (#

matches

diagram)

Absolute time after trigger (μsec)

Description

- 0.0 Trigger delivered to phase/amplitude detector digitizer (PAD)

1 7.2 ISR: signals “data ready” to wake up DAQ task

2 102.0 Channel0 readout and data processing begins (40 Is and 40 Qs)




6 1963.2 PAD starts stream of processed values to VME over private net

7 2349.6 PAD streaming completed. VME parses, does feedback, sends.

8 2508.0 Phase/amplitude controller (PAC) receives new setpts from VME

9 2524.0 PAC writes to FPGA which calcs new WF to send next trigger

10 2529.2 Data is ready




Conclusions

At 120 Hz operation, time budget=8.333 ms

LLRF PAD->VME->PAC throughput measured=2.529 ms for 4 channels of 40 points each, with no offsets,

adjust: subtract 2.5 μsec per pair of IRQ raise/lower calls (8 pairs = 20 μsec)

adjust: one socket sends to multiple PACs; add switching time




Acknowledgements

Thanks to Ron Akre and Klystron Department for setting up hardware, scopes and signal generators

Thanks to SLAC NAL Controls Group




Additional Information

Details of the PAD->VME transferDetails of the VME->PAC transferRF stability measurementPADPAD Block diagramLCLS LLRF website: http://www.slac.stanford.edu/grp/lcls/controls/global/subsystems/llrf




Details of the PAD->VME transfer

http://www.slac.stanford.edu/grp/lcls/controls/global/sw/epics/epics%20team%20meetings/presentations/lanIpBasic.pdfRaw ethernet packets with IP and UDP headers. Similar to BSD sockets.Solution is for low end CPU on small LAN.Requirement: ship 1 KB of data in ~200 μsecVME initializes, starts and stops PAD streamingWhen PAD is streaming, device support for waveform on VME parses out the values and uses them in the feedback calculations of new setpoints.




Details of the VME->PAC transfer

On VME, a subroutine record that has calculated new setpoints calls a driver routine that sends the values to the PAC via udp socket

PAC is has thread waiting to receive packet

When packet arrives, it parses out the setpoints and puts them into mem mapped FPGA




LCLS Jitter Specification for 2 Seconds is 0.14% Amplitude and 0.14 degree Phase

Feedback ON 20 Second Plot showsPhase Jitter 0.043 degreesAmplitude Jitter 0.022%


Short Term RF Jitter Specification for L0B are well Exceeded.This is as good as it gets – Don’t tell Physicists or they will expect it. Ron Akre 2007




About the PAD

CPU is MCF5282 (64MHz)The digitizer used is the Linear Technologies LTC2208. It was the first 16 bit digitizer chip on the market capable of running at 119MHz, it is specified to run up to 130MHz.At SLAC NAL, PAD digitizer used for RF, beam position monitors, beam charge monitors and bunch length monitors.Pohang Light Source is also using PAD for new RF system.




Phase/amplitude detector (PAD) Block

1 ohm

5VDCLM340Reg

4 X 16 bit ADC102MHz ClockLTC2208Transformer Coupled Inputs

Chan. 2

Chan. 1 FIFO 64k wordsWCLK

16bit DATA

16bit DATA

WCLK

CHAN 1RF INPUTRP N

CHAN 2RF INPUTRP N

ZX60-4016E

MIXER

LORFIF

MIXER

LORF

IF

RF Board25.5MHz IF

Control Board

Control

FIFO 64k words

FIFO 64k words

FIFO 64k words

WCLK

16bit DATA

WCLK

16bit DATA

RA

WE

TH

ER

NE

TC

OM

12VDCLM340Reg5 ohm

FRONT PANEL LEDs

1 ohm

10 ohm

1 A F B

110VAC

1 1 0 V A CP o w e r M o d uleC o rc o m6 E H L 1 S C GRN

RED

BRNBLU

YELBLK

REDVIO

TE62071-ND 2x9V 1.94A

5VDCLM340Reg

12VDCLM340Reg

2 ohm

5VDCLM340Reg

GRNRED

BRNBLU

YELBLK

REDVIO

TE62054-ND 2x18V 0.417A

CHAN 4RF INPUTRP N

CHAN 3INPUTRP BNC

Chan. 4

Chan. 3

MIXER

LORF

IF

QSPI5VDC0.8A x 2 Analog

5VDC0.5A DigitalLO OUTPUT

2830.5MHzFP N

24Bit Analog InputBoard

QSPI

20 pin ribbonTRIG MonFP BNC

ANALOG IN ANALOG IN

CLOCK Mon102MHzFP N

TRIG In120HzRP BNC

CLOCK IN102MHzRP N

ZN4PD1-50

CS/CLK

16 bitDATA

CONTROL /Arcturus uC5282Microcontroller Modulewith 10/100 Ethernet

ET

HE

RN

ET

LO INPUT2830.5MHzRP N

CPLD

-2dBm 10dBm

10dBm

+18dB

10dBm-8dBm

3x 75mA

FILTER 25.5MHz BP

5VD 5VA2 5VA1 12V2 12V1

Ron Akre





OutlineScope


Principal motivator

Solutions

Throughput measurement

Conclusions

Additional resources




Scope

The low level RF controls system consists of RF phase and amplitude controls at these locations:

LaserGunL0-A (a.k.a. L0-1)L0-B (a.k.a. L0-2)L0 Transverse cavityL1-SL1-XL2 – using 2 klystrons to control avg phase/ampl of L2L3 Transverse cavityL3 - here is a bit different (lots of klystrons!)





For LCLS, the general RF stability requirements are: 0.1 deg phase and 0.1% amplitude in L0 and L1 for S band.




Principal motivator

Placing the digitizers next to the low noise RF components eliminates transmission of low noise analog signals outside the chassis.





CPU

FIFOs

ADC

slow

DAC

slow

Thermocouple system

EVR

VME Crate

CPU

laser

L0-AL0-B

L1-SL1-XT Cav

gun

Beam-based longitudinal

fast feedback gigabit

ethernet


Eth recvr


PAD

ADC

I and Q Modulator

DAC

FIFOs

1 trigger for 4

channels of 1k

samples


Solution 2: Multiple VME crates with in-house modules

476 M

Hz R

F Refe

rence

cloc

k dist

ribute

d to a

ll 30 s

ector

s in t

he Li

nac a

nd be

yond



Controller with

ethernet

Controller with

ethernet

Local trigger

Possibly combined into one module

Slow adjustments to allow rotation of the

reference phase

ADC

fast

Other waveformsFast, but not 119

MHz. 59.5 MHz ok


feedback VME crate

L2: in sector 24, there are 3 stations to adjust in order to accurately control phase and amplitude for long, beam-based fast feedback

PAC

Sector 25 T Cav (new 4/2005)

RF Phase and Amplitude correction at 120 Hz for:laser, gun, L0-A, L0-B, L1-S, L1-X, T cav, L2 and S25 Tcav

10' accelerator

IQ Modulator: a phase shifter

and an attenuator

1 kW 1 kW

100 mW

Solid State Sub Booster

Klystron

SLED cavity

60 MW

HPRF240 MW

60 MW

1 kW

All except laser RF

100 mW

119 MHz Laser

Oscillator

Amps

GunNB: For the gun, SLED

cavity is shorted out

119 MHz120 Hz

UV

photodiode

photodiode

1 trigger to travel up to ½ sector

away




CPU

DAC (slow)Updates

@120 Hz;stable 20 µs before

beam arrives and able to distinguish beamcodes

CPU


feedback VME crate



RF Phase and Amplitude correction at 120 Hz for: L3

Beam-based longitudinal fast feedback gigabit

ethernet


476

MHz

RF

Refe

renc

e clo

ck d

istrib

uted

to a

ll 30

secto

rs in

the

Linac

and

bey

ond

500 W



klystron

10' accelerator

1 kW 60 MW

HPRF240 MW

60 MW

SLED cavity

500 W

klystron

10' accelerator

1 kW60 MW

HPRF240 MW

60 MW

SLED cavity

Sub Booster Klystron Sub Booster Klystron

100 mW 100 mW



1 kW1 kW

8 copies8 copies




Eth recvr

EVR



feedback.

CPU

PAC

Gun

PADPADPAD

SPAC

RF Dist’n

Laser

PAC

L0-B

PADPAD

PAC

L1-S

PADPAD

PACL0-Tcav

PADPAD

PACL0-A

PADPAD

SPAC

PAD

PAD PAC

Beam Phase Monitor

PAD

SPACSPACSPAC

SPACPAD

Fast PACs: Slow PACs (SPACs): PADs: VME crates:

Eth recvr

EVR

CPU

PAC

L24-1

PACL24-2

PACL24-3

PACTcav L24-8

SPAC

S29

SPAC

S30

PADPAD

RF phase and amplitude correction and global feedback at 120 Hz for LCLS LINAC

PAC

L1-X

PADPADPAD

PAD



feedback.

PAC

Key:

Indicates may be needed

86191

128212

4221

S20 S24 Total

The maybe is included in counts below

Indicates located in RF HutOtherwise at Klystron





CPU

FIFOs

DAC

slow

EVR

PAD

ADC

FPGADAC

1 trigger

Coldfire CPU

running RTEMS

and EPICS

Coldfire CPU

running RTEMS

and EPICS

PAC

DAC

slow

Example of generic LLRF control system instance

VMETemperature monitors Temperature monitors

1 trigger

IQ Modulator gives phase

and amplitude control

2 channels of 1k

samples

4 channels of 1k

samples




Phase/Amplitude Detector -> VME

Coldfire

LLRF PAD (Phase and amplitude detector)

ADCChannel

1

ADCChannel

0

ADCChannel

3

ADCChannel

2

LLRF VME(Handles phase and amplitude

conversions, feedback calculations and setting of PAC)

mvme6100

LCLSCA network

MCC workstation running lclshome GUI

Change PAD channels’ size and offsets

Set PDES, ADES and control RF feedback

Use secondary ethernet interface to send 4 channels of PAD data via UDP broadcast via private

switch. Time required: 100 nsec

- Receive I and Q averages- Convert to 4 channels of Phase and amplitude- Compute weighted averages- Trigger BSA - Do feedback calculation- Send I and Q adjustment to PAC

EVR timing trigger

Update GUIs with actual phases amplitudes, feedback

corrections, etc, every 2 seconds

Update GUIs with PAD

waveforms, temperature,

I and Q actuals every

2 seconds

Private gigabit switch




OS, BSP and EPICS versions

PAD: rtems4.9.1m68k uC5282epics-R3.14.10

VME: rtems4.9.1powerpc beatnik (mvme5500/mvme6100)epics-R3.14.8.2

PAC: rtems4.9.1 m68k uC5282epics-R3.14.10




Measuring Throughput: PAD->VME->PAC

1

2

3

4

5

6

7

8

910




Throughput steps: PAD->VME->PAC

Event (#

matches

diagram)

Absolute time after trigger (μsec)

Description

- 0.0 Trigger delivered to phase/amplitude detector digitizer (PAD)

1 7.2 ISR: signals “data ready” to wake up DAQ task





6 1963.2 PAD starts stream of processed values to VME over private net

7 2349.6 PAD streaming completed. VME parses, does feedback, sends.

8 2508.0 Phase/amplitude controller (PAC) receives new setpts from VME

9 2524.0 PAC writes to FPGA which calcs new WF to send next trigger

10 2529.2 Data is ready




Conclusions

At 120 Hz operation, time budget=8.333 ms

LLRF PAD->VME->PAC throughput measured=2.529 ms for 4 channels of 40 points each, with no offsets,

adjust: subtract 2.5 μsec per pair of IRQ raise/lower calls (8 pairs = 20 μsec)

adjust: one socket sends to multiple PACs; add switching time




Acknowledgements

Thanks as always to Ron Akre and Klystron Department for setting up hardware, scopes and signal generators

Thanks to SLAC NAL Controls Group




Additional Information

Details of the PAD->VME transferDetails of the VME->PAC transferRF stability measurementPADPAD Block diagramLCLS LLRF website: http://www.slac.stanford.edu/grp/lcls/controls/global/subsystems/llrf




Details of the PAD->VME transfer

http://www.slac.stanford.edu/grp/lcls/controls/global/sw/epics/epics%20team%20meetings/presentations/lanIpBasic.pdfRaw ethernet packets with IP and UDP headers. Similar to BSD sockets.Solution is for low end CPU on small LAN.VME initializes, starts and stops PAD streamingWhen PAD is streaming, device support for waveform on VME parses out the values and uses them in the feedback calculations of new setpoints.




Details of the VME->PAC transfer

On VME, a subroutine record that has calculated new setpoints calls a driver routine that sends the values to the PAC via udp socket

PAC is has thread waiting to receive packet

When packet arrives, it parses out the setpoints and puts them into mem mapped FPGA




LCLS Jitter Specification for 2 Seconds is 0.14% Amplitude and 0.14 degree Phase



Short Term RF Jitter Specification for L0B are well Exceeded.This is as good as it gets – Don’t tell Physicists or they will expect it. Ron Akre 2007




About the PAD

The digitizer used is the Linear Technologies LTC2208. It was the first 16 bit digitizer chip on the market capable of running at 119MHz, it is specified to run up to 130MHz.At SLAC NAL, PAD digitizer used for RF, beam position monitors, beam charge monitors and bunch length monitors.Pohang Light Source is also using PAD for new RF system.




Phase/amplitude detector (PAD) Block

1 ohm

5VDCLM340Reg

4 X 16 bit ADC102MHz ClockLTC2208Transformer Coupled Inputs

Chan. 2

Chan. 1 FIFO 64k wordsWCLK

16bit DATA

16bit DATA

WCLK

CHAN 1RF INPUTRP N

CHAN 2RF INPUTRP N

ZX60-4016E

MIXER

LORFIF

MIXER

LORF

IF

RF Board25.5MHz IF

Control Board

Control

FIFO 64k words

FIFO 64k words

FIFO 64k words

WCLK

16bit DATA

WCLK

16bit DATA

RA

WE

TH

ER

NE

TC

OM

12VDCLM340Reg5 ohm

FRONT PANEL LEDs

1 ohm

10 ohm

1 A F B

110VAC

1 1 0 V A CP o w e r M o d uleC o rc o m6 E H L 1 S C GRN

RED

BRNBLU

YELBLK

REDVIO

TE62071-ND 2x9V 1.94A

5VDCLM340Reg

12VDCLM340Reg

2 ohm

5VDCLM340Reg

GRNRED

BRNBLU

YELBLK

REDVIO

TE62054-ND 2x18V 0.417A

CHAN 4RF INPUTRP N

CHAN 3INPUTRP BNC

Chan. 4

Chan. 3

MIXER

LORF

IF

QSPI5VDC0.8A x 2 Analog

5VDC0.5A DigitalLO OUTPUT

2830.5MHzFP N

24Bit Analog InputBoard

QSPI

20 pin ribbonTRIG MonFP BNC

ANALOG IN ANALOG IN

CLOCK Mon102MHzFP N

TRIG In120HzRP BNC

CLOCK IN102MHzRP N

ZN4PD1-50

CS/CLK

16 bitDATA

CONTROL /Arcturus uC5282Microcontroller Modulewith 10/100 Ethernet

ET

HE

RN

ET

LO INPUT2830.5MHzRP N

CPLD

-2dBm 10dBm

10dBm

+18dB

10dBm-8dBm

3x 75mA

FILTER 25.5MHz BP

5VD 5VA2 5VA1 12V2 12V1

Ron Akre

lcls llrf distributed control system

Documents