CD FY08 Tactical Plan Status

FY08 Tactical Plan Status Report forDatabase Administration

J.TrumboPresented by

06/16/08

CD FY08 Tactical Plan Status

FY08 Tactical Plan for Databases

CD DocDB Document 2223-v3FY08 Tactical Plan for Database Administration

Core Services, Databases and ApplicationsTactical Plan Goals

– Achieve “operational excellence” by following best practices for service delivery quality and change control, customer service and satisfaction, etc.

– Continue to provide a stable, professional operation of laboratory scientific and infrastructure databases, providing appropriate uptime and technical support, and which is flexible and responsiveness to user needs.

– Services “marketed” and used throughout the Laboratory (scientific & business applications).

– Contribute to the Laboratory’s successful execution of the current scientific program and the LHC, while positioning the Laboratory to successfully compete for the ILC.

– Support running experiments at appropriate levels and be ready for LHC and ILC requirements.

CD FY08 Tactical Plan Status

FY08 Tactical Plan for Databases

FL/CD/LSC/DBI/DBA FY 2008 Oct. 07-May 08

Activity   Total FTEs

Authentication Services 0.18

Computer Facility Support 0.04

Database Administration 2.51

Maintain, Support, Integrate Lab Systems 0.50

SDSS II 1.1

Sub Total 4.33

Grand Total  (incl time off not specified above) 5.33

CD FY08 Tactical Plan Status

Activities Summary: FTEs

Level 0 Activity: Core Services % of FY Complete: 66%

Personnel Usage (FTEs)

  Allocation Actual YTD   Current

Tactical Plan Level 1 Activity FTE-yrs

FTE-mos FTE-yrs (Ave/mo.) FTE-mos

% Consumed YTD FY08 Forecast

Databases      

Database Administration 6.25 75.00 6.35 50.76 68% 103%

SDSS (per budget, all groups) 2.15 26.00 1.65 13.2 50%  

SDSS (dba specifically per mou) 1.75 21.00 1.65 13.2 63% 96%

Total 8.00 96.00 8.0 63.96 67%  

     

CD FY08 Tactical Plan Status

Activities Summary: M&S (Internal Funding)

Level 0 Activity: Databases % of FY Complete: 66%

Operating & Equipment M&S CD Internal Funding

  Operations M&S Equipment M&S

Tactical Plan Level 1 Activity

FY Obligation

Budget

YTD Obligations

+ RIPS % Spent

Current FY08

Forecast

FY Obligation

Budget

YTD Obligations

+ RIPS % Spent

Current FY08

Forecast

3par support 0 0

We would like $ for this, at least 16k.

D1 support for db boxes, both exp. & infra. Portions are charged back 40k 48.8 120%

inc. nimi,d0&cdfonline.

RedHat Licenses 24k 21k 88%

Will spend additional 3k.

Oracle License 360k 341k 94%  complete %  

Total 424k 410.8k 97%   %  

       

CD FY08 Tactical Plan Status

Service Activity: Database Administration

Cdf online production 100%

Cdf offline production 150 minutes, not down but dealing with index corruption 99+%

D0 online production 317 minutes down, 99+% uptime

D0 offline production 525 minutes down, 99+% uptime

Minos production 100%

* % does not include scheduled downtimes or building disasters

• Adjusted Goals Related to this Activity [given FY08 Budget adjustments]

• NONE

• Original Goals Related to this Activity – Achieve “operational excellence” by following

best practices for service delivery, quality and change control, customer service and satisfaction, etc.

– Continue to provide a stable, professional operation of laboratory scientific and infrastructure databases, providing appropriate uptime and technical support, and which is

flexible and responsiveness to user needs.

CD FY08 Tactical Plan Status

Tactical Plan Issues

Resolution of Past Action Items: Proprietary To be clear, points below include effort from sysadmins as well as dbas!

• Day to day support of production environments for postgres, oracle, mysql and sql server, providing 24.7 environment on several production instances.

• Kerberos for oracle proof of product completed. Still working on cron implementation.• Oracle Internet Directory (oid) proof of product. Moving forward with hardware purchases for

dev/int/prd installation.• Completed rman_dcache product, using for production dbs to tape.• Configured and implemented cad dev/int/prod environments.• D0 Luminosity has hardware installed to support proposed full dev copy with int snapshot. No

time yet to establish db.• Completed transition of d0 online support to dba.• Moved d0 offline database to 3par storage array, deprecated aging Clarion.• Removed 4 billion rows of event data from database without user interruption. Recovered space. • Turned over mgmt of d0 online boxes and dbs to dba.• Implemented LHC USCMS Tier 1 FTS (file transfer service) database, dba supporting.• Continued support of US CMS databases. • Upgrade of all v10 databases to a new version.

CD FY08 Tactical Plan Status

Tactical Plan Issues

Resolution of Past Action Items: SDSS• Planning hardware/software layouts for the next DR7 and final DR8 releases. Obtaining new

machines and licenses promptly. Reconfigured number of existing machines to meet our hardware requirements for the future maintenance of the SDSS machines to keep less machines in the rack.

• Implemented 2 mirror sites for DR5 Data Release in the Libraries of JHU and University of Chicago.

• SDSS team working close with team of the librarians to setup SDSS help desk support using new web application Question Point.

• SDSS DBA team successfully completed transition CasJobs web service from SQL2K to SQL2K5.

• Completed incremental loading from DR6 to DR7.1 (BestDR7.1, TargDR7.1, SegueDR7.1 - total more than 6TB data) and released it to the SDSS collaboration on February 14 - two weeks early.

• Working close with team of developers from JHU and Princeton to implement and test new code related to schema modifications for DR7.2 with major changes in data model and merging SegueDR7.1 to BestDR7.2.

• Completed loading 3TB data into new database Stripe82, this is a new• addition to the SDSS databases in respond to the SDSS scientists to • merge stripe 82 runs into one database. • Successfully upgraded 7TB RunsDB database from SQLServer2000 to SqlServer 2005

environment. • Working constantly on SDSS documentation and improving monitoring tools to maintain stable

production environment.

CD FY08 Tactical Plan Status

Tactical Plan Issues

Resolution of Past Action Items Open Source:To be clear, points below include effort from sysadmins as well as dbas!

• Established best practices, standards, how tos for open source implementations. • Cross training as much as possible.• Established postgres general purpose boxes for dev/int/prd. Installed several users including

content_mgmt, ilc ground motion and edu office lasso project dbs.• Took mgmt of mysql general purpose boxes. they need work to upgrade and bring to stds, work

is pending.• Migrating support of gratia db to dba. • Benchmarked MyISAM vs InnoDB engines for Gratia. • Evaluated several backup products for mysql operating under InnoDB engine.• Established nova dev/int/prd postgres boxes. • Took over support of NIMI. Modifying to standardized installs, upgrade os and db.• Continuing cross training of group. Attempted to set up an onsite postgres class for developers

and dbas, but furloughs disrupted attendance planning.• Assisted minerva with mysql and postgres installs on general purpose boxes.• Established the Content Mgmt postgres database dev/int/prd.• Still not quite transitioned minos production mysql db to dba.

CD FY08 Tactical Plan Status

Tactical Plan Risks

Issues and Risks (+ impact of FY08 DOE/FNAL budget)• Furloughs and forced vacation have impacted

productivity this year. • Aging hardware continues to be an issue. CDF’s online

and offline development, D0 offline development, all are getting long in the tooth.

• dba continues to acquire additional responsibilities, allow less ‘free’ time for cross training on needed skills.

CD FY08 Tactical Plan Status

Tactical Plan Status Summary

Have maintained a high level of uptime (99%+) on all the database platforms dba manages. A measure of success is the continual requests for dba support from the lab including NOvA, Minerva, ILC, DES, SNAP, etc.

Have made headway with open source dbs, especially in the difficult first stages of setting standards and procedures for installations. Have installed several databases. User base is becoming aware of available support for open source. Progress made in cross training as well, but not there yet.

When requested moved quickly to accommodate CMS Tier 1 request for database resources.

Continued support for Oracle prod/dev infrastructure, monitoring, tuning, assisting with an array of issues.