department of defense budget level analysis of science and ...seor.vse.gmu.edu › ~klaskey ›...

1

Department of Defense Budget Level Analysis of Science and Technology

(DoD BLAST)

SYST/OR 699Master’s Project

TeamCraig Jayson (SE)Josh Singh (SE)

Phillip Sutton (OR)

Sponsor

Systems Planning and

Analysis, Inc. (SPA) is a

professional services

contractor located in

Alexandria, VA.

Project

This project is a

continuation of a capstone

project selected last year

that supported categorizing

and analyzing past,

present, and future budget

submissions.

Purpose of BriefProvide an overview of the DoD BLAST team’s work on the Budget Exhibit Analysis Tool (BEAT). The team will cover where we started, what we did, and recommendations for future work.

Agenda

2

Problem Definition

Scope Architecture Data & Visualization

Way Forward

• Use Cases

• Structured Analysis

• BEAT Methodology

• Key Definitions

• Background

• Problem Statement

• Stakeholders

• BEAT Database Update

• Program Analysis Dashboard

• Project Analysis Dashboard

• Reporting Functions

• Scope

• Requirements

• Deliverables

• Third Offset Example

• Recommendations

• Conclusion

Key Defenitions

3

Problem Definition

Scope Architecture Data/Visualization Way Forward

Presidential BudgetThe president submits a comprehensive budget request to Congress in early February which outlines the administration’s policy and funding priorities and economic outlook for coming year.

DoD BudgetThe annual appropriation of discretionary spending that goes to any military related expenditures.

Source: OMB

RDT&ERDT&E appropriations finance research, development, test and evaluation efforts performed by contractors and government installations to develop equipment, material, or computer application software.

Budget ActivityA category within each appropriation and fund account that identify the purposes, projects, or types of activities financed by the appropriation or fund

Program ElementThe primary data element in the RDT&E budget and the smallest aggregation of resources used by the Office of the Secretary of Defense (OSD) for analysis.

Background (1 of 3)

In support of the National Military Strategy (NMS), the Department of Defense (DoD)budget places emphasis on a number of items to include the need to invest in futurecapabilities to ensure the U.S. remains a modernized, technologically superiorgloabal force that is able to project power across the globe. This funding is found inRDT&E appropriation.

4

Supplemental Funding

Figure: DoD Budget from 1950 – 2025 in FY16 dollar (future years estimated)

Figure: FY 2016 DoD and OCO budget (source: DoD.Gov)

Research, Development, Testing & Evaluation (RDT&E) makes up 11.8% of the total DoD budget in FY 2016.

Problem Definition


http://www.jcs.mil/Portals/36/Documents/Publications/2015_National_Military_Strategy.pdf

https://www.defense.gov/News/News-Releases/News-Release-View/Article/652687/department-of-defense-dod-releases-fiscal-year-2017-presidents-budget-proposal

Background (2 of 3)

Last Year: System Planning and Analysis teamed with students in the Master’s of Systems Engineering and Operations Research to initiate Phase 1 of a multi-year effort to develop a Decision Management System that assesses trends in Science & Technology (S&T) investing within DoD. 5

DOD Programs (Found in DoD RDT&E)

01 Strategic Forces

02 General Purpose Force

03 Intelligence and Communication

04 Mobility Forces

05 Guard and Reserve Forces

06 Research and Development

07 Central Supply and Maintenance

08 Training, Medical, and Other

09Administrative and Associated

Activities

10 Support of Other Nations

11 Special Operations Forces

Budget Activity

BA-1 Basic Research

BA-2 General Purpose Force

BA-3Advance Technology

Development

BA-4Advance Component and

Prototypes

BA-5System Development and

Demonstration

BA-6 RDT&E Management Support

BA-7 Operational System Development

PE

Number

PE Name

063286E Advanced Aerospace Systems

0603211F Aerospace Technology Dev/Demo

Problem Definition


PE Number Key06 – DoD Program

3 – Budget Activity

28 – Serial Number Assigned

6 – Equipment / Activity Type

E- Code for Service / Agency

PE / BLI Program Element / Budget Line Item (BLI) Title

Include

in

TOA

FY 2015

(Base &

OCO)

FY 2016

Base

Enacted

FY 2016

OCO

Enacted

FY 2016

Total

Enacted

FY 2017

Base

FY 2017

OCO

FY 2017

Total Classification

0603286E Advanced Aerospace Systems Y 123,292 173,631 0 173,631 182,327 0 182,327 U

0603287E Space Programs and Technology Y 172,504 126,692 0 126,692 175,240 0 175,240 U

0603739E Advanced Electronics Technologies Y 81,119 76,021 0 76,021 49,807 0 49,807 U

0603760E Command, Control and Communications Systems Y 229,945 201,335 0 201,335 155,081 0 155,081 U

0603766E Network-Centric Warfare Technology Y 350,323 425,861 0 425,861 428,894 0 428,894 U

0603767E Sensor Technology Y 283,905 240,127 0 240,127 241,288 0 241,288 U

PE: 0603286E – Advanced Aerospace Systems – FY17 Total $182,327,000

Background (3 of 3)

6

BEAT Version 1.0

Budget Exhibit Analysis Tool (BEAT) is a Decision Support Tool that that allows users to visualize RDT&E data based on selective queries in order to conduct follow-on exploratory analysis.

6

Problem Definition


Problem StatementThere is no known method or available tool to digest and depict multiple budget cycles(POM) to facilitate the analysis and characterization of past, current, and futurebudget information. The Fall 2016 Capstone team was successful in aggregatingbudget data into a database and creating an Excel tool that performed such analysis atthe PE level.

Understanding that there is important information available at the moregranular project level, the BEAT has the capacity to provide furtherinformation and insights, but currently that capability has not been realized.

7

• DoD Budget data is disbursed into many separate documents/locations

• Manual formatting of data standard of practice

• No efficient way exists to isolate and visualize data to meet user-specified criteria

Problem Definition


Stakeholders

8

Figure: Stakeholders include publishers, resourcing, and recipients (source: gmuseorspa.wordpress.com)

Industry: Private entities whose interests and level of involvement with the DoD vary significantly

FFRDCs/UARC Research partnerships in which the DoD is the sponsor of a research facility which is under the administration of either a corporate entity (FFRDCs) or aUniversity.

Community of Interest: A network of technical groups that serve as a lasting structure to integrate withexperience and technical expertise.

DoD: This includes the Services and the S&T community. They receive the tech. andsupport and collaborate with multiple stakeholders.

OSD: Primary defense policy advisor to the President andresponsible for formulatingand executing policy through the DoD.

President/Congress: The DoD submits an annual budget request to the President for review. The President then submits a budget request to Congress. This request is reviewed and subjected to a process of revisions before it is returned to the President, at which point it is either signed into law or vetoed.

Problem Definition


Scope/RequirementsThe project team for Spring 2017 Capstone team will focus on taking the PEs and continuing to add to

the database by including all project level data (excludes classified data).• Time Frame of Study (2011 – 2020) this covers FY’16 PB and POM.

9

Highest Level Requirements

MR1: The BEAT system shall be able to ingest project level information and data from

DoD documents that summarize PEs.

MR2: The BEAT system shall ingest project level description information for each project.

MR3: The BEAT system shall ingest five future years of expected project level data.

MR4: The BEAT system shall include accomplishment level data for specific projects.

MR5: The BEAT system shall visualize RDT&E, specifically project level data, over a pre-determined period.

MR6: The BEAT system shall be able to display Program Budget (PB) new

program/project/accomplishments have started anew or ended.

Link to Requirements Diagram

Problem Definition


10

Structural Analysis

A Process Oriented Approach of

BEAT

Use Cases (Two Primary)

Activity Modeling

• IDEF0 (A-0, A0, A5 diagrams)

Data Modeling

• Entity Relationship Diagram

Rule Modeling

• Decision Table

Dynamic Modeling

• Sequence Diagrams

Source: SYST 520 course at https://mymasonportal.gmu.edu

Use Cases

Problem Definition


https://mymasonportal.gmu.edu/

BEAT Version 1.0 (Fall 2016 Capstone Project)

R2 Output

Problem Definition


R1 Data

R1 Data: The R-1 document contains budget information for all RDT&E programs undertaken by the branches of the DoD. For each FY, the document lists the budget for the previous two FY, as well as the budget for the current FY.

R2 Data: The R-2 document is specific to each program, identified by a PE number in the R-1. The R-2 serves as the item justification for each program, as the proposed budget for the next four FYs.

12

Database Update

12

• Presidential Budget XML file parsing was migrated from Java to VBA and the databasewas moved from a MySQL database to MS Access

• The BEAT has been linked to the Access Database and two new worksheets have beencreated to query the data

• The current data set goes back to 2011 and contains 4366 Program Element Recordsacross the DoD comprising of 9388 Project Records

Problem Definition


BEAT Version 2.0 Methodology

13

http://www.dtic.mil/dtic/

Download PB data from DTIC

(Manual Process)

Open BEAT and Select

“Upload new R2 files”

BEAT (Excel) Opens Access and

runs the XML parsing Code

Excel Sheets Update Existing

Queries and Connections

Visualization

Visualization Tool within BEAT

that allows user to graphically

display data based on inputs

Input Cells for

scoping data

User

conducts

exploratory

analysis and

informs

managementDescriptive

Information

Problem Definition


14

• BEAT Program and Project Analysis Dashboards allows the User to analyzea specific program or projects funding trends and relate those variations tothe notes from the budgets all in a single environment

Program/Project Analysis Dashboard

Problem Definition


15

Program Analysis Dashboard (1 of 6)

• The User can narrow the organizations and programs by selecting a (or all) BudgetActivity of Interest

• From there, the User selects among the organizations and agencies that haveprograms under the selected Budget Activity

• The “Select Program” options are then updated and the user can select a Programbelonging to the organization or agency by name of Program Number

Problem Definition


16


• Additionally, the User can search for a program using the “Program Search” function• The User can filter all available Programs by Service/Agency, Budget Activity, or

Keyword Searches• Once a Program has been identified, the User can select that Program and the

Program Analysis Dashboard will update to the selected Program

Problem Definition


17

Program Analysis Dashboard (3 of 6)• Information about the Program selected is populated from the database:

• Program Description

• Earliest/Latest PB Years Available

• Program Notes Per PB Year

• Additionally there is a button below the Program Description to Select thecurrent Program for analysis of its Projects, described later

Problem Definition


18


• Chart plots funding across the various Presidential Budget years providingthe User a visual representation of the changes in funding across thebudgets

• Includes the data in a table below the chart to provide values for data

Problem Definition


19


• Summary statistics are generated to provide metrics describing the behaviorof each Fiscal Years budget projections

• Statistics include Mean, Standard Deviation, Variance, Min, Median, Max,and Range

• A stable budget will have minimal Standard Deviations and Variancesbetween budget submissions

Problem Definition


20


• A heat map highlights the variance in the deltas between PresidentialBudgets (PB)

• The N/A is used to indicate that there is no delta to calculate:(In PB14 we have a prior year data point in FY 2013, however in PB15, there is no value for FY13, resulting in an #N/A)Note: The PB covers current FY and next FY and then 4 years out (tied to the POM)

Problem Definition


21

Project Analysis Dashboard (1 of 4) • The Project Analysis allows the User to investigate the different projects

that belong to each program

• The down selecting process follows the same as the Program AnalysisDashboard, but adds a Project Selection by Project Name or ProjectNumber

Problem Definition


22

Project Analysis Dashboard (2 of 4)• Similar to the Program Analysis Dashboard, the User can filter all available Projects

by Service/Agency, Budget Activity, or Program and Project Keyword Searches

• Once a Project has been identified, the User can select that Project and the ProjectAnalysis Dashboard will update to the selected Project

Problem Definition


23

Project Analysis Dashboard (3 of 4)

• The Project Mission, (Earliest and Latest PB Data Years), and Notes are allimported for the select project, the same as the Program AnalysisDashboard

• Additionally, the tool provides the funding chart and heat map to highlightthe deltas in funding

Problem Definition


24

Project Analysis Dashboard (4 of 4)

• Removed from the Project Analysis Dashboard are the summary statistics• In their stead is a pie chart that describes the all the Projects under the

governing Programs control• This pie chart provides insight as to how the Project selected compares to

the other Projects in that purview

Problem Definition


25

Reporting Functions: Delta Analysis

• The BEAT also has an analytical feature that will analyze all Programs and Projectsto identify Programs/Projects that have user defined budget variations as apercentage of the budget

• Those Programs and Projects that exceed the delta threshold are logged in a reportthat can be reviewed and then explored further within the Program and ProjectAnalysis Dashboards

Problem Definition


26

Reporting Functions:ID New or Recently Ended Programs

• The BEAT will also parse through both Program and Project Level data to identifyprograms that are new to this years PB data or have ended in the last 3 years

Problem Definition


BEAT v1.0

• JAVA XML Parsing

• MySQL Database

• Database not directly linked to BEAT

• Manual data extraction, transformation, and loading from MySQL into BEAT

• Visualizes R-1 and R-2 data at the Program Element level

• Requires the purchase of Smart Charts for Excel by DEVSCOPE for R-1 Visualizations

•Compares multiple R-2 Programs in a single Presidential Budget year

BEAT v2.0

• VBA XML Parsing

• MS Access Database

• Access database directly linked to queries within BEAT (MS Excel)

• Queries automatically refresh after new data is uploaded into Access

• Program Analysis Dashboard allows for in-depth analysis of a specific program

• Project Analysis Dashboard allows for in-depth analysis of a specific project

• Reporting Functions

• Delta Analysis analyzes all data to identify programs and project with significant variation

• New/Ended Program Identification

BEAT Comparison

Scope Architecture Data/Visualization Way ForwardProblem Definition

The updates to the database and XML parsing resulted from sponsor

request during initial meetings

27

Deliverables

28

BEAT continued refinement and project level data uploaded

BEAT user interface improvement with ability to visualize new information

BEAT user manual updated

BEAT test case for final presentation (example of BEAT – the JLTV)

Written report and briefing summarizing results of the analysis

Website with all products (https://stbudgetgmuspring17.wixsite.com/gmuproject)

Listing of all sources

Visualization Tool that identifies when projects begin and end

Identification of future budget trends of projects that are nested with Office of Secretary of Defense’s offset strategy (Next Slides will Provide Example)

Problem Definition


https://stbudgetgmuspring17.wixsite.com/gmuproject

Third Offset Strategy Technological superiority has been a staple of the DoD strategy since post WWII.This philosophy and follow-on investment has allowed U.S. to out pace its peers andremain at the forefront of battlefield innovation. However, since the current budgetdrawdown and the military mission shift to counterinsurgency (COIN), it is no longersafe to assume technological superiority.

29

Problem Definition


TOTAL RDT&E BUDGET AUTHORITY BY SERVICE (FY 2001–FY 2017)

Counterinsurgency

Drawdown

Source: http://csbaonline.org/uploads/documents/CSBA6196-2017-Budget-Analysis

OpportunityDoD investing $3.6 billion in FY 2017 and $18 billion over the FYDP (in current year dollars) in capabilities related to the Third Offset Strategy.

Third Offset Strategy (if maintained) provides significant opportunities for future RDT&E

investment- should observe trends of increased investments in these capabilities.

Third Offset Strategy (PE and Project Examples of Tool)

The President’s FY2017 budget resulted in DoD investing in anti-access technology, undersea warfare capabilities, human-machine collaboration, precision munitions, and asymmetric capabilities (EW, Cyber, Space, Laser).

PE and projects of interest (should see an uptick in funding)

I) U.S. Army – High Energy Laser (PE: Advanced Weapons Technology)

II) U.S. Airforce- Cyber Technology Pattern Recognition and Autonomous Learning (PE: Aerospace Sensors)

30

Problem Definition


Existing Project with

Uptick in future spending

New Project

Recommendations

31

Problem Definition


Retrieve and house accomplishment data in the database

Improve the database by creating more tables and linking them appropriately

Explore and incorporate predictive analytics tools such as Linear Regressions, Support Vector Machines, K-Nearest Neighbors, etc. to better analyze data (may require more advanced analytical software packages/programming languages Julia, Python, R, SAS)

Explore and incorporate textual analytics on the program/project descriptions and notes to investigate correlations between keywords/phrases and budget projections (may require more advanced analytical software packages/programming languages (Julia, Python, R, SAS)

Continue to link policy (National Military Strategy, Defense Innovation Initiative) to budgetary trend analysis using BEAT (Significant opportunity for future RDT&E investment if current administration and DoD continues to invest in a Third Offset Strategy)

Upgrades to BEAT:•Heat Chart with conditional formatting based on relative changes•Link each program to its page in the corresponding budget file

Program/Project/Accomplishment Data• Projects, Accomplishment, and Plans

data fields are verbose• E.g. NLP, Support Vector Machine, K-

Nearest Neighbors

Conclusion

Decision Management System• Begin with the Decision in Mind• Transparent & Agile• Predictive• Test, learn, continuous improvement

FYDP DataLends itself to data analytics because:• Abundance of training data• Structured• Released annually PB17

Mile

1.0

Mile

5.0

Mile

20

Mile

10

Mile

2.5Program/Project Data• Create repeatable process for

importing future data• Run surface level analytics to

identify trends, significant changes, or beginning/ending of funding

• BEAT can present data in minutes

32

Next Step

Problem Definition


Acknowledgements

• System Planning and Analysis, Inc. – Andy Dichter, Slider Griese, Chris Anderson, and team

• SEOR Faculty – Prof. Laskey

• OR /SYST 699 Classmates

33

Questions

34

Backup

35

Project Acronyms

36

Acronym Name Definition

BA Budget Activity A Budget Activity (BA) is a category within each appropriation and fund account that identify the purposes, projects, or types of activities financed by the appropriation or fund

BEAT Budget Exhibit Analysis Tool

Decision Management Support Tool that allows users to visualize RDT&E data based on selective queries in order to conduct follow-on exploratory analysis

DoD Department of Defense The executive branch department of the federal government of the United States charged with coordinating and supervising all agencies and functions of the government concerned directly with national security and the United States Armed Forces

FFRDC Federally Funded Research and Development Centers

Federally Funded Research and Development Centers (FFRDCs) are public-private partnerships which conduct research for the United States Government

FY Fiscal Year Budgetary cycle for the DoD that starts in October and ends in September.

FYDP Future Years Defense Program

Summarizes resources (equipment, manpower, forces) associated by fiscal year, with DoDprograms, as approved by the Secretary of Defense.

PB President’s Budget The president submits a comprehensive budget request to Congress in early February which outlines the administration’s policy and funding priorities and economic outlook for coming year

PE Program Element The primary data element in the FYDP and normally the smallest aggregation of resources used by the Office of the Secretary of Defense (OSD) for analysis. PEs are designed and quantified to be comprehensive and mutually exclusive.

POM Program Objective Memorandum

The 5-year FYDP and presents the Services and Defense Agencies proposal on how they will balance their allocation of available resources. The POM includes an analysis of missions, objectives, alternative methods to accomplish objectives, and allocation of resources.

R1/R2 R1/R2 Data The R-1 document contains budget information for all RDT&E programs undertaken by the branches of the DoD. R-2 document is specific to each program, identified by a PE number in the R-1.

RDT&E Research, Development, Test & Evaluation

RDT&E appropriations finance research, development, test and evaluation efforts performed by contractors and government installations to develop equipment, material, or computer application software

Refrences

37

American Association for the Advancement of Science 2016 [3] - Govini. No Date.

DoD Financial Management Regulation Volume 2B, Chapter 5

Department of Defense, Future Years Defense Program (FYDP) Structure, April 2014 at

http://www.prim.osd.mil/Documents/DoD_704507_future_years_defense.pdf

Department of Defense (DoD) Releases Fiscal Year 2017 President’s Budget Proposal

Defense Technical Information Center (DTIC) – www.dtic.mil

GMUSEORSPA at https://gmuseorspa.wordpress.com/deliverables/

Military budget of the United States, Wikipedia

U.S. House of Representatives, Committee on the Budget at

http://budget.house.gov/budgetprocess/

"US Department of Defense Third Offset Standard Market Taxonomy." Analyst Report,

Washington, D.C.

http://www.prim.osd.mil/Documents/DoD_704507_future_years_defense.pdf

https://gmuseorspa.wordpress.com/deliverables/

Need StatementThere is a need to advance the BEAT system to analyze project level budget data inorder to project future trends for better decision making.

The Budget Exhibit Analysis Tool (BEAT) will provide a mechanism for categorizing andanalyzing past, present, and future Department of Defense (DoD), Research,Development, Testing & Evaluation (RDT&E) Presidential Budget (PB) submissions.

38

Mission Statement

BEAT SYSTEMUser

Problem Definition


Requirements Diagram

39Return to Scope/Requirements

Use Case 1: Load R-2 PE and Project Level Data into BEAT

40

System Boundary

Schedule: Main Use Case; must be tested before providing to sponsor

Priority: Top, must be executed

Performance Target: All project level, unclassified budget level data is available from FY’11 through FY’20

Frequency: Every time a download is required due to release of new Presidential Budget for corresponding FY

Return to BEAT Functional Description

Use Case 2: Perform Visualization of Project Level Data in BEAT

41

Schedule: Main Use Case; must be tested before providing to sponsor.

Priority: Top, must be executed

Performance Target: All project level, unclassified budget level data is available from FY’11 through FY’20

Frequency: Every time a query and filter for specific project level data is executed

Query data in order to determine what to visualize

Return to BEAT Functional Description

IDEF0 for BEAT (A-0 Diagram)

42Return to Structural Architecture

IDEF0 for BEAT (A0 Diagram)

43

Return to Structural Architecture

IDEF0 for BEAT (A5 Diagram)

44Return to Structural Architecture

Entity Relationship Diagram for BEAT

45


BEAT Decision Tables (1 of 2)(Based on IDEF0)

46


BEAT Decision Tables (2 of 2)(Based on IDEF0)

47


48

Sequence Diagrams for BEAT(Based on Use Case 1)


49

Sequence Diagrams for BEAT(Based on Use Case 2)


50

Types of Equipment / Activity

51

Types of Service /Agency