international test & evaluation association symposium ...assurance that test samples were...

Weapons, Evaluation and Capability Assurance

International Test & Evaluation Association Symposium

“Emerging Threats in a Complex Environment”

Dr Dai Morris

UK MoD Test & Evaluation and Complex WeaponsSenior Responsible Owner

Weapons, Evaluation and Capability AssuranceWeapons, Evaluation and Capability Assurance

Capability Assurance

WECA – What’s in a name?Evaluation WEAPONS!


Emerging T&E Threats/Opportunities Evolution of new military threats

from state and non-state actors

Reducing funding streams

Interconnectivity and complexity of new systems

Pace of technology advancement, and Near-Peer applications.

Demand for higher tempo of delivery of military capabilities

Aspirations to harness new technologies in own inventory

Impact of sovereignty issues on capability assurance

The Evaluation Community is well

aware of the emerging challenges, and is working to address

them ……


Capability Complexity

Confidence from use of bespoke test environments and assurance that test samples were representative of the production items.

BEFORE: • Mass production• Simple systems• Physical testing

NOW:Integration of Intelligent, Complex, ‘System-of-Systems’ capabilities from different industry sectors

Confidence from information-rich, connected, secure, agile and intelligent evaluation systems


The increase in complexity is recognised; we are changing the way we conduct T&E of our military capabilities increased emphasis on operationally-

representative testing. Recognition of the importance of

testing in a coalition environment Agility in the exploitation of our S&T

and Innovation portfolios, Big Data and many others

New processes, new methods…..

Do our traditional approaches provide the right understanding to the right people?

UK T&E ‘Tube/Process’ Map

Capability Complexity


Understanding the T&E Enterprise

Science & Technology Capabilities

Industry Capabilities

International capabilities

Test ranges

MOD Capabilities

T&EEnterprise

The future is about mastering the interfaces

Finance

Through Life

Costs

Industry Strategy

Anouncables

Stakeholders

Innovation

Near-Peer Application


A missing critical System component? The ‘Operating Environment’ is critical

to T&E● The T&E community is highly

trained and motivated to ‘Assure’

But do all our stakeholders connect to our community and understanding?● And are they similarly motivated?

-Megaprojects and Risk: An Anatomy of Ambition (2003, Bent Flyvbjerg)

Remember the ‘Operating Environment’ ‘Threat’


Language is Important “Integrated Test, Evaluation and Acceptance Plan”

● OK – just a hurdle to Acceptance; go around

“Acceptance”● Nothing about through life sustainment; one shot

“Investment Scrutiny”● Its about the customer process, not the supplier

End User● Let’s focus on the current SRO (or maybe DOTE…);

we can ignore all the other stakeholders

The T&E Community understands: Do others?


Stakeholder Messaging 1Focus on the Myth-Busting: Testing delays programmes

T&E is always over-specified

Aims to break equipment/systems

Stifles innovation

Needs to be conducted as part of acceptance only – i.e. can’t happen earlier

Is the most costly part of any programme

T&E does not make programmes late or reject good ideas. It simply highlight existing issues for resolution!


Stakeholder Messaging 2 We need to be careful of negative messaging:

● “if you don’t do T&E then….” – the ‘big whip’ tactic!

More positive messaging might land better:● “by embracing T&E early, your programmes have a better

chance of being cheaper, on time, more resilient”

Let’s seek to proactively change the nature of the T&E debate to focus of something Senior Decision Makers and programme managers value, rather feel compelled to do.

The critical outputs T&E enables must be ‘sold’ outside the community: And we must use bespoke messaging


Stakeholder Messaging 3 Make the messages bespoke and understandable.

● And you may need to invert cause and effect.

T&E should be seen as an enabler to reduce organisational pressures, rather than as a blocker to demands

0

10

20

30

40

50

60

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Annu

al T

&E

Spen

d

“Programmes must spend x% of total budget on T&E, and y% of that in the first Z years”


UK T&E Benefits Analysis Tool (TEBAT) Helps program Senior Stakeholders prioritise T&E and to understand

where and why they will get most benefit;

Provides confidence that risk areas are addressed, and allows assessment of how well the planned T&E mitigates the project / programme risks;

Provides clear metrics and context of benefits and an indication of VFM;

Uses a simple Excel tool to deliver a ‘plan-on-a-page’ summary for Senior Responsible Officer (SRO) review.

What started as a planning tool, is now an engagement tool


The Live / Virtual Balance Requires Validated and Verified (V&V) models;

Clear distinction between discovery and validation evaluation;

Ability to blend Live/Virtual, to test hardware in the loop;

Be a credible and cost-effective option to live-testing – e.g. provide similar confidence levels as real-world tests would provide the operator;

Be integrated into programme T&E strategies from the outset;

Address and integrate the human component –noting it is difficult to simulate the human response, especially at higher levels of command.


Using the Live / Virtual to Best EffectSea Ceptor Showed how T&E can be optimised to save

money & time without sacrificing confidence; <10 live firings!

Through life, fully V&V’d models available;

Lab-based integration of ship equipment & Combat Management System;

Compatibility laboratory, Chambers for seeker & data-link trials and Hardware in the Loop;

Type 23 platform system “on land” to demonstrate Ceptor functioning with Medium Range Radar & Combat Management System.

Success from Finance & Capability Senior Stakeholders


Some Final Thoughts Engaging Senior (non-T&E) leadership is essential

● Engaging SROs in a risk-based approach to define T&E Programme Time/Cost/Risk profiles essential.

● We must be better at harnessing & presenting our data to support wider Stakeholders’ organisational interests.

All T&E is ‘artificial’ as it is necessarily conducted in non-operational settings● Or is it?● Increasingly synthetic-testing is driving live-testing;

To succeed, we must think outside the Organisational Box


Questions?

international test & evaluation association symposium ...assurance that test samples were...

Documents