defence presentation 331016

Fakultät für InformatikProfessur Technische Informatik

www.tu-chemnitz.de

Professur Technische InformatikProf. Dr. Wolfram Hardt

Master Thesis DefenceDesign and development of an automated

regression test suite for UEFI

Efforts

University Supervisor: Prof. Wolfram Hardt

Managerial Supervisor: Mr. Bertram Metz

Technical Supervisor: Mr. Robert Fendt

Coordinator: Dr. Ariane Heller

Master Thesis Student: Huzaifa Saadat

www.tu-chemnitz.de


Agenda

• Introduction to UEFI

• Motivation

• State of the art

Phase 1

• UEFI test tools

• Results

• Further options

• Conclusion

Phase 2

• Test driven development

Phase 3

• Code Analysis- Static analysis- Dynamic analysis

15.04.2023 Huzaifa Saadat 2

Agenda • UEFI • Motivation • State of the art • Test tools • TDD • Static analysis • Dynamic analysis • Results • Further options • Conclusion • References

www.tu-chemnitz.de


Introduction to UEFIThe future of BIOS

BIOS

• Poll driven

• Lack of extensibility

• Assembly language implementation

UEFI

• Partially interrupt driven (TPLs)

• Modular and extensible (GUIDs)

• C language implementation



UEFI/BIOS [9]

www.tu-chemnitz.de


Motivation

Matthew Garrett

Criticism: “The only people to enable UEFI are enthusiasts.”

Future: “After a few years of iterative improvements it stands a good chance of being more reliable and useful than BIOS.”



www.tu-chemnitz.de


State of the art

Issues with UEFI testing

• Debugging

• Manual testing

• Hardware diagnostics



Debugging [7]

www.tu-chemnitz.de


UEFI test tools


Chipsec PI SCT (PI Specification) fwts (UEFI Specification)


UEFI Boot Sequence [4]

www.tu-chemnitz.de


UEFI test tools (cont.)

Chipsec

• Security features

• Python scripts

• Shell, Linux and Windows

• e.g. BIOS protection



Chipsec [6]

www.tu-chemnitz.de



Platform Initialization Self-Certification Test (PI SCT)

• PI specification

• Shell application

• Verifies functions

• e.g. ResetSystem()



PI SCT [8]

www.tu-chemnitz.de



firmware test suite (fwts)

• UEFI specification

• OS perspective

• Live version

• Non-live version

• 6 test classes

• e.g. Power management



fwts [5]

www.tu-chemnitz.de



Automated regression test suite

• fwts

• Chipsec

• PI SCT

• STAF/STAX integration


Enter UEFI Shell

Run Chipsec

Run PI SCT

Boots fwts live

Exit UEFI Shell

Regression tests work flow


www.tu-chemnitz.de


Test driven development

AMI (Aptio V)

• Visual eBIOS

• Standard C lib. absent

• Metadata files


Aptio V project


www.tu-chemnitz.de


Test driven development (cont.)

Unity

• Unit test for C

• Comparison functions

• Modified for UEFI

• e.g. SerialPortWrite()


Unity and example


www.tu-chemnitz.de


Test driven development (cont.)

TDD work flow


TDD

Target Platform

Debug OutputDebug OutputBurn Serial Out

Write/Refactor code


www.tu-chemnitz.de


Static analysis

Cppcheck

• Directory check

• Code oriented

• Open source

• Light weight

PVS-Studio

• Build monitor

• Code oriented

• Proprietary

• Normal

CppDepend

• Build monitor

• Architecture oriented

• Proprietary

• Extremely heavy



www.tu-chemnitz.de


Dynamic analysis

Dr. Memory

• Dynamic memory analyzer

• Windows based

• 32 bit applications


Application with Dr. Memory


www.tu-chemnitz.de


Dynamic analysis (cont.)

Avatar

• Emulate UEFI (QEMU)

• Use target platform

• Avatar in the middle

• Run analysis scripts

• Slow and unstable

Architecture


EmulatorTarget platform

Avatar

Dynamic firmware analysis

Analysis scripts

Analysis scripts


www.tu-chemnitz.de


Results

Results criteria

• Increase test coverage

• Maximize automation

• Portability

• Check PI/UEFI specification

• Check proprietary specification

• Ability to automate regression test



www.tu-chemnitz.de


Results (cont.)

Test coverage, automation and portability


Current

Manual

Tests

fwts PI SCT Chipsec Automated

Regression Test

Suite

Approximate

Time

Required

4 weeks 5 minutes 4 hours 1 minute 4 hours and 6

minutes

Approximate

Test Cases

32 1,182 10,737 9 11,928

Approximate

Failed Test

Cases

1 74 18 1 93

UEFI Phases

Covered

Negligible (BDS, TSL, RT

and AL)

(PEI, DXE

and BDS)

(SEC, PEI) (All)


www.tu-chemnitz.de


Results (cont.)

Specifications

• PI SCT (PI Specification)

• fwts (UEFI Specification)

• Chipsec and PI SCT (open source extendible)

Automate

• Combined regression test suite

• STAF\STAX regression test framework



www.tu-chemnitz.de


Results (cont.)

Test driven development

• Possible for UEFI

• Small modifications

• Minimize bugs

• Maximize test coverage

Code analysis

• Improves C code

• Minimization of bugs

• Code review alternative

• Dynamic analysis concept



www.tu-chemnitz.de


Further options

Research

• Unit tests for UEFI

• Static analysis tools for UEFI

• Dynamic analysis for UEFI

Possible result criteria

• Minimum modification, maximum usefulness

• User-friendliness, minimum false alarms, maximum errors

• Helpfulness, performance optimization



www.tu-chemnitz.de


Conclusion

Concluding remarks

• Focus on UEFI testing

• Avoid reinventing the wheel

• Test early (TDD)

• Benefits of migration (Assembly to C)

• Try dynamic analysis


Agenda • UEFI • Motivation • State of the art • Test tools • TDD • Static analysis • Dynamic analysis • Results • Further research • Conclusion • References

www.tu-chemnitz.de


References

[1] Master_Thesis_Huzaifa_Saadat_Final.pdf

[2] Test Driven Development for embedded C – by James Grenning

[3] Beyond BIOS – by Intel Press

[4] http://tianocore.sourceforge.net/wiki/PI_Boot_Flow

[5] A. Hung, „Firmware Test Suite (fwts),“ Canonical, 2013

[6] https://github.com/chipsec/chipsec

[7] http://www.ibmsystemsmag.com

[8] Self-Certification Test (SCT) User Guide

[9] http://www.t-online.de


Agenda • UEFI • Motivation • State of the art • Test tools • TDD • Static analysis • Dynamic analysis • Results • Further research • Conclusion • References

http://tianocore.sourceforge.net/wiki/PI_Boot_Flow

https://github.com/chipsec/chipsec

https://github.com/chipsec/chipsec

http://www.ibmsystemsmag.com/

http://www.ibmsystemsmag.com/

http://www.t-online.de/

www.tu-chemnitz.de



Q&A

defence presentation 331016

Documents

agenda uefi motivation

uefi testing

huzaifa saadat

options conclusion phase

art phase

test driven development

art issues

bios protection