Double Trouble: Fuzzing Kernel ModulesDerrick McKee, Prashast Srivastava, Hui Peng, Mathias Payer

Abstract

Despite its importance, the Linux kernel remains a difficult target for fuzzers. While prior work, such as syzkaller, allows for fuzzing the kernel, it is still difficult to fuzz arbitrary loadable kernel modules (LKMs). These LKMs represent the majority of source code in the Linux kernel, as well as a major source of CVEs.

We present Double Trouble, a general framework for fuzzing LKMs. Double Trouble requires a one-time minimal programming effort to support fuzzing any subsystem. Once that effort is complete, any LKM within the subsystem can be fuzzed from the two possible sources of inputs: an upper half direction representing a system call execution, and a lower half direction representing either hardware interrupt or exposed kernel function. As a case study, we are adding support for fuzzing filesystems. We use the state-of-the-art generational fuzzer, af-fuzz, for the upper half fuzzing, and a deterministic fuzzer for the lower half.

Problems

●568 critical vulnerabilities found in the Linux kernel last year.

●Emulating a whole system drastically limits fuzzing throughput.

●The complexity of the kernel also greatly hinders the fuzzer code coverage.

●Often difficult to fuzz specific modules.

mutex_lock

_cond_resched

msdos_format_name

strchr

fat_scan

strncmp

fat_get_short_entry

__brelse

__warn_printk

mutex_unlock

current_time

current_kernel_time64

fat_alloc_new_dir

fat_free_clusters

fat_alloc_clusters

__getblk_gfp

fat_time_unix2fat

__memset

mark_buffer_dirty_inode

fat_zeroed_cluster.constprop.25

msdos_add_entry

fat_add_entries

fat_sync_inode

__mark_inode_dirty

inc_nlinkfat_build_inode

fat_iget

new_inode

iunique

fat_fill_inode

iput

fat_attach

__insert_inode_hash

set_nlink

d_instantiate

security_d_instantiate

_raw_spin_lock

__d_instantiate

fat_flush_inodes

writeback_inode

fat_collect_bhsmark_fsinfo_dirtyfat_mirror_bhs

fat_sync_bhsfat_ent_read

blkdev_issue_discard

__fat_fs_error

msdos_mkdir

d_flags_for_inode

printkvprintk

fat__get_entry

inode_needs_sync

__find_get_block

pagecache_get_page

try_to_free_buffers

alloc_page_buffers

init_page_buffers

unlock_page

__put_page

dump_stack

time64_to_tm

mark_buffer_dirty

__bforget

fat_msg

__memcpy

sync_dirty_buffer

__fat_remove_entries

fat_add_new_entries

fat_chain_add

block_dump___mark_inode_dirty

inode_io_list_move_locked I_BDEV

write_dirty_buffer

__wait_on_buffer

sync_inode_metadata

__bread_gfp

___ratelimit

fat_bmap__breadahead

page_mapping

__set_page_dirty

writeback_single_inode

out_of_line_wait_on_bit_lock

rcu_all_qs__schedule

__wait_on_bit_lock

d_find_alias

wb_io_lists_depopulated

blk_start_plug

__blkdev_issue_discard

submit_bio_wait

bio_put

blk_finish_plug

panic

mark_page_accessed

find_get_entry

__page_cache_alloc

add_to_page_cache_lru

__lock_pagedrop_buffers

free_buffer_head

__cancel_dirty_page

alloc_buffer_head set_bh_page.part.29

__page_cache_release

vprintk_func

dump_stack_print_info show_stack

__inode_wait_for_writeback

mapping_tagged

__writeback_single_inode

inode_to_wb_and_lock_list

inode_sync_complete inode_io_list_del_locked

submit_bh_wbc.isra.46

unlock_buffer

out_of_line_wait_on_bit

_raw_spin_trylock

_raw_spin_unlock_irqrestore

printk_deferred

ll_rw_block

rcu_note_context_switch

switch_mm_irqs_off

__switch_to_asm

finish_task_switch

deactivate_task

update_rq_clock.part.80

___perf_sw_event

enter_lazy_tlb

__perf_event_task_sched_out

__delayacct_blkio_start

wq_worker_sleeping

__schedule_bugprofile_hits

__balance_callback

activate_task wq_worker_waking_up

ttwu_do_wakeup.isra.82ttwu_stat.part.49 __delayacct_blkio_end

vprintk_deferred

igrab

new_inode_pseudo

inode_sb_list_add

get_seconds

fat_calc_dir_size

fat_subdirs fat_time_fat2unix

_atomic_dec_and_lock

list_lru_del

write_inode_now inode_lru_list_add

rcu_sched_qs rcu_dynticks_momentary_idle

load_new_mm_cr3

rcu_irq_enter_disabledrcu_irq_enter_irqsonrcu_irq_exit_irqson

__pti_set_user_pgdclear_asid_other

__perf_event_task_sched_in

__mmdrop

kprobe_flush_task

put_task_stack __put_task_struct

sched_clock_cpu

perf_exclude_event perf_swevent_event

perf_pmu_sched_task perf_iterate_sbtask_ctx_sched_out perf_event_update_timeperf_event_update_userpage

ktime_get

kthread_data

print_modules

_raw_spin_lock_irqsave

check_preempt_curr

__const_udelay

panic_smp_self_stopbust_spinlocks

vsnprintf

printk_safe_flush_on_panic

__crash_kexec

crash_smp_send_stop

atomic_notifier_call_chain

kmsg_dump

debug_locks_off console_flush_on_panic

emergency_restart

__mod_node_page_state

__mod_zone_page_state

vprintk_emit

irq_work_queue

fat_get_clusterfat_ent_write

fat_cache_inval_inode

strcspnprint_tainted

queued_spin_lock_slowpath

resched_curr

activate_page

fat_cache_add.part.10

kmem_cache_free

list_lru_add

__printk_safe_enter

vscnprintf

cont_flush

log_store

__printk_safe_exit

cont_add

__down_trylock_console_sem.isra.20console_trylock.part.21

console_unlock

llist_add_batch arch_irq_work_raise tick_nohz_tick_stopped

native_sched_clock

sched_clock_local

recalc_bh_state.part.44

perf_iterate_ctxcalc_timer_values arch_perf_update_userpage

finish_wait

prepare_to_wait_exclusive

bio_alloc_bioset

bio_add_page

guard_bio_eod

submit_bio

__wait_on_bit

mempool_alloc

punt_bios_to_rescuer

bvec_alloc

__kmalloc

mempool_free

__disk_get_part

generic_make_request

__bdevname

this_cpu_cmpxchg16b_emu

__slab_free

cpumask_next

prepare_to_wait

kmem_cache_alloc fat_cache_merge

cmpxchg_double_slab.isra.73 free_debug_processing

using_native_sched_clock sched_clock_stablecyc2ns_read_begin cyc2ns_read_end

remove_element.isra.9.part.10io_schedule_timeout kmalloc_slab __slab_alloc

log_make_free_spacedown_trylock

__up_console_sem.isra.19 scnprintf

msg_print_text

sprintf

msg_print_ext_body

___slab_alloc

generic_make_request_checks

blk_queue_enter

bio_endio

account_page_dirtiedradix_tree_tag_set

__page_file_indexpagevec_lru_move_fn alloc_inode

release_pages

module_flags

up

bit_waitqueue

do_writepages is_bad_inodefilemap_fdatawait_range inode_add_lru

init_object

on_freelist

check_objectset_track

print_hex_dump

slab_errslab_fix

slab_bug

print_trailer

check_slab

unblank_screenconsole_unblank

format_decode

number

pointer

string

set_precisionset_field_width

__sw_hweight64

mutex_trylock crash_setup_regs crash_save_vmcoreinfomachine_crash_shutdown

machine_kexec

console_trylock

__cmpxchg_double_slab.isra.61

get_partial_node.isra.84

gfp_pfmemalloc_allowed

deactivate_slab.isra.82

mempolicy_slab_node__cpuset_node_allowed

__next_zones_zonelist

new_slab

slab_out_of_memory

alloc_debug_processing

next_bio

wait_for_completion_io

blk_status_to_errno

blk_flush_plug_list

check_bytes_and_reportsave_stack_trace

hex_dump_to_bufferadd_taint

print_tracking.part.58

slab_pad_check.part.64

rcu_irq_enterrcu_irq_exit

print_track

vmcoreinfo_append_str

relocate_kerneldisable_IO_APIC

radix_tree_lookup_slot

__alloc_pages_nodemask

cpuset_mem_spread_node

__add_to_page_cache_locked

lru_cache_add

workingset_refaultworkingset_activation

_raw_spin_lock_irq

io_schedule

ctx_sched_out

perf_event_sched_in.isra.117

pgd_free

destroy_context_ldt

__mmu_notifier_mm_destroy

kfreerecycle_rp_inst

account_kernel_stack arch_release_thread_stack

vfree_atomic

__free_pages

cgroup_free

security_task_freeexit_creds profile_handoff_task

queue_work_on

module_flags_taint

memchr_inv

__put_compound_pagefree_unref_page_list

find_next_bit

__radix_tree_lookup

bio_chain

free_ldt_struct.part.6

put_cpu_partial

discard_slab

__next_node_in

init_tracking.part.68

mod_node_page_state alloc_pages_current

count_partial

__up.isra.2

group_sched_out.part.103

ctx_sched_in.isra.115

clear_IO_APIC

__inc_zone_page_state__inc_node_page_state

percpu_counter_add_batch

mod_zone_page_state

console_lock

blk_mq_flush_plug_list

list_sort

queue_unplugged __elv_add_request__blk_end_request_all

PageHugefree_unref_page_preparefree_unref_page_commit.isra.92

congestion_wait

write_cache_pages__filemap_fdatawait_range

pagevec_lookup_range_tag

wait_on_page_bit_common

__pagevec_release

event_sched_out.isra.100

unfreeze_partials.isra.83

find_get_pages_range_taglru_add_drain_cpu

handle_bad_sectorcreate_task_io_context

init_wait_entry

schedule

prepare_to_wait_event

io_schedule_prepare

schedule_timeout

free_pages

inode_init_always

down

security_inode_alloc __init_rwsem

skip_atoi

put_dec

dentry_name pointer_string siphash_1u64 restricted_pointeruuid_string resource_stringdevice_node_string.isra.5 netdev_bitsmac_address_string escaped_stringclock ip4_addr_string bitmap_list_string.isra.6address_val symbol_string hex_stringbdev_name.isra.8flags_string bitmap_string.isra.7ip6_addr_string_saip6_addr_string ip4_addr_string_sa

lock_timer_basedetach_if_pending

__internal_add_timer trigger_dyntick_cpu.isra.35del_timer_sync

get_nohz_timer_target

get_page_from_freelist

__alloc_pages_slowpath

policy_nodemaskpolicy_nodeget_task_policy.part.41

jiffies_to_usecs

wait_on_page_bit

clear_page_dirty_for_iotag_pages_for_writeback

print_prefixmemchr

snprintf

node_random

group_sched_in

perf_event_set_state.part.76

radix_tree_maybe_preload page_cache_tree_insertlruvec_lru_size

find_first_bit

refcount_dec_not_one

put_css_set_locked

__put_cred

__queue_work

clear_IO_APIC_pin

pagevec_move_tail

has_capability_noaudit string_escape_mem__clk_get_name clk_get_rate

ip4_string

sprint_symbolsprint_symbol_no_offset sprint_backtracestrlen ip6_string ip6_compressed_string

__down

kmem_cache_alloc_node

radix_tree_next_chunk

blk_mq_sched_insert_requests __blk_run_queue blk_run_queue_async elv_drain_elevatorelv_attempt_insert_merge elv_rqhash_add __pm_runtime_resume__blk_end_bidi_request

ioapic_read_entry __eoi_ioapic_pinioapic_mask_entry ioapic_write_entry

cpuset_nodemask_valid_mems_allowed

free_pcppages_bulk

__radix_tree_create

shmem_mapping__radix_tree_replace

security_capable_noaudit

event_sched_in.isra.113.part.114perf_mux_hrtimer_restart

radix_tree_node_alloc.constprop.25

radix_tree_extend

node_tag_get.constprop.24

__zone_watermark_ok

node_dirty_ok

__inc_numa_state clear_page_orig

get_pfnblock_flags_mask set_pageblock_migratetype move_freepages_block

__node_distancenode_reclaim

find_suitable_fallback

steal_suitable_fallback

check_new_page_bad

__cpuset_memory_pressure_bump

try_to_free_pages

unreserve_highatomic_pageblock

drain_all_pages

__alloc_pages_direct_compact

zone_reclaimable_pages

schedule_timeout_uninterruptible

wake_all_kswapds

out_of_memory

compaction_zonelist_suitable

warn_alloc

put_dec_full8

get_random_u32

perf_log_itrace_startperf_log_throttlehrtimer_forwardhrtimer_start_range_ns

free_pages_check_bad

page_mkcleandec_zone_page_stateset_page_dirtyradix_tree_iter_tag_set radix_tree_iter_resume

fmeter_update

throttle_direct_reclaim

do_try_to_free_pages

first_online_pgdatnext_zone

flush_work

try_to_compact_pages

compaction_defer_reset

wakeup_kswapd

task_will_free_mem

oom_kill_process

schedule_timeout_killableblocking_notifier_call_chain

mark_oom_victimwake_oom_reaper.part.28

has_intersects_mems_allowed

dump_header.isra.30oom_badness

__compaction_suitable.part.38

cpuset_print_current_mems_allowed

show_mem

get_work_pool

check_flush_dependency

__init_waitqueue_head

insert_work

wait_for_completion

node_page_state

node_pagecache_reclaimable

shrink_node

blk_attempt_req_merge elv_rqhash_find rpm_resumeblk_mq_insert_requestsblk_update_bidi_request blk_finish_request

blk_update_request add_disk_randomness blk_delete_timerblk_account_io_done

blk_queue_end_tag

blk_stat_addlaptop_io_completion

__blk_put_request

allow_direct_reclaim.part.78

compaction_suitable

__delayacct_freepages_start

zone_watermark_ok_safe

__delayacct_freepages_end

blk_account_io_completion

bio_advance

blk_dump_rq_flags

blk_recalc_rq_segments add_timer_randomness

__ioapic_read_entry

part_round_stats part_dec_in_flight

elv_completed_request freed_requestelv_put_request put_io_context

__wake_up

__pm_runtime_idle

del_timer

__rpm_get_callback

dev_pm_disable_wake_irq_checkrpm_callback dev_pm_enable_wake_irq_checkrpm_idle

_warn_unseeded_randomness_raw_read_lock_irqsaveextract_crng _raw_read_unlock_irqrestore

pr_cont_kernfs_name

show_free_areas

next_online_pgdat

page_mapped rmap_walk

pgdat_balanced

shrink_node_memcg.constprop.84

shrink_slab.part.69

part_in_flight

queued_read_lock_slowpath

sum_zone_node_page_state

inactive_list_is_low.constprop.87 shrink_active_list

shrink_inactive_list

down_read_trylock up_read

fragmentation_index

kernfs_name_locked

__freed_request

attempt_merge

lock_hrtimer_base.isra.20

enqueue_hrtimer__remove_hrtimertick_program_event wake_up_nohz_cpu

ktime_get_seconds

mix_pool_bytescredit_entropy_bits

show_mem_node_skip.part.76

hugetlb_show_meminfo

show_swap_cache_info

call_rwsem_wake

zone_watermark_ok

strlcpy

calc_wheel_index

try_to_del_timer_sync

housekeeping_test_cpu

compaction_deferredcompact_zone_order defer_compaction

put_dec_trunc8

perf_output_begin __perf_event_header__init_id

perf_output_copy

__perf_event__output_id_sample perf_output_end

perf_event_pid_type

cpumask_next_and

__blk_recalc_rq_segments

find_lock_task_mm

_raw_read_lock

put_task_struct

do_send_sig_info

down_read notifier_call_chain

__thaw_task

mempolicy_nodemask_intersects

cpuset_mems_allowed_intersects

dump_unreclaimable_slab

int_sqrt lru_add_drain

isolate_lru_pages.isra.79

page_evictable

page_referenced

move_active_pages_to_lru

putback_lru_page try_to_release_page

msleep shrink_page_list

putback_inactive_pages

current_may_throttle

wakeup_flusher_threads

wait_iff_congested

blk_mq_hctx_mark_pending.isra.43

get_slabinfo

blk_rq_set_mixed_mergeelv_merge_requests ioprio_best

rwsem_wake

perf_event_header__init_idperf_event__output_id_sample perf_output_put_handle

__rpm_callback rpm_check_suspend_allowed

total_swapcache_pages

clear_wb_congested

__msecs_to_jiffies

__isolate_lru_page

try_to_free_swap

try_to_unmap_flushadd_to_swap

__remove_mapping

try_to_unmap

try_to_unmap_flush_dirty

inc_node_page_state

__filemap_set_wb_err

__wakeup_flusher_threads_bdi.part.53

__fragmentation_index

timerqueue_addtimerqueue_del

clockevents_switch_state

__page_mapcount

page_rmapping

add_page_to_unevictable_list isolate_lru_page

__lock_task_sighand send_signal

__dev_pm_qos_read_valuedevice_links_read_lock device_links_read_unlock

rb_insert_color

wb_wakeup

__task_pid_nr_ns

elv_rqhash_repositionelv_rqhash_del.part.12

_raw_spin_lock_bh mod_delayed_work_on

__rwsem_mark_wake.constprop.5 wake_up_q

call_rwsem_down_read_failed

wake_up_process

page_swapcount

pm_suspended_storagedelete_from_swap_cache

arch_tlbbatch_flushget_swap_page

add_to_swap_cache

put_swap_page __delete_from_swap_cache

__delete_from_page_cache workingset_eviction

rmap_walk_locked

errseq_set

try_to_grab_pending__queue_delayed_work

task_active_pid_ns

rb_eraserb_next

cpumask_any_but

native_flush_tlb_others flush_tlb_func_common.constprop.11

rwsem_down_read_failed

_swap_info_get

try_to_wake_up

get_swap_pages

__drain_swap_slots_cache.constprop.1

radix_tree_maybe_preload_order__add_to_swap_cache

__swap_entry_free

__radix_tree_insert

radix_tree_delete

plist_requeue

__del_from_avail_list

scan_swap_map_slots

drain_slots_cache_cpu.constprop.2

swapcache_free_entries

kvfree

unaccount_page_cache_pageradix_tree_clear_tags

__swap_info_get

swap_count_continued

pwq_dec_nr_in_flight

pwq_activate_delayed_work

set_task_cpu

__task_rq_lock

ttwu_do_activate.isra.83 select_fallback_rq

dump_pagenode_tag_clear

set_task_rq_fair__printk_ratelimit do_set_cpus_allowedcpuset_cpus_allowed_fallback

plist_del

scan_swap_map_try_ssd_cluster inc_cluster_info_page

compact_zone

_mix_pool_bytes

kill_fasync

crng_reseed

swap_do_scheduled_discard

__free_cluster

migrate_prep_local isolate_migratepages_block drain_local_pages

putback_movable_pages __reset_isolation_suitable

__pageblock_pfn_to_pagemigrate_pages

release_freepages

__update_load_avg_blocked_se.isra.33

vmalloc_to_page

complete

sort

add_to_avail_list

send_sigio

extract_entropy.constprop.43 memzero_explicit

invalidate_batched_entropy

module_put _extract_crng _crng_backtrack_protect

compact_unlock_should_abort.isra.35

compact_trylock_irqsave.isra.20

update_pageblock_skipisolate_movable_page

putback_active_hugepage PageMovableputback_movable_page set_pfnblock_flags_mask

size_to_hstate

move_to_new_page

__put_anon_vma

page_get_anon_vma

remove_migration_ptes

xfer_secondary_poolaccount.constprop.44

_raw_write_lock_irqsave

chacha20_block

plist_add

__wake_up_locked

queued_write_lock_slowpath

decay_load

send_sigio_to_task

migrate_page

down_write

up_write

security_file_send_sigiotask

migrate_page_move_mapping migrate_page_copymigrate_page_states

call_rwsem_down_write_failed

rwsem_down_write_failed

radix_tree_replace_slot__dec_node_state __inc_node_statebuffer_migrate_lock_buffers __dec_zone_state __inc_zone_state copy_page

__lock_buffer

osq_lock rwsem_spin_on_owner osq_unlock__rwsem_mark_wake.part.2

wake_q_add

Our Solution

●Ignore kernel features that are orthogonal to correct module functionality.

●Implement “kernel gateway functions” that provide simple services, such as memory allocation.

●Provide a generic, simple API that developers can use to exercise their module functionality.

●Generate a user space application that is as fast and scalable as the underlying fuzzer.

mutex_lock

fat_dir_empty

fat_get_short_entry

__brelse

mutex_unlockmsdos_find

msdos_format_namefat_scan fat_remove_entries

current_time

mark_buffer_dirty_inode sync_dirty_buffer

fat_sync_inode__fat_remove_entries

fat_msg

__mark_inode_dirty

drop_nlink clear_nlink fat_detach

_raw_spin_lock

fat_flush_inodes

writeback_inode

msdos_rmdir

fat__get_entry

printk

__bread_gfp ___ratelimit fat_bmap __find_get_block __breadahead

strchrstrncmp