FYI, we noticed the following commit:
git://bee.sh.intel.com/git/yhuang/linux.git percpug
commit bf2477a85ce3bb257080ec4b6b7ed19aeda544d9 ("mm, page_alloc: Use per CPU group
page set")
in testcase: trinity
with following parameters:
runtime: 300s
Trinity is a linux system call fuzz tester.
on test machine: qemu-system-x86_64 -enable-kvm -cpu IvyBridge -m 360M
caused below changes:
+-------------------------------------------------------------------+------------+------------+
| | 2ae275ab8b |
bf2477a85c |
+-------------------------------------------------------------------+------------+------------+
| boot_successes | 686 | 658
|
| boot_failures | 1118 | 1145
|
| BUG:kernel_reboot-without-warning_in_test_stage | 2 | 5
|
| invoked_oom-killer:gfp_mask=0x | 1110 | 1122
|
| Mem-Info | 1114 | 1123
|
| Out_of_memory:Kill_process | 160 | 153
|
| warn_alloc_failed+0x | 8 | 5
|
| calltrace:compat_SyS_ioctl | 5 | 2
|
| Kernel_panic-not_syncing:Out_of_memory_and_no_killable_processes | 5 | 2
|
| Kernel_panic-not_syncing:Attempted_to_kill_init!exitcode= | 2 | 2
|
| inconsistent{IN-HARDIRQ-W}->{HARDIRQ-ON-W}usage | 0 | 11
|
| calltrace:compat_SyS_pwritev | 0 | 7
|
| BUG:sleeping_function_called_from_invalid_context_at_mm/filemap.c | 0 | 18
|
| INFO:lockdep_is_turned_off | 0 | 18
|
| INFO:possible_irq_lock_inversion_dependency_detected | 0 | 8
|
| calltrace:compat_SyS_getdents64 | 0 | 3
|
| calltrace:kswapd | 0 | 2
|
| calltrace:SyS_exit_group | 0 | 7
|
| calltrace:compat_SyS_writev | 0 | 5
|
| calltrace:compat_SyS_pwritev2 | 0 | 3
|
| calltrace:compat_SyS_readv | 0 | 1
|
| calltrace:SyS_clone | 0 | 2
|
| calltrace:compat_SyS_open | 0 | 1
|
+-------------------------------------------------------------------+------------+------------+
[ 67.829848] trinity-c3 (427) used greatest stack depth: 27520 bytes left
[ 71.008609]
[ 71.009352] =========================================================
[ 71.010551] [ INFO: possible irq lock inversion dependency detected ]
[ 71.011741] 4.8.0-rc5-00007-gbf2477a #1 Not tainted
[ 71.012777] ---------------------------------------------------------
[ 71.013966] trinity-c3/1466 just changed the state of lock:
[ 71.015054] (&(&p->lock)->rlock){+.....}, at: [<ffffffff8125097e>]
buffered_rmqueue+0xce/0xc20
[ 71.017233] but this lock was taken by another, HARDIRQ-safe lock in the past:
[ 71.018891] (&(&sighand->siglock)->rlock){-.....}
and interrupts could create inverse lock ordering between them.
[ 71.021989]
[ 71.021989] other info that might help us debug this:
[ 71.023637] Possible interrupt unsafe locking scenario:
[ 71.023637]
[ 71.025251] CPU0 CPU1
[ 71.026252] ---- ----
[ 71.027225] lock(&(&p->lock)->rlock);
[ 71.028274] local_irq_disable();
[ 71.029415]
lock(&(&sighand->siglock)->rlock);
[ 71.030829] lock(&(&p->lock)->rlock);
[ 71.032161] <Interrupt>
[ 71.032939] lock(&(&sighand->siglock)->rlock);
[ 71.034093]
[ 71.034093] *** DEADLOCK ***
[ 71.034093]
[ 71.036073] 3 locks held by trinity-c3/1466:
[ 71.037020] #0: (sb_writers#5){.+.+.+}, at: [<ffffffff812fc31b>]
__sb_start_write+0x14b/0x160
[ 71.039347] #1: (&sb->s_type->i_mutex_key#11){++++++}, at:
[<ffffffff8123cae2>] generic_file_write_iter+0x42/0x280
[ 71.041846] #2: (&(&p->lock)->rlock){+.....}, at:
[<ffffffff8125097e>] buffered_rmqueue+0xce/0xc20
[ 71.044134]
[ 71.044134] the shortest dependencies between 2nd lock and 1st lock:
[ 71.045961] -> (&(&sighand->siglock)->rlock){-.....} ops: 42345 {
[ 71.047507] IN-HARDIRQ-W at:
[ 71.048418] [<ffffffff8112e8f4>] mark_irqflags+0xc4/0x2a0
[ 71.050209] [<ffffffff81130d66>]
__lock_acquire+0x426/0x850
[ 71.051981] [<ffffffff8113157f>] lock_acquire+0x3ef/0x440
[ 71.053770] [<ffffffff81e696f8>] _raw_spin_lock+0x58/0xd0
[ 71.055537] [<ffffffff810cea2b>]
__lock_task_sighand+0x21b/0x390
[ 71.057388] [<ffffffff810cebe2>]
do_send_sig_info+0x42/0xb0
[ 71.059184] [<ffffffff810cf57f>]
group_send_sig_info+0x1cf/0x1f0
[ 71.061009] [<ffffffff810cf6be>] kill_pid_info+0x11e/0x210
[ 71.062813] [<ffffffff8116cc38>] it_real_fn+0x328/0x350
[ 71.064165] [<ffffffff8116b83e>] __run_hrtimer+0x72e/0xb20
[ 71.065953] [<ffffffff8116bd3d>]
hrtimer_run_queues+0x10d/0x160
[ 71.067777] [<ffffffff811644ed>]
run_local_timers+0x2d/0xb0
[ 71.069605] [<ffffffff811645a2>]
update_process_times+0x32/0x90
[ 71.071431] [<ffffffff81185c51>]
tick_nohz_handler+0xf1/0x150
[ 71.073246] [<ffffffff810685ac>]
local_apic_timer_interrupt+0x6c/0x80
[ 71.075131] [<ffffffff81e6d38c>]
smp_apic_timer_interrupt+0x7c/0xa0
[ 71.077018] [<ffffffff81e6c431>]
apic_timer_interrupt+0x91/0xa0
[ 71.078837] [<ffffffff8103c030>] default_idle+0x1f0/0x3d0
[ 71.080635] [<ffffffff8103b352>] arch_cpu_idle+0x12/0x20
[ 71.082383] [<ffffffff81122924>]
default_idle_call+0x84/0xa0
[ 71.084193] [<ffffffff81122a80>]
cpuidle_idle_call+0xb0/0x230
[ 71.086005] [<ffffffff8112328c>] cpu_idle_loop+0x68c/0x730
[ 71.087778] [<ffffffff81123364>]
cpu_startup_entry+0x34/0x50
[ 71.089595] [<ffffffff81e5e9d6>] rest_init+0x216/0x230
[ 71.090932] [<ffffffff829a0970>] start_kernel+0x6db/0x6e5
[ 71.092740] [<ffffffff8299f2c7>]
x86_64_start_reservations+0x4a/0x54
[ 71.094606] [<ffffffff8299f737>]
x86_64_start_kernel+0x1bd/0x1d4
[ 71.096467] INITIAL USE at:
[ 71.097369] [<ffffffff81130d90>] __lock_acquire+0x450/0x850
[ 71.099158] [<ffffffff8113157f>] lock_acquire+0x3ef/0x440
[ 71.100909] [<ffffffff81e69a08>]
_raw_spin_lock_irqsave+0x88/0x110
[ 71.102777] [<ffffffff810c8765>] flush_signals+0x25/0x90
[ 71.104124] [<ffffffff810c8813>] ignore_signals+0x43/0x50
[ 71.105891] [<ffffffff810ecbea>] kthreadd+0x4a/0x240
[ 71.107202] [<ffffffff81e6aecf>] ret_from_fork+0x1f/0x40
[ 71.108544] }
[ 71.109232] ... key at: [<ffffffff82b73050>] __key.76640+0x0/0x8
[ 71.110520] ... acquired at:
[ 71.111328] [<ffffffff8113080c>] validate_chain+0xa2c/0xb60
[ 71.112545] [<ffffffff8113109e>] __lock_acquire+0x75e/0x850
[ 71.113743] [<ffffffff8113157f>] lock_acquire+0x3ef/0x440
[ 71.114919] [<ffffffff81e696f8>] _raw_spin_lock+0x58/0xd0
[ 71.116152] [<ffffffff8125097e>] buffered_rmqueue+0xce/0xc20
[ 71.117361] [<ffffffff81251764>] get_page_from_freelist+0x294/0x390
[ 71.118632] [<ffffffff81253026>] __alloc_pages_nodemask+0x1e6/0x590
[ 71.119999] [<ffffffff812c7311>] alloc_pages_current+0x171/0x1a0
[ 71.121248] [<ffffffff812cf7a6>] allocate_slab+0xc6/0x6d0
[ 71.122442] [<ffffffff812cfe24>] new_slab+0x74/0x90
[ 71.123563] [<ffffffff812d5f42>] ___slab_alloc+0x402/0x650
[ 71.124750] [<ffffffff812d6207>] __slab_alloc+0x77/0xf0
[ 71.125927] [<ffffffff812d7555>] kmem_cache_alloc+0x135/0x680
[ 71.127147] [<ffffffff810cb05c>] __sigqueue_alloc+0x27c/0x360
[ 71.128361] [<ffffffff810cb2f2>] T.1599+0x162/0xa70
[ 71.129510] [<ffffffff810cbc16>] __group_send_sig_info+0x16/0x20
[ 71.130756] [<ffffffff810ce7ba>] do_notify_parent+0x88a/0x8e0
[ 71.131973] [<ffffffff810b7155>] exit_notify+0x155/0x2f0
[ 71.133168] [<ffffffff810b7b75>] do_exit+0x885/0xbe0
[ 71.134294] [<ffffffff810b801e>] do_group_exit+0x14e/0x150
[ 71.135478] [<ffffffff810b803a>] SyS_exit_group+0x1a/0x30
[ 71.136670] [<ffffffff8100464c>] do_fast_syscall_32+0x29c/0x5a0
[ 71.137975] [<ffffffff81e6c2b1>] entry_SYSENTER_compat+0x51/0x60
[ 71.139288]
[ 71.139927] -> (&(&p->lock)->rlock){+.....} ops: 116196 {
[ 71.141384] HARDIRQ-ON-W at:
[ 71.142279] [<ffffffff8112eb7c>] mark_held_locks+0xac/0x100
[ 71.144063] [<ffffffff8112f116>]
trace_hardirqs_on_caller+0x246/0x300
[ 71.145938] [<ffffffff8112f1ed>] trace_hardirqs_on+0x1d/0x30
[ 71.147707] [<ffffffff81251486>]
buffered_rmqueue+0xbd6/0xc20
[ 71.149512] [<ffffffff81251764>]
get_page_from_freelist+0x294/0x390
[ 71.151347] [<ffffffff81253026>]
__alloc_pages_nodemask+0x1e6/0x590
[ 71.153202] [<ffffffff812cb05f>] alloc_pages_vma+0x25f/0x2a0
[ 71.165309] [<ffffffff812721ef>] shmem_alloc_page+0x6f/0xd0
[ 71.167096] [<ffffffff81272340>]
shmem_alloc_and_acct_page+0xf0/0x2c0
[ 71.168956] [<ffffffff81278bdc>]
shmem_getpage_gfp+0x74c/0x1010
[ 71.170781] [<ffffffff81279562>] shmem_getpage+0x32/0x40
[ 71.172110] [<ffffffff812795da>] shmem_write_begin+0x6a/0x80
[ 71.173907] [<ffffffff81239b50>]
generic_perform_write+0x110/0x2a0
[ 71.175734] [<ffffffff8123bdb0>]
__generic_file_write_iter+0x210/0x2b0
[ 71.177652] [<ffffffff8123cc7a>]
generic_file_write_iter+0x1da/0x280
[ 71.179529] [<ffffffff812f89de>]
do_iter_readv_writev+0x13e/0x180
[ 71.181358] [<ffffffff812f8e5c>]
compat_do_readv_writev+0x11c/0x310
[ 71.183219] [<ffffffff812f9121>]
do_compat_pwritev64+0xd1/0x140
[ 71.185011] [<ffffffff812f9587>]
compat_SyS_pwritev2+0x47/0x60
[ 71.186844] [<ffffffff8100464c>]
do_fast_syscall_32+0x29c/0x5a0
[ 71.188643] [<ffffffff81e6c2b1>]
entry_SYSENTER_compat+0x51/0x60
[ 71.190477] INITIAL USE at:
[ 71.191367] [<ffffffff81130d90>] __lock_acquire+0x450/0x850
[ 71.193123] [<ffffffff8113157f>] lock_acquire+0x3ef/0x440
[ 71.194468] [<ffffffff81e696f8>] _raw_spin_lock+0x58/0xd0
[ 71.195831] [<ffffffff8124e0b5>]
free_hot_cold_page+0x7c5/0x910
[ 71.197621] [<ffffffff8124f280>] __free_pages+0xf0/0x130
[ 71.198941] [<ffffffff829d388a>]
__free_pages_boot_core+0x176/0x1b5
[ 71.200799] [<ffffffff829d38e4>]
__free_pages_bootmem+0x1b/0x25
[ 71.202611] [<ffffffff829d92cf>] __free_memory_core+0xa9/0xda
[ 71.204388] [<ffffffff829d93e5>]
free_low_memory_core_early+0xe5/0x1ae
[ 71.206272] [<ffffffff829d95c1>] free_all_bootmem+0x1e/0x2f
[ 71.208020] [<ffffffff829beccf>] mem_init+0x80/0xa5
[ 71.209362] [<ffffffff829a0500>] start_kernel+0x26b/0x6e5
[ 71.210704] [<ffffffff8299f2c7>]
x86_64_start_reservations+0x4a/0x54
[ 71.212566] [<ffffffff8299f737>]
x86_64_start_kernel+0x1bd/0x1d4
[ 71.214367] }
[ 71.215019] ... key at: [<ffffffff83920814>] __key.60973+0x0/0x8
[ 71.216321] ... acquired at:
[ 71.217114] [<ffffffff8112d3d0>] check_usage_backwards+0x150/0x170
[ 71.218383] [<ffffffff8112e2cd>] mark_lock_irq+0x17d/0x410
[ 71.219594] [<ffffffff8112e72b>] mark_lock+0x1cb/0x2d0
[ 71.220746] [<ffffffff8112eb7c>] mark_held_locks+0xac/0x100
[ 71.221939] [<ffffffff8112f116>] trace_hardirqs_on_caller+0x246/0x300
[ 71.223268] [<ffffffff8112f1ed>] trace_hardirqs_on+0x1d/0x30
[ 71.224480] [<ffffffff81251486>] buffered_rmqueue+0xbd6/0xc20
[ 71.225690] [<ffffffff81251764>] get_page_from_freelist+0x294/0x390
[ 71.226992] [<ffffffff81253026>] __alloc_pages_nodemask+0x1e6/0x590
[ 71.228263] [<ffffffff812cb05f>] alloc_pages_vma+0x25f/0x2a0
[ 71.229495] [<ffffffff812721ef>] shmem_alloc_page+0x6f/0xd0
[ 71.230695] [<ffffffff81272340>] shmem_alloc_and_acct_page+0xf0/0x2c0
[ 71.231988] [<ffffffff81278bdc>] shmem_getpage_gfp+0x74c/0x1010
[ 71.233276] [<ffffffff81279562>] shmem_getpage+0x32/0x40
[ 71.234447] [<ffffffff812795da>] shmem_write_begin+0x6a/0x80
[ 71.235653] [<ffffffff81239b50>] generic_perform_write+0x110/0x2a0
[ 71.236951] [<ffffffff8123bdb0>] __generic_file_write_iter+0x210/0x2b0
[ 71.238252] [<ffffffff8123cc7a>] generic_file_write_iter+0x1da/0x280
[ 71.239561] [<ffffffff812f89de>] do_iter_readv_writev+0x13e/0x180
[ 71.240822] [<ffffffff812f8e5c>] compat_do_readv_writev+0x11c/0x310
[ 71.242097] [<ffffffff812f9121>] do_compat_pwritev64+0xd1/0x140
[ 71.243378] [<ffffffff812f9587>] compat_SyS_pwritev2+0x47/0x60
[ 71.244605] [<ffffffff8100464c>] do_fast_syscall_32+0x29c/0x5a0
[ 71.245872] [<ffffffff81e6c2b1>] entry_SYSENTER_compat+0x51/0x60
[ 71.247121]
[ 71.247760]
[ 71.247760] stack backtrace:
[ 71.249127] CPU: 0 PID: 1466 Comm: trinity-c3 Not tainted 4.8.0-rc5-00007-gbf2477a #1
[ 71.250871] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Debian-1.8.2-1
04/01/2014
[ 71.252748] 0000000000000000 ffff8800029d73a8 ffffffff81982620 0000000000000007
[ 71.254752] 0000000000000006 ffffffff82dd5f40 ffffffff82dd5f40 ffff8800029d73f8
[ 71.256804] ffffffff8112ae40 ffffffff82da7340 ffffffff82317b3f 0000000000015560
[ 71.258808] Call Trace:
[ 71.259574] [<ffffffff81982620>] dump_stack+0xfe/0x18e
[ 71.260629] [<ffffffff8112ae40>] print_irq_inversion_bug+0x320/0x340
[ 71.261825] [<ffffffff8112d280>] ? check_usage_forwards+0x180/0x180
[ 71.263028] [<ffffffff8112d3d0>] check_usage_backwards+0x150/0x170
[ 71.264201] [<ffffffff8112e2cd>] mark_lock_irq+0x17d/0x410
[ 71.265295] [<ffffffff8112e72b>] mark_lock+0x1cb/0x2d0
[ 71.266367] [<ffffffff8112eb7c>] mark_held_locks+0xac/0x100
[ 71.267466] [<ffffffff81251486>] ? buffered_rmqueue+0xbd6/0xc20
[ 71.268604] [<ffffffff8112f116>] trace_hardirqs_on_caller+0x246/0x300
[ 71.269827] [<ffffffff8112f1ed>] trace_hardirqs_on+0x1d/0x30
[ 71.270941] [<ffffffff81251486>] buffered_rmqueue+0xbd6/0xc20
[ 71.272057] [<ffffffff812d7736>] ? kmem_cache_alloc+0x316/0x680
[ 71.273227] [<ffffffff8198b441>] ? radix_tree_node_alloc+0x51/0xf0
[ 71.274400] [<ffffffff81279500>] ? shmem_read_mapping_page_gfp+0x60/0x90
To reproduce:
git clone
git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp run job.yaml
Thanks,
Xiaolong