FYI, we noticed the following commit: git://bee.sh.intel.com/git/yhuang/linux.git percpug commit bf2477a85ce3bb257080ec4b6b7ed19aeda544d9 ("mm, page_alloc: Use per CPU group page set") in testcase: trinity with following parameters: runtime: 300s Trinity is a linux system call fuzz tester. on test machine: qemu-system-x86_64 -enable-kvm -cpu IvyBridge -m 360M caused below changes: +-------------------------------------------------------------------+------------+------------+ | | 2ae275ab8b | bf2477a85c | +-------------------------------------------------------------------+------------+------------+ | boot_successes | 686 | 658 | | boot_failures | 1118 | 1145 | | BUG:kernel_reboot-without-warning_in_test_stage | 2 | 5 | | invoked_oom-killer:gfp_mask=0x | 1110 | 1122 | | Mem-Info | 1114 | 1123 | | Out_of_memory:Kill_process | 160 | 153 | | warn_alloc_failed+0x | 8 | 5 | | calltrace:compat_SyS_ioctl | 5 | 2 | | Kernel_panic-not_syncing:Out_of_memory_and_no_killable_processes | 5 | 2 | | Kernel_panic-not_syncing:Attempted_to_kill_init!exitcode= | 2 | 2 | | inconsistent{IN-HARDIRQ-W}->{HARDIRQ-ON-W}usage | 0 | 11 | | calltrace:compat_SyS_pwritev | 0 | 7 | | BUG:sleeping_function_called_from_invalid_context_at_mm/filemap.c | 0 | 18 | | INFO:lockdep_is_turned_off | 0 | 18 | | INFO:possible_irq_lock_inversion_dependency_detected | 0 | 8 | | calltrace:compat_SyS_getdents64 | 0 | 3 | | calltrace:kswapd | 0 | 2 | | calltrace:SyS_exit_group | 0 | 7 | | calltrace:compat_SyS_writev | 0 | 5 | | calltrace:compat_SyS_pwritev2 | 0 | 3 | | calltrace:compat_SyS_readv | 0 | 1 | | calltrace:SyS_clone | 0 | 2 | | calltrace:compat_SyS_open | 0 | 1 | +-------------------------------------------------------------------+------------+------------+ [ 67.829848] trinity-c3 (427) used greatest stack depth: 27520 bytes left [ 71.008609] [ 71.009352] ========================================================= [ 71.010551] [ INFO: possible irq lock inversion dependency detected ] [ 71.011741] 4.8.0-rc5-00007-gbf2477a #1 Not tainted [ 71.012777] --------------------------------------------------------- [ 71.013966] trinity-c3/1466 just changed the state of lock: [ 71.015054] (&(&p->lock)->rlock){+.....}, at: [<ffffffff8125097e>] buffered_rmqueue+0xce/0xc20 [ 71.017233] but this lock was taken by another, HARDIRQ-safe lock in the past: [ 71.018891] (&(&sighand->siglock)->rlock){-.....} and interrupts could create inverse lock ordering between them. [ 71.021989] [ 71.021989] other info that might help us debug this: [ 71.023637] Possible interrupt unsafe locking scenario: [ 71.023637] [ 71.025251] CPU0 CPU1 [ 71.026252] ---- ---- [ 71.027225] lock(&(&p->lock)->rlock); [ 71.028274] local_irq_disable(); [ 71.029415] lock(&(&sighand->siglock)->rlock); [ 71.030829] lock(&(&p->lock)->rlock); [ 71.032161] <Interrupt> [ 71.032939] lock(&(&sighand->siglock)->rlock); [ 71.034093] [ 71.034093] *** DEADLOCK *** [ 71.034093] [ 71.036073] 3 locks held by trinity-c3/1466: [ 71.037020] #0: (sb_writers#5){.+.+.+}, at: [<ffffffff812fc31b>] __sb_start_write+0x14b/0x160 [ 71.039347] #1: (&sb->s_type->i_mutex_key#11){++++++}, at: [<ffffffff8123cae2>] generic_file_write_iter+0x42/0x280 [ 71.041846] #2: (&(&p->lock)->rlock){+.....}, at: [<ffffffff8125097e>] buffered_rmqueue+0xce/0xc20 [ 71.044134] [ 71.044134] the shortest dependencies between 2nd lock and 1st lock: [ 71.045961] -> (&(&sighand->siglock)->rlock){-.....} ops: 42345 { [ 71.047507] IN-HARDIRQ-W at: [ 71.048418] [<ffffffff8112e8f4>] mark_irqflags+0xc4/0x2a0 [ 71.050209] [<ffffffff81130d66>] __lock_acquire+0x426/0x850 [ 71.051981] [<ffffffff8113157f>] lock_acquire+0x3ef/0x440 [ 71.053770] [<ffffffff81e696f8>] _raw_spin_lock+0x58/0xd0 [ 71.055537] [<ffffffff810cea2b>] __lock_task_sighand+0x21b/0x390 [ 71.057388] [<ffffffff810cebe2>] do_send_sig_info+0x42/0xb0 [ 71.059184] [<ffffffff810cf57f>] group_send_sig_info+0x1cf/0x1f0 [ 71.061009] [<ffffffff810cf6be>] kill_pid_info+0x11e/0x210 [ 71.062813] [<ffffffff8116cc38>] it_real_fn+0x328/0x350 [ 71.064165] [<ffffffff8116b83e>] __run_hrtimer+0x72e/0xb20 [ 71.065953] [<ffffffff8116bd3d>] hrtimer_run_queues+0x10d/0x160 [ 71.067777] [<ffffffff811644ed>] run_local_timers+0x2d/0xb0 [ 71.069605] [<ffffffff811645a2>] update_process_times+0x32/0x90 [ 71.071431] [<ffffffff81185c51>] tick_nohz_handler+0xf1/0x150 [ 71.073246] [<ffffffff810685ac>] local_apic_timer_interrupt+0x6c/0x80 [ 71.075131] [<ffffffff81e6d38c>] smp_apic_timer_interrupt+0x7c/0xa0 [ 71.077018] [<ffffffff81e6c431>] apic_timer_interrupt+0x91/0xa0 [ 71.078837] [<ffffffff8103c030>] default_idle+0x1f0/0x3d0 [ 71.080635] [<ffffffff8103b352>] arch_cpu_idle+0x12/0x20 [ 71.082383] [<ffffffff81122924>] default_idle_call+0x84/0xa0 [ 71.084193] [<ffffffff81122a80>] cpuidle_idle_call+0xb0/0x230 [ 71.086005] [<ffffffff8112328c>] cpu_idle_loop+0x68c/0x730 [ 71.087778] [<ffffffff81123364>] cpu_startup_entry+0x34/0x50 [ 71.089595] [<ffffffff81e5e9d6>] rest_init+0x216/0x230 [ 71.090932] [<ffffffff829a0970>] start_kernel+0x6db/0x6e5 [ 71.092740] [<ffffffff8299f2c7>] x86_64_start_reservations+0x4a/0x54 [ 71.094606] [<ffffffff8299f737>] x86_64_start_kernel+0x1bd/0x1d4 [ 71.096467] INITIAL USE at: [ 71.097369] [<ffffffff81130d90>] __lock_acquire+0x450/0x850 [ 71.099158] [<ffffffff8113157f>] lock_acquire+0x3ef/0x440 [ 71.100909] [<ffffffff81e69a08>] _raw_spin_lock_irqsave+0x88/0x110 [ 71.102777] [<ffffffff810c8765>] flush_signals+0x25/0x90 [ 71.104124] [<ffffffff810c8813>] ignore_signals+0x43/0x50 [ 71.105891] [<ffffffff810ecbea>] kthreadd+0x4a/0x240 [ 71.107202] [<ffffffff81e6aecf>] ret_from_fork+0x1f/0x40 [ 71.108544] } [ 71.109232] ... key at: [<ffffffff82b73050>] __key.76640+0x0/0x8 [ 71.110520] ... acquired at: [ 71.111328] [<ffffffff8113080c>] validate_chain+0xa2c/0xb60 [ 71.112545] [<ffffffff8113109e>] __lock_acquire+0x75e/0x850 [ 71.113743] [<ffffffff8113157f>] lock_acquire+0x3ef/0x440 [ 71.114919] [<ffffffff81e696f8>] _raw_spin_lock+0x58/0xd0 [ 71.116152] [<ffffffff8125097e>] buffered_rmqueue+0xce/0xc20 [ 71.117361] [<ffffffff81251764>] get_page_from_freelist+0x294/0x390 [ 71.118632] [<ffffffff81253026>] __alloc_pages_nodemask+0x1e6/0x590 [ 71.119999] [<ffffffff812c7311>] alloc_pages_current+0x171/0x1a0 [ 71.121248] [<ffffffff812cf7a6>] allocate_slab+0xc6/0x6d0 [ 71.122442] [<ffffffff812cfe24>] new_slab+0x74/0x90 [ 71.123563] [<ffffffff812d5f42>] ___slab_alloc+0x402/0x650 [ 71.124750] [<ffffffff812d6207>] __slab_alloc+0x77/0xf0 [ 71.125927] [<ffffffff812d7555>] kmem_cache_alloc+0x135/0x680 [ 71.127147] [<ffffffff810cb05c>] __sigqueue_alloc+0x27c/0x360 [ 71.128361] [<ffffffff810cb2f2>] T.1599+0x162/0xa70 [ 71.129510] [<ffffffff810cbc16>] __group_send_sig_info+0x16/0x20 [ 71.130756] [<ffffffff810ce7ba>] do_notify_parent+0x88a/0x8e0 [ 71.131973] [<ffffffff810b7155>] exit_notify+0x155/0x2f0 [ 71.133168] [<ffffffff810b7b75>] do_exit+0x885/0xbe0 [ 71.134294] [<ffffffff810b801e>] do_group_exit+0x14e/0x150 [ 71.135478] [<ffffffff810b803a>] SyS_exit_group+0x1a/0x30 [ 71.136670] [<ffffffff8100464c>] do_fast_syscall_32+0x29c/0x5a0 [ 71.137975] [<ffffffff81e6c2b1>] entry_SYSENTER_compat+0x51/0x60 [ 71.139288] [ 71.139927] -> (&(&p->lock)->rlock){+.....} ops: 116196 { [ 71.141384] HARDIRQ-ON-W at: [ 71.142279] [<ffffffff8112eb7c>] mark_held_locks+0xac/0x100 [ 71.144063] [<ffffffff8112f116>] trace_hardirqs_on_caller+0x246/0x300 [ 71.145938] [<ffffffff8112f1ed>] trace_hardirqs_on+0x1d/0x30 [ 71.147707] [<ffffffff81251486>] buffered_rmqueue+0xbd6/0xc20 [ 71.149512] [<ffffffff81251764>] get_page_from_freelist+0x294/0x390 [ 71.151347] [<ffffffff81253026>] __alloc_pages_nodemask+0x1e6/0x590 [ 71.153202] [<ffffffff812cb05f>] alloc_pages_vma+0x25f/0x2a0 [ 71.165309] [<ffffffff812721ef>] shmem_alloc_page+0x6f/0xd0 [ 71.167096] [<ffffffff81272340>] shmem_alloc_and_acct_page+0xf0/0x2c0 [ 71.168956] [<ffffffff81278bdc>] shmem_getpage_gfp+0x74c/0x1010 [ 71.170781] [<ffffffff81279562>] shmem_getpage+0x32/0x40 [ 71.172110] [<ffffffff812795da>] shmem_write_begin+0x6a/0x80 [ 71.173907] [<ffffffff81239b50>] generic_perform_write+0x110/0x2a0 [ 71.175734] [<ffffffff8123bdb0>] __generic_file_write_iter+0x210/0x2b0 [ 71.177652] [<ffffffff8123cc7a>] generic_file_write_iter+0x1da/0x280 [ 71.179529] [<ffffffff812f89de>] do_iter_readv_writev+0x13e/0x180 [ 71.181358] [<ffffffff812f8e5c>] compat_do_readv_writev+0x11c/0x310 [ 71.183219] [<ffffffff812f9121>] do_compat_pwritev64+0xd1/0x140 [ 71.185011] [<ffffffff812f9587>] compat_SyS_pwritev2+0x47/0x60 [ 71.186844] [<ffffffff8100464c>] do_fast_syscall_32+0x29c/0x5a0 [ 71.188643] [<ffffffff81e6c2b1>] entry_SYSENTER_compat+0x51/0x60 [ 71.190477] INITIAL USE at: [ 71.191367] [<ffffffff81130d90>] __lock_acquire+0x450/0x850 [ 71.193123] [<ffffffff8113157f>] lock_acquire+0x3ef/0x440 [ 71.194468] [<ffffffff81e696f8>] _raw_spin_lock+0x58/0xd0 [ 71.195831] [<ffffffff8124e0b5>] free_hot_cold_page+0x7c5/0x910 [ 71.197621] [<ffffffff8124f280>] __free_pages+0xf0/0x130 [ 71.198941] [<ffffffff829d388a>] __free_pages_boot_core+0x176/0x1b5 [ 71.200799] [<ffffffff829d38e4>] __free_pages_bootmem+0x1b/0x25 [ 71.202611] [<ffffffff829d92cf>] __free_memory_core+0xa9/0xda [ 71.204388] [<ffffffff829d93e5>] free_low_memory_core_early+0xe5/0x1ae [ 71.206272] [<ffffffff829d95c1>] free_all_bootmem+0x1e/0x2f [ 71.208020] [<ffffffff829beccf>] mem_init+0x80/0xa5 [ 71.209362] [<ffffffff829a0500>] start_kernel+0x26b/0x6e5 [ 71.210704] [<ffffffff8299f2c7>] x86_64_start_reservations+0x4a/0x54 [ 71.212566] [<ffffffff8299f737>] x86_64_start_kernel+0x1bd/0x1d4 [ 71.214367] } [ 71.215019] ... key at: [<ffffffff83920814>] __key.60973+0x0/0x8 [ 71.216321] ... acquired at: [ 71.217114] [<ffffffff8112d3d0>] check_usage_backwards+0x150/0x170 [ 71.218383] [<ffffffff8112e2cd>] mark_lock_irq+0x17d/0x410 [ 71.219594] [<ffffffff8112e72b>] mark_lock+0x1cb/0x2d0 [ 71.220746] [<ffffffff8112eb7c>] mark_held_locks+0xac/0x100 [ 71.221939] [<ffffffff8112f116>] trace_hardirqs_on_caller+0x246/0x300 [ 71.223268] [<ffffffff8112f1ed>] trace_hardirqs_on+0x1d/0x30 [ 71.224480] [<ffffffff81251486>] buffered_rmqueue+0xbd6/0xc20 [ 71.225690] [<ffffffff81251764>] get_page_from_freelist+0x294/0x390 [ 71.226992] [<ffffffff81253026>] __alloc_pages_nodemask+0x1e6/0x590 [ 71.228263] [<ffffffff812cb05f>] alloc_pages_vma+0x25f/0x2a0 [ 71.229495] [<ffffffff812721ef>] shmem_alloc_page+0x6f/0xd0 [ 71.230695] [<ffffffff81272340>] shmem_alloc_and_acct_page+0xf0/0x2c0 [ 71.231988] [<ffffffff81278bdc>] shmem_getpage_gfp+0x74c/0x1010 [ 71.233276] [<ffffffff81279562>] shmem_getpage+0x32/0x40 [ 71.234447] [<ffffffff812795da>] shmem_write_begin+0x6a/0x80 [ 71.235653] [<ffffffff81239b50>] generic_perform_write+0x110/0x2a0 [ 71.236951] [<ffffffff8123bdb0>] __generic_file_write_iter+0x210/0x2b0 [ 71.238252] [<ffffffff8123cc7a>] generic_file_write_iter+0x1da/0x280 [ 71.239561] [<ffffffff812f89de>] do_iter_readv_writev+0x13e/0x180 [ 71.240822] [<ffffffff812f8e5c>] compat_do_readv_writev+0x11c/0x310 [ 71.242097] [<ffffffff812f9121>] do_compat_pwritev64+0xd1/0x140 [ 71.243378] [<ffffffff812f9587>] compat_SyS_pwritev2+0x47/0x60 [ 71.244605] [<ffffffff8100464c>] do_fast_syscall_32+0x29c/0x5a0 [ 71.245872] [<ffffffff81e6c2b1>] entry_SYSENTER_compat+0x51/0x60 [ 71.247121] [ 71.247760] [ 71.247760] stack backtrace: [ 71.249127] CPU: 0 PID: 1466 Comm: trinity-c3 Not tainted 4.8.0-rc5-00007-gbf2477a #1 [ 71.250871] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Debian-1.8.2-1 04/01/2014 [ 71.252748] 0000000000000000 ffff8800029d73a8 ffffffff81982620 0000000000000007 [ 71.254752] 0000000000000006 ffffffff82dd5f40 ffffffff82dd5f40 ffff8800029d73f8 [ 71.256804] ffffffff8112ae40 ffffffff82da7340 ffffffff82317b3f 0000000000015560 [ 71.258808] Call Trace: [ 71.259574] [<ffffffff81982620>] dump_stack+0xfe/0x18e [ 71.260629] [<ffffffff8112ae40>] print_irq_inversion_bug+0x320/0x340 [ 71.261825] [<ffffffff8112d280>] ? check_usage_forwards+0x180/0x180 [ 71.263028] [<ffffffff8112d3d0>] check_usage_backwards+0x150/0x170 [ 71.264201] [<ffffffff8112e2cd>] mark_lock_irq+0x17d/0x410 [ 71.265295] [<ffffffff8112e72b>] mark_lock+0x1cb/0x2d0 [ 71.266367] [<ffffffff8112eb7c>] mark_held_locks+0xac/0x100 [ 71.267466] [<ffffffff81251486>] ? buffered_rmqueue+0xbd6/0xc20 [ 71.268604] [<ffffffff8112f116>] trace_hardirqs_on_caller+0x246/0x300 [ 71.269827] [<ffffffff8112f1ed>] trace_hardirqs_on+0x1d/0x30 [ 71.270941] [<ffffffff81251486>] buffered_rmqueue+0xbd6/0xc20 [ 71.272057] [<ffffffff812d7736>] ? kmem_cache_alloc+0x316/0x680 [ 71.273227] [<ffffffff8198b441>] ? radix_tree_node_alloc+0x51/0xf0 [ 71.274400] [<ffffffff81279500>] ? shmem_read_mapping_page_gfp+0x60/0x90 To reproduce: git clone git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml Thanks, Xiaolong