Greetings, 0day kernel testing robot got the below dmesg and the first bad commit is https://github.com/0day-ci/linux/commits/Uladzislau-Rezki-Sony/rcu-tree-s... commit 883a2cefc0684ec945a069e1091407f1f07a0558 Author: Uladzislau Rezki (Sony) <urezki(a)gmail.com&gt; AuthorDate: Tue Dec 31 13:22:41 2019 +0100 Commit: 0day robot <lkp(a)intel.com&gt; CommitDate: Fri Jan 3 11:52:33 2020 +0800 rcu/tree: support kfree_bulk() interface in kfree_rcu() kfree_rcu() logic can be improved further by using kfree_bulk() interface along with "basic batching support" introduced earlier. The are at least two advantages of using "bulk" interface: - in case of large number of kfree_rcu() requests kfree_bulk() reduces the per-object overhead caused by calling kfree() per-object. - reduces the number of cache-misses due to "pointer chasing" between objects which can be far spread between each other. This approach defines a new kfree_rcu_bulk_data structure that stores pointers in an array with a specific size. Number of entries in that array depends on PAGE_SIZE making kfree_rcu_bulk_data structure to be exactly one page. Since it deals with "block-chain" technique there is an extra need in dynamic allocation when a new block is required. Memory is allocated with GFP_NOWAIT | __GFP_NOWARN flags, i.e. that allows to skip direct reclaim under low memory condition to prevent stalling and fails silently under high memory pressure. The "emergency path" gets maintained when a system is run out of memory. In that case objects are linked into regular list and that is it. In order to evaluate it, the "rcuperf" was run to analyze how much memory is consumed and what is kfree_bulk() throughput. Testing on the HiKey-960, arm64, 8xCPUs with below parameters: CONFIG_SLAB=y kfree_loops=200000 kfree_alloc_num=1000 kfree_rcu_test=1 102898760401 ns, loops: 200000, batches: 5822, memory footprint: 158MB 89947009882 ns, loops: 200000, batches: 6715, memory footprint: 115MB rcuperf shows approximately ~12% better throughput(Total time) in case of using "bulk" interface. The "drain logic" or its RCU callback does the work faster that leads to better throughput. Signed-off-by: Uladzislau Rezki (Sony) <urezki(a)gmail.com&gt; 4892cf6a83 doc: Add some more RCU list patterns in the kernel 883a2cefc0 rcu/tree: support kfree_bulk() interface in kfree_rcu() +---------------------------------------------------+------------+------------+ | | 4892cf6a83 | 883a2cefc0 | +---------------------------------------------------+------------+------------+ | boot_successes | 30 | 0 | | boot_failures | 0 | 10 | | WARNING:at_lib/debugobjects.c:#debug_print_object | 0 | 10 | | RIP:debug_print_object | 0 | 10 | | INFO:rcu_preempt_detected_stalls_on_CPUs/tasks | 0 | 2 | | RIP:ftrace_likely_update | 0 | 2 | | BUG:soft_lockup-CPU##stuck_for#s![ksoftirqd:#] | 0 | 3 | | RIP:_raw_spin_unlock_irqrestore | 0 | 3 | | Kernel_panic-not_syncing:softlockup:hung_tasks | 0 | 3 | | BUG:kernel_timeout_in_test_stage | 0 | 7 | | WARNING:at_kernel/rcu/tree.c:#call_rcu | 0 | 4 | | RIP:call_rcu | 0 | 4 | +---------------------------------------------------+------------+------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <lkp(a)intel.com&gt; [child0:632] kexec_load (283:[32BIT]) returned ENOSYS, marking as inactive. [child0:669] pkey_alloc (381:[32BIT]) returned ENOSYS, marking as inactive. [child0:669] acct (51:[32BIT]) returned ENOSYS, marking as inactive. [ 13.957168] ------------[ cut here ]------------ [ 13.958256] ODEBUG: free active (active state 1) object type: rcu_head hint: 0x0 [ 13.962148] WARNING: CPU: 0 PID: 212 at lib/debugobjects.c:484 debug_print_object+0x95/0xd0 [ 13.964298] Modules linked in: [ 13.964960] CPU: 0 PID: 212 Comm: kworker/0:2 Not tainted 5.5.0-rc1-00136-g883a2cefc0684 #1 [ 13.966712] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014 [ 13.968528] Workqueue: events kfree_rcu_work [ 13.969466] RIP: 0010:debug_print_object+0x95/0xd0 [ 13.970480] Code: d2 e8 2f 06 d6 ff 8b 43 10 4d 89 f1 4c 89 e6 8b 4b 14 48 c7 c7 88 73 be 82 4d 8b 45 00 48 8b 14 c5 a0 5f 6d 82 e8 7b 65 c6 ff <0f> 0b b9 01 00 00 00 31 d2 be 01 00 00 00 48 c7 c7 98 b8 0c 83 e8 [ 13.974435] RSP: 0000:ffff888231677bf8 EFLAGS: 00010282 [ 13.975531] RAX: 0000000000000000 RBX: ffff88822d4200e0 RCX: 0000000000000000 [ 13.976730] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff8306e028 [ 13.977568] RBP: ffff888231677c18 R08: 0000000000000000 R09: ffff888231670790 [ 13.978412] R10: ffff888231670000 R11: 0000000000000003 R12: ffffffff82bc5299 [ 13.979250] R13: ffffffff82e77360 R14: 0000000000000000 R15: dead000000000100 [ 13.980089] FS: 0000000000000000(0000) GS:ffffffff82e4f000(0000) knlGS:0000000000000000 [ 13.981069] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 13.981746] CR2: 00007f1e913fc77c CR3: 0000000225ce9000 CR4: 00000000000006f0 [ 13.982587] Call Trace: [ 13.982911] __debug_check_no_obj_freed+0x19a/0x200 [ 13.983494] debug_check_no_obj_freed+0x14/0x20 [ 13.984036] free_pcp_prepare+0xee/0x1d0 [ 13.984541] free_unref_page+0x1b/0x80 [ 13.984994] __free_pages+0x19/0x20 [ 13.985503] __free_pages+0x13/0x20 [ 13.985924] slob_free_pages+0x7d/0x90 [ 13.986373] slob_free+0x34f/0x530 [ 13.986784] kfree+0x154/0x210 [ 13.987155] __kmem_cache_free_bulk+0x44/0x60 [ 13.987673] kmem_cache_free_bulk+0xe/0x10 [ 13.988163] kfree_rcu_work+0x95/0x310 [ 13.989010] ? kfree_rcu_work+0x64/0x310 [ 13.989884] process_one_work+0x378/0x7c0 [ 13.990770] worker_thread+0x40/0x600 [ 13.991587] kthread+0x14e/0x170 [ 13.992344] ? process_one_work+0x7c0/0x7c0 [ 13.993256] ? kthread_create_on_node+0x70/0x70 [ 13.994246] ret_from_fork+0x3a/0x50 [ 13.995039] ---[ end trace cdf242638b0e32a0 ]--- [child0:632] trace_fd was -1 # HH:MM RESULT GOOD BAD GOOD_BUT_DIRTY DIRTY_NOT_BAD git bisect start 1407715fd1b13c01ffeeec8b3837c2e1b4cbb025 e42617b825f8073569da76dc4510bfa019b1c35a -- git bisect bad 044c40eba826e8d47d53edfb2c5fe866d0f36134 # 12:47 B 0 1 17 0 Merge 'linux-review/Uladzislau-Rezki-Sony/rcu-tree-support-kfree_bulk-interface-in-kfree_rcu/20200103-115231' into devel-catchup-202001031304 git bisect good 01975b69f841f967f889a32ef34425cdb33c3d84 # 13:40 G 10 0 0 0 0day base guard for 'devel-catchup-202001031304' git bisect good b081888d4e95ae70b3e3098fc26a536d3e3da6ad # 16:03 G 11 0 1 1 Merge branches 'doc.2019.12.10a', 'exp.2019.12.09a', 'fixes.2019.12.12a', 'kfree_rcu.2019.12.09a', 'list.2019.12.09a', 'preempt.2019.12.09a' and 'torture.2019.12.09a' into HEAD git bisect good 5887751baba7ce39d88b154d7c1a4d53f4608509 # 16:56 G 11 0 0 0 tools/memory-model: Use "-unroll 0" to keep --hw runs finite git bisect good 59a1e11f586e5635f35beaf9f89e4badd93c3087 # 17:26 G 10 0 0 0 torture: Make results-directory date format completion-friendly git bisect good 96c4fe7d5735476c2e972b367af4b1a5bf831899 # 17:50 G 10 0 0 0 rcu: Fix spelling mistake "leval" -> "level" git bisect good de73daec335f0227b7303abb0814a3c00c0c55b2 # 18:16 G 10 0 0 0 rcutorture: Make kvm-find-errors.sh abort on bad directory git bisect good 8bf389d441538030d07a9a0f9e38ec0843f7a83e # 18:39 G 11 0 0 0 rcuperf: Measure memory footprint during kfree_rcu() test git bisect bad 883a2cefc0684ec945a069e1091407f1f07a0558 # 19:10 B 0 2 18 0 rcu/tree: support kfree_bulk() interface in kfree_rcu() git bisect good 4892cf6a83bc59897fc77a270b4098f152e14079 # 19:50 G 10 0 0 0 doc: Add some more RCU list patterns in the kernel # first bad commit: [883a2cefc0684ec945a069e1091407f1f07a0558] rcu/tree: support kfree_bulk() interface in kfree_rcu() git bisect good 4892cf6a83bc59897fc77a270b4098f152e14079 # 20:14 G 30 0 0 0 doc: Add some more RCU list patterns in the kernel # extra tests with debug options git bisect good 883a2cefc0684ec945a069e1091407f1f07a0558 # 21:31 G 10 0 10 10 rcu/tree: support kfree_bulk() interface in kfree_rcu() # extra tests on head commit of linux-review/Uladzislau-Rezki-Sony/rcu-tree-support-kfree_bulk-interface-in-kfree_rcu/20200103-115231 git bisect bad 883a2cefc0684ec945a069e1091407f1f07a0558 # 21:38 B 0 10 26 0 rcu/tree: support kfree_bulk() interface in kfree_rcu() # bad: [883a2cefc0684ec945a069e1091407f1f07a0558] rcu/tree: support kfree_bulk() interface in kfree_rcu() # extra tests on revert first bad commit git bisect good 8af7d370bd521685c4a599ceb820299360478fed # 22:43 G 11 0 0 0 Revert "rcu/tree: support kfree_bulk() interface in kfree_rcu()" # good: [8af7d370bd521685c4a599ceb820299360478fed] Revert "rcu/tree: support kfree_bulk() interface in kfree_rcu()" --- 0-DAY kernel test infrastructure Open Source Technology Center https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation