36b74766f2 ("mm: security: introduce init_on_alloc=1 and .."): general protection fault: 0000 [#1] PREEMPT SMP PTI
11:18 a.m.
Greetings,
0day kernel testing robot got the below dmesg and the first bad commit is
https://git.kernel.org/pub/scm/linux/kernel/git/kees/linux.git kspp/meminit/heap-v2
commit 36b74766f295e0898345865a0511a443cf90aa13
Author: Alexander Potapenko <glider(a)google.com>
AuthorDate: Tue May 14 16:35:34 2019 +0200
Commit: Kees Cook <keescook(a)chromium.org>
CommitDate: Wed May 15 15:23:43 2019 -0700
mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
The new options are needed to prevent possible information leaks and
make control-flow bugs that depend on uninitialized values more
deterministic.
init_on_alloc=1 makes the kernel initialize newly allocated pages and heap
objects with zeroes. Initialization is done at allocation time at the
places where checks for __GFP_ZERO are performed.
init_on_free=1 makes the kernel initialize freed pages and heap objects
with zeroes upon their deletion. This helps to ensure sensitive data
doesn't leak via use-after-free accesses.
Both init_on_alloc=1 and init_on_free=1 guarantee that the allocator
returns zeroed memory. The only exception is slab caches with
constructors. Those are never zero-initialized to preserve their semantics.
For SLOB allocator init_on_free=1 also implies init_on_alloc=1 behavior,
i.e. objects are zeroed at both allocation and deallocation time.
This is done because SLOB may otherwise return multiple freelist pointers
in the allocated object. For SLAB and SLUB enabling either init_on_alloc
or init_on_free leads to one-time initialization of the object.
Both init_on_alloc and init_on_free default to zero, but those defaults
can be overridden with CONFIG_INIT_ON_ALLOC_DEFAULT_ON and
CONFIG_INIT_ON_FREE_DEFAULT_ON.
Slowdown for the new features compared to init_on_free=0,
init_on_alloc=0:
hackbench, init_on_free=1: +7.62% sys time (st.err 0.74%)
hackbench, init_on_alloc=1: +7.75% sys time (st.err 2.14%)
Linux build with -j12, init_on_free=1: +8.38% wall time (st.err 0.39%)
Linux build with -j12, init_on_free=1: +24.42% sys time (st.err 0.52%)
Linux build with -j12, init_on_alloc=1: -0.13% wall time (st.err 0.42%)
Linux build with -j12, init_on_alloc=1: +0.57% sys time (st.err 0.40%)
The slowdown for init_on_free=0, init_on_alloc=0 compared to the
baseline is within the standard error.
Signed-off-by: Alexander Potapenko <glider(a)google.com>
To: Andrew Morton <akpm(a)linux-foundation.org>
To: Christoph Lameter <cl(a)linux.com>
To: Kees Cook <keescook(a)chromium.org>
Cc: Masahiro Yamada <yamada.masahiro(a)socionext.com>
Cc: James Morris <jmorris(a)namei.org>
Cc: "Serge E. Hallyn" <serge(a)hallyn.com>
Cc: Nick Desaulniers <ndesaulniers(a)google.com>
Cc: Kostya Serebryany <kcc(a)google.com>
Cc: Dmitry Vyukov <dvyukov(a)google.com>
Cc: Sandeep Patil <sspatil(a)android.com>
Cc: Laura Abbott <labbott(a)redhat.com>
Cc: Randy Dunlap <rdunlap(a)infradead.org>
Cc: Jann Horn <jannh(a)google.com>
Cc: Mark Rutland <mark.rutland(a)arm.com>
Cc: linux-mm(a)kvack.org
Cc: linux-security-module(a)vger.kernel.org
Cc: kernel-hardening(a)lists.openwall.com
Signed-off-by: Kees Cook <keescook(a)chromium.org>
2bbacd1a92 Merge tag 'kconfig-v5.2' of
git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild
36b74766f2 mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
83e3ae9e99 mm: Introduce SLAB_NO_FREE_INIT and mark excluded caches
+---------------------------------------------+------------+------------+------------+
| | 2bbacd1a92 | 36b74766f2 | 83e3ae9e99 |
+---------------------------------------------+------------+------------+------------+
| boot_successes | 33 | 0 | 0 |
| boot_failures | 0 | 15 | 11 |
| BUG:kernel_NULL_pointer_dereference,address | 0 | 4 | |
| Oops:#[##] | 0 | 4 | |
| RIP:kernel_init_free_pages | 0 | 4 | |
| Kernel_panic-not_syncing:Fatal_exception | 0 | 15 | 11 |
| general_protection_fault:#[##] | 0 | 11 | 11 |
| RIP:clear_page_orig | 0 | 11 | 11 |
+---------------------------------------------+------------+------------+------------+
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <lkp(a)intel.com>
[ 0.088943] Built 1 zonelists, mobility grouping on. Total pages: 129129
[ 0.089917] Kernel command line: root=/dev/ram0 hung_task_panic=1 debug apic=debug
sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 net.ifnames=0 printk.devkmsg=on
panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0
drbd.minor_count=8 systemd.log_level=err ignore_loglevel console=tty0
earlyprintk=ttyS0,115200 console=ttyS0,115200 vga=normal rw
link=/cephfs/kbuild/run-queue/yocto-vm-yocto/x86_64-randconfig-s3-05171126/linux-devel:devel-hourly-2019051709:36b74766f295e0898345865a0511a443cf90aa13:bisect-linux-36/.vmlinuz-36b74766f295e0898345865a0511a443cf90aa13-20190519080629-7:yocto-vm-yocto-107
branch=linux-devel/devel-hourly-2019051709
BOOT_IMAGE=/pkg/linux/x86_64-randconfig-s3-05171126/gcc-7/36b74766f295e0898345865a0511a443cf90aa13/vmlinuz-5.1.0-10910-g36b7476
drbd.minor_count=8 rcuperf.shutdown=0
[ 0.100683] sysrq: sysrq always enabled.
[ 0.102667] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
[ 0.103751] Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
[ 0.124335] general protection fault: 0000 [#1] PREEMPT SMP PTI
[ 0.125142] CPU: 0 PID: 0 Comm: swapper Not tainted 5.1.0-10910-g36b7476 #1
[ 0.126049] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1
04/01/2014
[ 0.127161] RIP: 0010:clear_page_orig+0x12/0x40
[ 0.127763] Code: 90 90 90 90 90 90 90 90 b9 00 02 00 00 31 c0 f3 48 ab c3 0f 1f 44 00
00 31 c0 b9 40 00 00 00 66 0f 1f 84 00 00 00 00 00 ff c9 <48> 89 07 48 89 47 08 48
89 47 10 48 89 47 18 48 89 47 20 48 89 47
[ 0.130272] RSP: 0000:ffffffff83403dc0 EFLAGS: 00010016
[ 0.130953] RAX: 0000000000000000 RBX: ffff88801f2c4000 RCX: 000000000000003f
[ 0.131876] RDX: ffff88801fb85000 RSI: 0000000000000000 RDI: b6daf636cb800000
[ 0.132834] RBP: ffffffff83403df0 R08: 0000000000000000 R09: 0000000000000000
[ 0.133800] R10: ffffffff83427800 R11: 0000000000000000 R12: 0000000000000400
[ 0.134760] R13: 6db6db6db6db6db7 R14: ffff88801f2d2000 R15: 0000000000010400
[ 0.135721] FS: 0000000000000000(0000) GS:ffff88801ee00000(0000)
knlGS:0000000000000000
[ 0.136814] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 0.137622] CR2: ffff888005401000 CR3: 0000000003420001 CR4: 00000000000606b0
[ 0.138598] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 0.139561] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 0.140523] Call Trace:
[ 0.140868] ? kernel_init_free_pages+0xa3/0x170
[ 0.141504] memblock_free_pages+0x87/0xda
[ 0.142066] memblock_free_all+0x2b9/0x372
[ 0.142642] mem_init+0x30/0xc6
[ 0.143078] start_kernel+0x361/0xc99
[ 0.143591] x86_64_start_reservations+0x55/0x76
[ 0.144223] x86_64_start_kernel+0x7e/0xa1
[ 0.144776] secondary_startup_64+0xa4/0xb0
[ 0.145328] Modules linked in:
[ 0.145759] random: get_random_bytes called from init_oops_id+0x39/0x70 with
crng_init=0
[ 0.146827] ---[ end trace 25feee6b5690789c ]---
[ 0.147480] RIP: 0010:clear_page_orig+0x12/0x40
# HH:MM RESULT GOOD BAD
GOOD_BUT_DIRTY DIRTY_NOT_BAD
git bisect start c6a087131e35cda9ec9cc98e348ebc7136b798f4
e93c9c99a629c61837d5a7fc2120cd2b6c70dbdd --
git bisect bad b49a6b15d52eb982290433d73e0274a152ef101d # 04:00 B 0 11 26 0
Merge 'linux-de/upstream/rem_rq_load' into devel-hourly-2019051709
git bisect bad d0dd933bf0bd0aff98240ee09eb7f3d05b721fff # 04:11 B 0 11 26 0
Merge
'linux-review/Thomas-Pedersen/mac80211-mesh-fix-RCU-warning/20190517-081950' into
devel-hourly-2019051709
git bisect good e5066b7e86afb919312ecdfc1fd897028d1b08fe # 04:33 G 10 0 0 0
Merge
'linux-review/Yannick-Fertr/drm-stm-ltdc-remove-clk_round_rate-comment/20190514-034940'
into devel-hourly-2019051709
git bisect bad 166c2c2157380ec4423b077f90070db53d43d29a # 04:58 B 0 11 26 0
Merge
'linux-review/Serge-Semin/usb-cp210x-Add-cp2108-GPIOs-support/20190514-193210'
into devel-hourly-2019051709
git bisect good 381f62cd950e5d795a3377400160a9c089b5d9d0 # 05:20 G 10 0 0 0
Merge 'khilman-linux-amlogic/v5.2/fixes' into devel-hourly-2019051709
git bisect bad c1efdb0b692eebfe39f30758055f5e6453833033 # 05:34 B 0 9 24 0
Merge
'linux-review/George-G-Davis/net-sysctl-cleanup-net_sysctl_init-error-exit-paths/20190517-035042'
into devel-hourly-2019051709
git bisect bad 1ccab7297016467f16ee8967e9229bf9a810053b # 06:04 B 0 10 25 0
Merge 'kees/kspp/meminit/heap-v2' into devel-hourly-2019051709
git bisect good 7085c7ff97531f647adb2a71de3c3fb97d5c6653 # 06:52 G 11 0 0 0
Merge
'linux-review/Aneesh-Kumar-K-V/mm-nvdimm-Fix-kernel-crash-on-devm_mremap_pages_release/20190515-010312'
into devel-hourly-2019051709
git bisect bad 96a09f754689f7839134ebc186cc9d0dc0d79a67 # 07:09 B 0 6 21 0
gfp: mm: introduce __GFP_NO_AUTOINIT
git bisect bad 708d942c1103d0811d5ca754ab4f256692f44b3a # 07:32 B 0 11 26 0
lib: introduce test_meminit module
git bisect bad 36b74766f295e0898345865a0511a443cf90aa13 # 08:07 B 0 6 21 0
mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
# first bad commit: [36b74766f295e0898345865a0511a443cf90aa13] mm: security: introduce
init_on_alloc=1 and init_on_free=1 boot options
git bisect good 2bbacd1a92788ee334c7e92b765ea16ebab68dfe # 08:30 G 32 0 0 0
Merge tag 'kconfig-v5.2' of
git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild
# extra tests with debug options
git bisect bad 36b74766f295e0898345865a0511a443cf90aa13 # 08:43 B 0 11 26 0
mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
# extra tests on HEAD of linux-devel/devel-hourly-2019051709
git bisect bad c6a087131e35cda9ec9cc98e348ebc7136b798f4 # 08:44 B 0 13 31 0
0day head guard for 'devel-hourly-2019051709'
# extra tests on tree/branch kees/kspp/meminit/heap-v2
git bisect bad 83e3ae9e9987893af618ac396b027ef236efdd92 # 09:17 B 0 11 26 0
mm: Introduce SLAB_NO_FREE_INIT and mark excluded caches
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/lkp Intel Corporation
11:47 p.m.
On Sun, May 19, 2019 at 3:19 AM kernel test robot <lkp(a)intel.com> wrote:
Greetings,
0day kernel testing robot got the below dmesg and the first bad commit is
https://git.kernel.org/pub/scm/linux/kernel/git/kees/linux.git kspp/meminit/heap-v2
commit 36b74766f295e0898345865a0511a443cf90aa13
Author: Alexander Potapenko <glider(a)google.com>
AuthorDate: Tue May 14 16:35:34 2019 +0200
Commit: Kees Cook <keescook(a)chromium.org>
CommitDate: Wed May 15 15:23:43 2019 -0700
mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
The new options are needed to prevent possible information leaks and
make control-flow bugs that depend on uninitialized values more
deterministic.
init_on_alloc=1 makes the kernel initialize newly allocated pages and heap
objects with zeroes. Initialization is done at allocation time at the
places where checks for __GFP_ZERO are performed.
init_on_free=1 makes the kernel initialize freed pages and heap objects
with zeroes upon their deletion. This helps to ensure sensitive data
doesn't leak via use-after-free accesses.
Both init_on_alloc=1 and init_on_free=1 guarantee that the allocator
returns zeroed memory. The only exception is slab caches with
constructors. Those are never zero-initialized to preserve their semantics.
For SLOB allocator init_on_free=1 also implies init_on_alloc=1 behavior,
i.e. objects are zeroed at both allocation and deallocation time.
This is done because SLOB may otherwise return multiple freelist pointers
in the allocated object. For SLAB and SLUB enabling either init_on_alloc
or init_on_free leads to one-time initialization of the object.
Both init_on_alloc and init_on_free default to zero, but those defaults
can be overridden with CONFIG_INIT_ON_ALLOC_DEFAULT_ON and
CONFIG_INIT_ON_FREE_DEFAULT_ON.
Slowdown for the new features compared to init_on_free=0,
init_on_alloc=0:
hackbench, init_on_free=1: +7.62% sys time (st.err 0.74%)
hackbench, init_on_alloc=1: +7.75% sys time (st.err 2.14%)
Linux build with -j12, init_on_free=1: +8.38% wall time (st.err 0.39%)
Linux build with -j12, init_on_free=1: +24.42% sys time (st.err 0.52%)
Linux build with -j12, init_on_alloc=1: -0.13% wall time (st.err 0.42%)
Linux build with -j12, init_on_alloc=1: +0.57% sys time (st.err 0.40%)
The slowdown for init_on_free=0, init_on_alloc=0 compared to the
baseline is within the standard error.
Signed-off-by: Alexander Potapenko <glider(a)google.com>
To: Andrew Morton <akpm(a)linux-foundation.org>
To: Christoph Lameter <cl(a)linux.com>
To: Kees Cook <keescook(a)chromium.org>
Cc: Masahiro Yamada <yamada.masahiro(a)socionext.com>
Cc: James Morris <jmorris(a)namei.org>
Cc: "Serge E. Hallyn" <serge(a)hallyn.com>
Cc: Nick Desaulniers <ndesaulniers(a)google.com>
Cc: Kostya Serebryany <kcc(a)google.com>
Cc: Dmitry Vyukov <dvyukov(a)google.com>
Cc: Sandeep Patil <sspatil(a)android.com>
Cc: Laura Abbott <labbott(a)redhat.com>
Cc: Randy Dunlap <rdunlap(a)infradead.org>
Cc: Jann Horn <jannh(a)google.com>
Cc: Mark Rutland <mark.rutland(a)arm.com>
Cc: linux-mm(a)kvack.org
Cc: linux-security-module(a)vger.kernel.org
Cc: kernel-hardening(a)lists.openwall.com
Signed-off-by: Kees Cook <keescook(a)chromium.org>
2bbacd1a92 Merge tag 'kconfig-v5.2' of
git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild
36b74766f2 mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
83e3ae9e99 mm: Introduce SLAB_NO_FREE_INIT and mark excluded caches
+---------------------------------------------+------------+------------+------------+
| | 2bbacd1a92 | 36b74766f2 | 83e3ae9e99 |
+---------------------------------------------+------------+------------+------------+
| boot_successes | 33 | 0 | 0 |
| boot_failures | 0 | 15 | 11 |
| BUG:kernel_NULL_pointer_dereference,address | 0 | 4 | |
| Oops:#[##] | 0 | 4 | |
| RIP:kernel_init_free_pages | 0 | 4 | |
| Kernel_panic-not_syncing:Fatal_exception | 0 | 15 | 11 |
| general_protection_fault:#[##] | 0 | 11 | 11 |
| RIP:clear_page_orig | 0 | 11 | 11 |
+---------------------------------------------+------------+------------+------------+
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <lkp(a)intel.com>
[ 0.088943] Built 1 zonelists, mobility grouping on. Total pages: 129129
[ 0.089917] Kernel command line: root=/dev/ram0 hung_task_panic=1 debug apic=debug
sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 net.ifnames=0 printk.devkmsg=on
panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0
drbd.minor_count=8 systemd.log_level=err ignore_loglevel console=tty0
earlyprintk=ttyS0,115200 console=ttyS0,115200 vga=normal rw
link=/cephfs/kbuild/run-queue/yocto-vm-yocto/x86_64-randconfig-s3-05171126/linux-devel:devel-hourly-2019051709:36b74766f295e0898345865a0511a443cf90aa13:bisect-linux-36/.vmlinuz-36b74766f295e0898345865a0511a443cf90aa13-20190519080629-7:yocto-vm-yocto-107
branch=linux-devel/devel-hourly-2019051709
BOOT_IMAGE=/pkg/linux/x86_64-randconfig-s3-05171126/gcc-7/36b74766f295e0898345865a0511a443cf90aa13/vmlinuz-5.1.0-10910-g36b7476
drbd.minor_count=8 rcuperf.shutdown=0
[ 0.100683] sysrq: sysrq always enabled.
[ 0.102667] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
[ 0.103751] Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
[ 0.124335] general protection fault: 0000 [#1] PREEMPT SMP PTI
[ 0.125142] CPU: 0 PID: 0 Comm: swapper Not tainted 5.1.0-10910-g36b7476 #1
[ 0.126049] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1
04/01/2014
[ 0.127161] RIP: 0010:clear_page_orig+0x12/0x40
[ 0.127763] Code: 90 90 90 90 90 90 90 90 b9 00 02 00 00 31 c0 f3 48 ab c3 0f 1f 44 00
00 31 c0 b9 40 00 00 00 66 0f 1f 84 00 00 00 00 00 ff c9 <48> 89 07 48 89 47 08 48
89 47 10 48 89 47 18 48 89 47 20 48 89 47
[ 0.130272] RSP: 0000:ffffffff83403dc0 EFLAGS: 00010016
[ 0.130953] RAX: 0000000000000000 RBX: ffff88801f2c4000 RCX: 000000000000003f
[ 0.131876] RDX: ffff88801fb85000 RSI: 0000000000000000 RDI: b6daf636cb800000
[ 0.132834] RBP: ffffffff83403df0 R08: 0000000000000000 R09: 0000000000000000
[ 0.133800] R10: ffffffff83427800 R11: 0000000000000000 R12: 0000000000000400
[ 0.134760] R13: 6db6db6db6db6db7 R14: ffff88801f2d2000 R15: 0000000000010400
[ 0.135721] FS: 0000000000000000(0000) GS:ffff88801ee00000(0000)
knlGS:0000000000000000
[ 0.136814] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 0.137622] CR2: ffff888005401000 CR3: 0000000003420001 CR4: 00000000000606b0
[ 0.138598] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 0.139561] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 0.140523] Call Trace:
[ 0.140868] ? kernel_init_free_pages+0xa3/0x170
[ 0.141504] memblock_free_pages+0x87/0xda
[ 0.142066] memblock_free_all+0x2b9/0x372
[ 0.142642] mem_init+0x30/0xc6
[ 0.143078] start_kernel+0x361/0xc99
[ 0.143591] x86_64_start_reservations+0x55/0x76
[ 0.144223] x86_64_start_kernel+0x7e/0xa1
[ 0.144776] secondary_startup_64+0xa4/0xb0
[ 0.145328] Modules linked in:
[ 0.145759] random: get_random_bytes called from init_oops_id+0x39/0x70 with
crng_init=0
[ 0.146827] ---[ end trace 25feee6b5690789c ]---
[ 0.147480] RIP: 0010:clear_page_orig+0x12/0x40
Turns out we can't call
kernel_init_free_pages() from memblock_free_pages()
# HH:MM
RESULT GOOD BAD GOOD_BUT_DIRTY DIRTY_NOT_BAD
git bisect start c6a087131e35cda9ec9cc98e348ebc7136b798f4
e93c9c99a629c61837d5a7fc2120cd2b6c70dbdd --
git bisect bad b49a6b15d52eb982290433d73e0274a152ef101d # 04:00 B 0 11 26
0 Merge 'linux-de/upstream/rem_rq_load' into devel-hourly-2019051709
git bisect bad d0dd933bf0bd0aff98240ee09eb7f3d05b721fff # 04:11 B 0 11 26
0 Merge
'linux-review/Thomas-Pedersen/mac80211-mesh-fix-RCU-warning/20190517-081950' into
devel-hourly-2019051709
git bisect good e5066b7e86afb919312ecdfc1fd897028d1b08fe # 04:33 G 10 0 0
0 Merge
'linux-review/Yannick-Fertr/drm-stm-ltdc-remove-clk_round_rate-comment/20190514-034940'
into devel-hourly-2019051709
git bisect bad 166c2c2157380ec4423b077f90070db53d43d29a # 04:58 B 0 11 26
0 Merge
'linux-review/Serge-Semin/usb-cp210x-Add-cp2108-GPIOs-support/20190514-193210'
into devel-hourly-2019051709
git bisect good 381f62cd950e5d795a3377400160a9c089b5d9d0 # 05:20 G 10 0 0
0 Merge 'khilman-linux-amlogic/v5.2/fixes' into devel-hourly-2019051709
git bisect bad c1efdb0b692eebfe39f30758055f5e6453833033 # 05:34 B 0 9 24
0 Merge
'linux-review/George-G-Davis/net-sysctl-cleanup-net_sysctl_init-error-exit-paths/20190517-035042'
into devel-hourly-2019051709
git bisect bad 1ccab7297016467f16ee8967e9229bf9a810053b # 06:04 B 0 10 25
0 Merge 'kees/kspp/meminit/heap-v2' into devel-hourly-2019051709
git bisect good 7085c7ff97531f647adb2a71de3c3fb97d5c6653 # 06:52 G 11 0 0
0 Merge
'linux-review/Aneesh-Kumar-K-V/mm-nvdimm-Fix-kernel-crash-on-devm_mremap_pages_release/20190515-010312'
into devel-hourly-2019051709
git bisect bad 96a09f754689f7839134ebc186cc9d0dc0d79a67 # 07:09 B 0 6 21
0 gfp: mm: introduce __GFP_NO_AUTOINIT
git bisect bad 708d942c1103d0811d5ca754ab4f256692f44b3a # 07:32 B 0 11 26
0 lib: introduce test_meminit module
git bisect bad 36b74766f295e0898345865a0511a443cf90aa13 # 08:07 B 0 6 21
0 mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
# first bad commit: [36b74766f295e0898345865a0511a443cf90aa13] mm: security: introduce
init_on_alloc=1 and init_on_free=1 boot options
git bisect good 2bbacd1a92788ee334c7e92b765ea16ebab68dfe # 08:30 G 32 0 0
0 Merge tag 'kconfig-v5.2' of
git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild
# extra tests with debug options
git bisect bad 36b74766f295e0898345865a0511a443cf90aa13 # 08:43 B 0 11 26
0 mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
# extra tests on HEAD of linux-devel/devel-hourly-2019051709
git bisect bad c6a087131e35cda9ec9cc98e348ebc7136b798f4 # 08:44 B 0 13 31
0 0day head guard for 'devel-hourly-2019051709'
# extra tests on tree/branch kees/kspp/meminit/heap-v2
git bisect bad 83e3ae9e9987893af618ac396b027ef236efdd92 # 09:17 B 0 11 26
0 mm: Introduce SLAB_NO_FREE_INIT and mark excluded caches
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/lkp Intel Corporation
--
Alexander Potapenko
Software Engineer
Google Germany GmbH
Erika-Mann-Straße, 33
80636 München
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
6:21 a.m.
On Tue, May 21, 2019 at 03:47:16PM +0200, Alexander Potapenko wrote:
On Sun, May 19, 2019 at 3:19 AM kernel test robot
<lkp(a)intel.com> wrote:
>
> Greetings,
>
> 0day kernel testing robot got the below dmesg and the first bad commit is
>
> https://git.kernel.org/pub/scm/linux/kernel/git/kees/linux.git kspp/meminit/heap-v2
>
> commit 36b74766f295e0898345865a0511a443cf90aa13
> Author: Alexander Potapenko <glider(a)google.com>
> AuthorDate: Tue May 14 16:35:34 2019 +0200
> Commit: Kees Cook <keescook(a)chromium.org>
> CommitDate: Wed May 15 15:23:43 2019 -0700
>
> mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
>
> The new options are needed to prevent possible information leaks and
> make control-flow bugs that depend on uninitialized values more
> deterministic.
>
> init_on_alloc=1 makes the kernel initialize newly allocated pages and heap
> objects with zeroes. Initialization is done at allocation time at the
> places where checks for __GFP_ZERO are performed.
>
> init_on_free=1 makes the kernel initialize freed pages and heap objects
> with zeroes upon their deletion. This helps to ensure sensitive data
> doesn't leak via use-after-free accesses.
>
> Both init_on_alloc=1 and init_on_free=1 guarantee that the allocator
> returns zeroed memory. The only exception is slab caches with
> constructors. Those are never zero-initialized to preserve their semantics.
>
> For SLOB allocator init_on_free=1 also implies init_on_alloc=1 behavior,
> i.e. objects are zeroed at both allocation and deallocation time.
> This is done because SLOB may otherwise return multiple freelist pointers
> in the allocated object. For SLAB and SLUB enabling either init_on_alloc
> or init_on_free leads to one-time initialization of the object.
>
> Both init_on_alloc and init_on_free default to zero, but those defaults
> can be overridden with CONFIG_INIT_ON_ALLOC_DEFAULT_ON and
> CONFIG_INIT_ON_FREE_DEFAULT_ON.
>
> Slowdown for the new features compared to init_on_free=0,
> init_on_alloc=0:
>
> hackbench, init_on_free=1: +7.62% sys time (st.err 0.74%)
> hackbench, init_on_alloc=1: +7.75% sys time (st.err 2.14%)
>
> Linux build with -j12, init_on_free=1: +8.38% wall time (st.err 0.39%)
> Linux build with -j12, init_on_free=1: +24.42% sys time (st.err 0.52%)
> Linux build with -j12, init_on_alloc=1: -0.13% wall time (st.err 0.42%)
> Linux build with -j12, init_on_alloc=1: +0.57% sys time (st.err 0.40%)
>
> The slowdown for init_on_free=0, init_on_alloc=0 compared to the
> baseline is within the standard error.
>
> Signed-off-by: Alexander Potapenko <glider(a)google.com>
> To: Andrew Morton <akpm(a)linux-foundation.org>
> To: Christoph Lameter <cl(a)linux.com>
> To: Kees Cook <keescook(a)chromium.org>
> Cc: Masahiro Yamada <yamada.masahiro(a)socionext.com>
> Cc: James Morris <jmorris(a)namei.org>
> Cc: "Serge E. Hallyn" <serge(a)hallyn.com>
> Cc: Nick Desaulniers <ndesaulniers(a)google.com>
> Cc: Kostya Serebryany <kcc(a)google.com>
> Cc: Dmitry Vyukov <dvyukov(a)google.com>
> Cc: Sandeep Patil <sspatil(a)android.com>
> Cc: Laura Abbott <labbott(a)redhat.com>
> Cc: Randy Dunlap <rdunlap(a)infradead.org>
> Cc: Jann Horn <jannh(a)google.com>
> Cc: Mark Rutland <mark.rutland(a)arm.com>
> Cc: linux-mm(a)kvack.org
> Cc: linux-security-module(a)vger.kernel.org
> Cc: kernel-hardening(a)lists.openwall.com
> Signed-off-by: Kees Cook <keescook(a)chromium.org>
>
> 2bbacd1a92 Merge tag 'kconfig-v5.2' of
git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild
> 36b74766f2 mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
> 83e3ae9e99 mm: Introduce SLAB_NO_FREE_INIT and mark excluded caches
>
+---------------------------------------------+------------+------------+------------+
> | | 2bbacd1a92 | 36b74766f2 | 83e3ae9e99
|
>
+---------------------------------------------+------------+------------+------------+
> | boot_successes | 33 | 0 | 0
|
> | boot_failures | 0 | 15 | 11
|
> | BUG:kernel_NULL_pointer_dereference,address | 0 | 4 |
|
> | Oops:#[##] | 0 | 4 |
|
> | RIP:kernel_init_free_pages | 0 | 4 |
|
> | Kernel_panic-not_syncing:Fatal_exception | 0 | 15 | 11
|
> | general_protection_fault:#[##] | 0 | 11 | 11
|
> | RIP:clear_page_orig | 0 | 11 | 11
|
>
+---------------------------------------------+------------+------------+------------+
>
> If you fix the issue, kindly add following tag
> Reported-by: kernel test robot <lkp(a)intel.com>
>
> [ 0.088943] Built 1 zonelists, mobility grouping on. Total pages: 129129
> [ 0.089917] Kernel command line: root=/dev/ram0 hung_task_panic=1 debug
apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 net.ifnames=0
printk.devkmsg=on panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2
prompt_ramdisk=0 drbd.minor_count=8 systemd.log_level=err ignore_loglevel console=tty0
earlyprintk=ttyS0,115200 console=ttyS0,115200 vga=normal rw
link=/cephfs/kbuild/run-queue/yocto-vm-yocto/x86_64-randconfig-s3-05171126/linux-devel:devel-hourly-2019051709:36b74766f295e0898345865a0511a443cf90aa13:bisect-linux-36/.vmlinuz-36b74766f295e0898345865a0511a443cf90aa13-20190519080629-7:yocto-vm-yocto-107
branch=linux-devel/devel-hourly-2019051709
BOOT_IMAGE=/pkg/linux/x86_64-randconfig-s3-05171126/gcc-7/36b74766f295e0898345865a0511a443cf90aa13/vmlinuz-5.1.0-10910-g36b7476
drbd.minor_count=8 rcuperf.shutdown=0
> [ 0.100683] sysrq: sysrq always enabled.
> [ 0.102667] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
> [ 0.103751] Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
> [ 0.124335] general protection fault: 0000 [#1] PREEMPT SMP PTI
> [ 0.125142] CPU: 0 PID: 0 Comm: swapper Not tainted 5.1.0-10910-g36b7476 #1
> [ 0.126049] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1
04/01/2014
> [ 0.127161] RIP: 0010:clear_page_orig+0x12/0x40
> [ 0.127763] Code: 90 90 90 90 90 90 90 90 b9 00 02 00 00 31 c0 f3 48 ab c3 0f 1f
44 00 00 31 c0 b9 40 00 00 00 66 0f 1f 84 00 00 00 00 00 ff c9 <48> 89 07 48 89 47
08 48 89 47 10 48 89 47 18 48 89 47 20 48 89 47
> [ 0.130272] RSP: 0000:ffffffff83403dc0 EFLAGS: 00010016
> [ 0.130953] RAX: 0000000000000000 RBX: ffff88801f2c4000 RCX: 000000000000003f
> [ 0.131876] RDX: ffff88801fb85000 RSI: 0000000000000000 RDI: b6daf636cb800000
> [ 0.132834] RBP: ffffffff83403df0 R08: 0000000000000000 R09: 0000000000000000
> [ 0.133800] R10: ffffffff83427800 R11: 0000000000000000 R12: 0000000000000400
> [ 0.134760] R13: 6db6db6db6db6db7 R14: ffff88801f2d2000 R15: 0000000000010400
> [ 0.135721] FS: 0000000000000000(0000) GS:ffff88801ee00000(0000)
knlGS:0000000000000000
> [ 0.136814] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 0.137622] CR2: ffff888005401000 CR3: 0000000003420001 CR4: 00000000000606b0
> [ 0.138598] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 0.139561] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> [ 0.140523] Call Trace:
> [ 0.140868] ? kernel_init_free_pages+0xa3/0x170
> [ 0.141504] memblock_free_pages+0x87/0xda
> [ 0.142066] memblock_free_all+0x2b9/0x372
> [ 0.142642] mem_init+0x30/0xc6
> [ 0.143078] start_kernel+0x361/0xc99
> [ 0.143591] x86_64_start_reservations+0x55/0x76
> [ 0.144223] x86_64_start_kernel+0x7e/0xa1
> [ 0.144776] secondary_startup_64+0xa4/0xb0
> [ 0.145328] Modules linked in:
> [ 0.145759] random: get_random_bytes called from init_oops_id+0x39/0x70 with
crng_init=0
> [ 0.146827] ---[ end trace 25feee6b5690789c ]---
> [ 0.147480] RIP: 0010:clear_page_orig+0x12/0x40
Turns out we can't call kernel_init_free_pages() from memblock_free_pages()
That's just a "old frame" (the leading "?" means that it found a
frame,
but it's not actually part of a real frame, IIUC). i.e. it was just left
over on the stack. memblock_free_pages() called clear_page_orig() and
crashed -- that's the place to be looking. It's not clear to me yet what
could cause this...
--
Kees Cook
1:26 a.m.
On Tue, May 21, 2019 at 10:21 PM Kees Cook <keescook(a)chromium.org> wrote:
On Tue, May 21, 2019 at 03:47:16PM +0200, Alexander Potapenko wrote:
> On Sun, May 19, 2019 at 3:19 AM kernel test robot <lkp(a)intel.com> wrote:
> >
> > Greetings,
> >
> > 0day kernel testing robot got the below dmesg and the first bad commit is
> >
> > https://git.kernel.org/pub/scm/linux/kernel/git/kees/linux.git
kspp/meminit/heap-v2
> >
> > commit 36b74766f295e0898345865a0511a443cf90aa13
> > Author: Alexander Potapenko <glider(a)google.com>
> > AuthorDate: Tue May 14 16:35:34 2019 +0200
> > Commit: Kees Cook <keescook(a)chromium.org>
> > CommitDate: Wed May 15 15:23:43 2019 -0700
> >
> > mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
> >
> > The new options are needed to prevent possible information leaks and
> > make control-flow bugs that depend on uninitialized values more
> > deterministic.
> >
> > init_on_alloc=1 makes the kernel initialize newly allocated pages and heap
> > objects with zeroes. Initialization is done at allocation time at the
> > places where checks for __GFP_ZERO are performed.
> >
> > init_on_free=1 makes the kernel initialize freed pages and heap objects
> > with zeroes upon their deletion. This helps to ensure sensitive data
> > doesn't leak via use-after-free accesses.
> >
> > Both init_on_alloc=1 and init_on_free=1 guarantee that the allocator
> > returns zeroed memory. The only exception is slab caches with
> > constructors. Those are never zero-initialized to preserve their
semantics.
> >
> > For SLOB allocator init_on_free=1 also implies init_on_alloc=1 behavior,
> > i.e. objects are zeroed at both allocation and deallocation time.
> > This is done because SLOB may otherwise return multiple freelist pointers
> > in the allocated object. For SLAB and SLUB enabling either init_on_alloc
> > or init_on_free leads to one-time initialization of the object.
> >
> > Both init_on_alloc and init_on_free default to zero, but those defaults
> > can be overridden with CONFIG_INIT_ON_ALLOC_DEFAULT_ON and
> > CONFIG_INIT_ON_FREE_DEFAULT_ON.
> >
> > Slowdown for the new features compared to init_on_free=0,
> > init_on_alloc=0:
> >
> > hackbench, init_on_free=1: +7.62% sys time (st.err 0.74%)
> > hackbench, init_on_alloc=1: +7.75% sys time (st.err 2.14%)
> >
> > Linux build with -j12, init_on_free=1: +8.38% wall time (st.err 0.39%)
> > Linux build with -j12, init_on_free=1: +24.42% sys time (st.err 0.52%)
> > Linux build with -j12, init_on_alloc=1: -0.13% wall time (st.err 0.42%)
> > Linux build with -j12, init_on_alloc=1: +0.57% sys time (st.err 0.40%)
> >
> > The slowdown for init_on_free=0, init_on_alloc=0 compared to the
> > baseline is within the standard error.
> >
> > Signed-off-by: Alexander Potapenko <glider(a)google.com>
> > To: Andrew Morton <akpm(a)linux-foundation.org>
> > To: Christoph Lameter <cl(a)linux.com>
> > To: Kees Cook <keescook(a)chromium.org>
> > Cc: Masahiro Yamada <yamada.masahiro(a)socionext.com>
> > Cc: James Morris <jmorris(a)namei.org>
> > Cc: "Serge E. Hallyn" <serge(a)hallyn.com>
> > Cc: Nick Desaulniers <ndesaulniers(a)google.com>
> > Cc: Kostya Serebryany <kcc(a)google.com>
> > Cc: Dmitry Vyukov <dvyukov(a)google.com>
> > Cc: Sandeep Patil <sspatil(a)android.com>
> > Cc: Laura Abbott <labbott(a)redhat.com>
> > Cc: Randy Dunlap <rdunlap(a)infradead.org>
> > Cc: Jann Horn <jannh(a)google.com>
> > Cc: Mark Rutland <mark.rutland(a)arm.com>
> > Cc: linux-mm(a)kvack.org
> > Cc: linux-security-module(a)vger.kernel.org
> > Cc: kernel-hardening(a)lists.openwall.com
> > Signed-off-by: Kees Cook <keescook(a)chromium.org>
> >
> > 2bbacd1a92 Merge tag 'kconfig-v5.2' of
git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild
> > 36b74766f2 mm: security: introduce init_on_alloc=1 and init_on_free=1 boot
options
> > 83e3ae9e99 mm: Introduce SLAB_NO_FREE_INIT and mark excluded caches
> >
+---------------------------------------------+------------+------------+------------+
> > | | 2bbacd1a92 | 36b74766f2 |
83e3ae9e99 |
> >
+---------------------------------------------+------------+------------+------------+
> > | boot_successes | 33 | 0 | 0
|
> > | boot_failures | 0 | 15 | 11
|
> > | BUG:kernel_NULL_pointer_dereference,address | 0 | 4 |
|
> > | Oops:#[##] | 0 | 4 |
|
> > | RIP:kernel_init_free_pages | 0 | 4 |
|
> > | Kernel_panic-not_syncing:Fatal_exception | 0 | 15 | 11
|
> > | general_protection_fault:#[##] | 0 | 11 | 11
|
> > | RIP:clear_page_orig | 0 | 11 | 11
|
> >
+---------------------------------------------+------------+------------+------------+
> >
> > If you fix the issue, kindly add following tag
> > Reported-by: kernel test robot <lkp(a)intel.com>
> >
> > [ 0.088943] Built 1 zonelists, mobility grouping on. Total pages: 129129
> > [ 0.089917] Kernel command line: root=/dev/ram0 hung_task_panic=1 debug
apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 net.ifnames=0
printk.devkmsg=on panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2
prompt_ramdisk=0 drbd.minor_count=8 systemd.log_level=err ignore_loglevel console=tty0
earlyprintk=ttyS0,115200 console=ttyS0,115200 vga=normal rw
link=/cephfs/kbuild/run-queue/yocto-vm-yocto/x86_64-randconfig-s3-05171126/linux-devel:devel-hourly-2019051709:36b74766f295e0898345865a0511a443cf90aa13:bisect-linux-36/.vmlinuz-36b74766f295e0898345865a0511a443cf90aa13-20190519080629-7:yocto-vm-yocto-107
branch=linux-devel/devel-hourly-2019051709
BOOT_IMAGE=/pkg/linux/x86_64-randconfig-s3-05171126/gcc-7/36b74766f295e0898345865a0511a443cf90aa13/vmlinuz-5.1.0-10910-g36b7476
drbd.minor_count=8 rcuperf.shutdown=0
> > [ 0.100683] sysrq: sysrq always enabled.
> > [ 0.102667] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
> > [ 0.103751] Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
> > [ 0.124335] general protection fault: 0000 [#1] PREEMPT SMP PTI
> > [ 0.125142] CPU: 0 PID: 0 Comm: swapper Not tainted 5.1.0-10910-g36b7476 #1
> > [ 0.126049] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
1.10.2-1 04/01/2014
> > [ 0.127161] RIP: 0010:clear_page_orig+0x12/0x40
> > [ 0.127763] Code: 90 90 90 90 90 90 90 90 b9 00 02 00 00 31 c0 f3 48 ab c3
0f 1f 44 00 00 31 c0 b9 40 00 00 00 66 0f 1f 84 00 00 00 00 00 ff c9 <48> 89 07 48
89 47 08 48 89 47 10 48 89 47 18 48 89 47 20 48 89 47
> > [ 0.130272] RSP: 0000:ffffffff83403dc0 EFLAGS: 00010016
> > [ 0.130953] RAX: 0000000000000000 RBX: ffff88801f2c4000 RCX:
000000000000003f
> > [ 0.131876] RDX: ffff88801fb85000 RSI: 0000000000000000 RDI:
b6daf636cb800000
> > [ 0.132834] RBP: ffffffff83403df0 R08: 0000000000000000 R09:
0000000000000000
> > [ 0.133800] R10: ffffffff83427800 R11: 0000000000000000 R12:
0000000000000400
> > [ 0.134760] R13: 6db6db6db6db6db7 R14: ffff88801f2d2000 R15:
0000000000010400
> > [ 0.135721] FS: 0000000000000000(0000) GS:ffff88801ee00000(0000)
knlGS:0000000000000000
> > [ 0.136814] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [ 0.137622] CR2: ffff888005401000 CR3: 0000000003420001 CR4:
00000000000606b0
> > [ 0.138598] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
> > [ 0.139561] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
> > [ 0.140523] Call Trace:
> > [ 0.140868] ? kernel_init_free_pages+0xa3/0x170
> > [ 0.141504] memblock_free_pages+0x87/0xda
> > [ 0.142066] memblock_free_all+0x2b9/0x372
> > [ 0.142642] mem_init+0x30/0xc6
> > [ 0.143078] start_kernel+0x361/0xc99
> > [ 0.143591] x86_64_start_reservations+0x55/0x76
> > [ 0.144223] x86_64_start_kernel+0x7e/0xa1
> > [ 0.144776] secondary_startup_64+0xa4/0xb0
> > [ 0.145328] Modules linked in:
> > [ 0.145759] random: get_random_bytes called from init_oops_id+0x39/0x70 with
crng_init=0
> > [ 0.146827] ---[ end trace 25feee6b5690789c ]---
> > [ 0.147480] RIP: 0010:clear_page_orig+0x12/0x40
> Turns out we can't call kernel_init_free_pages() from memblock_free_pages()
That's just a "old frame" (the leading "?" means that it found a
frame,
but it's not actually part of a real frame, IIUC). i.e. it was just left
over on the stack. memblock_free_pages() called clear_page_orig() and
crashed -- that's the place to be looking. It's not clear to me yet what
could cause this...
No, I believe my analysis is correct.
I've built the kernel with debug info and saw the same calls of
kernel_init_free_pages() from memblock_free_pages().
The call happened from the early_page_uninitialised(pfn) branch, which
is generally incorrect, as the struct page itself is, well,
uninitialized.
As such pages aren't passed to __free_pages_core(), it should be fine
to skip their initialization.
--
Kees Cook
--
Alexander Potapenko
Software Engineer
Google Germany GmbH
Erika-Mann-Straße, 33
80636 München
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
6:59 a.m.
On Wed, May 22, 2019 at 05:26:00PM +0200, Alexander Potapenko wrote:
No, I believe my analysis is correct.
I've built the kernel with debug info and saw the same calls of
kernel_init_free_pages() from memblock_free_pages().
The call happened from the early_page_uninitialised(pfn) branch, which
is generally incorrect, as the struct page itself is, well,
uninitialized.
As such pages aren't passed to __free_pages_core(), it should be fine
to skip their initialization.
Ah! I see. I misunderstood what you meant in the other email. Sorry! :)
Sounds like you've got a fix under way. Thanks!
--
Kees Cook
1215
days inactive
1218
days old
4 comments
3 participants
participants (3)
-
Alexander Potapenko -
Kees Cook -
kernel test robot