Greetings, 0day kernel testing robot got the below dmesg and the first bad commit is https://github.com/0day-ci/linux Aravinda-Prasad/perf-Container-aware-tracing-support/20160615-005257 commit f9eeff22064598831f35cbf00558468281ca87e0 Author: Aravinda Prasad <aravinda(a)linux.vnet.ibm.com&gt; AuthorDate: Tue Jun 14 22:19:51 2016 +0530 Commit: 0day robot <fengguang.wu(a)intel.com&gt; CommitDate: Wed Jun 15 00:52:59 2016 +0800 perf: Filter events based on perf-namespace Whenever perf tool is executed inside a container, this patch restricts the events to the perf-namespace in which the perf tool is executing. This patch is based on the existing support available for tracing with cgroups. TODO: - Avoid code duplication. Signed-off-by: Aravinda Prasad <aravinda(a)linux.vnet.ibm.com&gt; +------------------------------------------+------------+------------+------------+ | | 913295f26e | f9eeff2206 | df17832634 | +------------------------------------------+------------+------------+------------+ | boot_successes | 169 | 28 | 25 | | boot_failures | 0 | 39 | 5 | | BUG:unable_to_handle_kernel | 0 | 39 | 5 | | Oops | 0 | 39 | 5 | | RIP:ctx_sched_in | 0 | 39 | 5 | | Kernel_panic-not_syncing:Fatal_exception | 0 | 39 | 5 | | backtrace:do_group_exit | 0 | 39 | 5 | | backtrace:SyS_exit_group | 0 | 39 | 5 | +------------------------------------------+------------+------------+------------+ [ 4.451906] init: Failed to create pty - disabling logging for job [ 4.454930] init: Failed to create pty - disabling logging for job Kernel tests: Boot OK! [ 14.348543] BUG: unable to handle kernel NULL pointer dereference at 0000000000000038 [ 14.350389] IP: [<ffffffff811ba9c2>] ctx_sched_in+0xd4/0x5d0 [ 14.351524] PGD 0 [ 14.352217] Oops: 0000 [#1] SMP [ 14.353022] Modules linked in: [ 14.353861] CPU: 0 PID: 1962 Comm: trinity-main Not tainted 4.7.0-rc3-00022-gf9eeff2 #1 [ 14.355511] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Debian-1.8.2-1 04/01/2014 [ 14.357260] task: ffff8800119cb200 ti: ffff88000a6dc000 task.ti: ffff88000a6dc000 [ 14.358879] RIP: 0010:[<ffffffff811ba9c2>] [<ffffffff811ba9c2>] ctx_sched_in+0xd4/0x5d0 [ 14.360585] RSP: 0018:ffff88000a6dfa40 EFLAGS: 00010086 [ 14.361622] RAX: 0000000000000000 RBX: ffff8800119cb200 RCX: 0000000000000001 [ 14.362807] RDX: 0000000000000006 RSI: 0000000000000006 RDI: ffff880011521000 [ 14.363963] RBP: ffff88000a6dfa68 R08: ffff88000a6dc000 R09: 0000000000000002 [ 14.365149] R10: ffffffff82203e38 R11: 00000000000000de R12: ffff88000a65b700 [ 14.366356] R13: ffff88001121a0a0 R14: 0000000000000006 R15: 0000000000000008 [ 14.367538] FS: 00007fd05fbc3700(0000) GS:ffff880011200000(0000) knlGS:0000000000000000 [ 14.369189] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 14.370236] CR2: 0000000000000038 CR3: 0000000002206000 CR4: 00000000000006f0 [ 14.371433] Stack: [ 14.372066] ffff88001121a0a0 ffff88000a65b700 ffff8800119cb200 ffffffff8220b500 [ 14.373908] 0000000000000008 ffff88000a6dfa90 ffffffff811bb04a ffff88001121a0a0 [ 14.375752] ffff8800119cb200 ffff88000a65b700 ffff88000a6dfae8 ffffffff811bbbba [ 14.377681] Call Trace: [ 14.378368] [<ffffffff811bb04a>] perf_event_sched_in+0x3a/0x6c [ 14.379444] [<ffffffff811bbbba>] __perf_event_task_sched_in+0x102/0x179 [ 14.380631] [<ffffffff811bb950>] ? __perf_event_task_sched_out+0x256/0x3be [ 14.381783] [<ffffffff81123393>] finish_task_switch+0xba/0x1d2 [ 14.382850] [<ffffffff81a98bb4>] __schedule+0x3da/0x628 [ 14.383831] [<ffffffff81a98e87>] schedule+0x85/0x9a [ 14.384860] [<ffffffff81a9b598>] schedule_timeout+0x3c/0x1d2 [ 14.385878] [<ffffffff811d43a6>] ? release_pages+0xee/0x2a7 [ 14.386969] [<ffffffff811b2c98>] ? perf_event__header_size+0x23/0x26 [ 14.388092] [<ffffffff811b2b9a>] ? update_group_times+0xf/0x3c [ 14.389246] [<ffffffff81a997df>] wait_for_common+0x144/0x19b [ 14.390288] [<ffffffff81a997df>] ? wait_for_common+0x144/0x19b [ 14.391426] [<ffffffff811259a7>] ? wake_up_q+0x56/0x56 [ 14.392463] [<ffffffff81a99853>] wait_for_completion+0x1d/0x1f [ 14.393585] [<ffffffff81151246>] __wait_rcu_gp+0xc3/0xd8 [ 14.394600] [<ffffffff811534de>] synchronize_sched+0x75/0x77 [ 14.395683] [<ffffffff8115545f>] ? call_rcu_bh+0x19/0x19 [ 14.396745] [<ffffffff8115116f>] ? trace_raw_output_rcu_utilization+0x62/0x62 [ 14.398331] [<ffffffff811a584b>] perf_trace_event_unreg+0x37/0x78 [ 14.399542] [<ffffffff811a5b5c>] perf_trace_destroy+0x36/0x45 [ 14.400641] [<ffffffff811b5027>] tp_perf_event_destroy+0x9/0xb [ 14.401696] [<ffffffff811b8534>] _free_event+0x1da/0x220 [ 14.402708] [<ffffffff811b8906>] put_event+0x14/0x16 [ 14.403661] [<ffffffff811b8b7b>] perf_event_release_kernel+0x273/0x282 [ 14.404833] [<ffffffff811b8b9a>] perf_release+0x10/0x14 [ 14.405832] [<ffffffff81235228>] __fput+0x107/0x1ba [ 14.406801] [<ffffffff81235311>] ____fput+0xe/0x10 [ 14.407751] [<ffffffff8111c10f>] task_work_run+0x6f/0x9a [ 14.408817] [<ffffffff81106381>] do_exit+0x462/0x9a6 [ 14.409791] [<ffffffff811a4f3e>] ? perf_syscall_enter+0x1a/0x199 [ 14.410882] [<ffffffff8110693f>] do_group_exit+0x44/0xad [ 14.411866] [<ffffffff811069bc>] SyS_exit_group+0x14/0x14 [ 14.412936] [<ffffffff810019d0>] do_syscall_64+0x69/0xb1 [ 14.413969] [<ffffffff81a9c53a>] entry_SYSCALL64_slow_path+0x25/0x25 [ 14.415137] Code: 48 8b 93 50 07 00 00 48 8b 92 80 00 00 00 48 8b 92 b8 00 00 00 65 48 03 15 54 f8 e4 7e 48 89 42 08 48 8b 83 b0 05 00 00 48 85 db <48> 8b 40 38 74 28 41 83 bc 24 d0 00 00 00 00 74 1d 48 8b 40 08 [ 14.423010] RIP [<ffffffff811ba9c2>] ctx_sched_in+0xd4/0x5d0 [ 14.424132] RSP <ffff88000a6dfa40> [ 14.424990] CR2: 0000000000000038 [ 14.425796] ---[ end trace 3a0c938f2607610f ]--- [ 14.426758] Kernel panic - not syncing: Fatal exception git bisect start df17832634217924387de85509f91b7c5b42d25e 5edb56491d4812c42175980759da53388e5d86f5 -- git bisect good a0a92f717a9ef5e60cdce0bd17593a779da50436 # 02:23 57+ 2 Merge 'arm-soc/for-next' into devel-catchup-201606150139 git bisect good 5e2bf2ba113169070cf93227b9992e3811b10e9e # 02:40 54+ 0 Merge 'kvm/queue' into devel-catchup-201606150139 git bisect bad 92c3d5d8c5ebe63b70622dcb25bcb1f9748c2def # 02:48 0- 13 Merge 'linux-review/Aravinda-Prasad/perf-Container-aware-tracing-support/20160615-005257' into devel-catchup-201606150139 git bisect good 9092b855997732c61755d21902115a05dc69bfb2 # 04:13 57+ 0 Merge '96boards/96b/defconfigs' into devel-catchup-201606150139 git bisect good 4daa6d79bb7b79762350c6b6dac801afcfff171f # 04:19 57+ 0 Merge 'asoc/topic/arizona' into devel-catchup-201606150139 git bisect good b7fa30c9cc48c4f55663420472505d3b4f6e1705 # 04:25 54+ 0 sched/fair: Fix post_init_entity_util_avg() serialization git bisect good 03c041c5bf6ed584dff36b7cd509e0146a124277 # 04:27 56+ 2 sched/debug: Always show 'nr_migrations' git bisect good 3d89e5478bf550a50c99e93adf659369798263b0 # 04:34 57+ 0 sched/cputime: Fix prev steal time accouting during CPU hotplug git bisect good 913295f26e7a810ad1b6649b1a14d665519d210a # 04:41 57+ 0 perf/namespaces: Add a new namespace for isolated tracing git bisect bad f9eeff22064598831f35cbf00558468281ca87e0 # 04:47 0- 25 perf: Filter events based on perf-namespace # first bad commit: [f9eeff22064598831f35cbf00558468281ca87e0] perf: Filter events based on perf-namespace git bisect good 913295f26e7a810ad1b6649b1a14d665519d210a # 04:50 158+ 0 perf/namespaces: Add a new namespace for isolated tracing # extra tests on HEAD of linux-devel/devel-catchup-201606150139 git bisect bad df17832634217924387de85509f91b7c5b42d25e # 04:50 0- 5 0day head guard for 'devel-catchup-201606150139' # extra tests on tree/branch linux-review/Aravinda-Prasad/perf-Container-aware-tracing-support/20160615-005257 git bisect bad f9eeff22064598831f35cbf00558468281ca87e0 # 04:56 0- 33 perf: Filter events based on perf-namespace # extra tests with first bad commit reverted git bisect good 5a97357b9e737438817fab62c6364c4a2c009632 # 05:09 163+ 0 Revert "perf: Filter events based on perf-namespace" # extra tests on tree/branch linus/master git bisect good db06d759d6cf903aeda8c107fd3abd366dd80200 # 05:57 166+ 0 Merge branch 'for-4.7-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu # extra tests on tree/branch linux-next/master git bisect good 81e615f6c645cb39359a9a7e40177682c299e597 # 06:02 166+ 0 Add linux-next specific files for 20160614 This script may reproduce the error. ---------------------------------------------------------------------------- #!/bin/bash kernel=$1 initrd=quantal-core-x86_64.cgz wget --no-clobber https://github.com/fengguang/reproduce-kernel-bug/raw/master/initrd/$initrd kvm=( qemu-system-x86_64 -enable-kvm -cpu kvm64 -kernel $kernel -initrd $initrd -m 300 -smp 2 -device e1000,netdev=net0 -netdev user,id=net0 -boot order=nc -no-reboot -watchdog i6300esb -rtc base=localtime -serial stdio -display none -monitor null ) append=( hung_task_panic=1 earlyprintk=ttyS0,115200 systemd.log_level=err debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0 console=ttyS0,115200 console=tty0 vga=normal root=/dev/ram0 rw drbd.minor_count=8 ) "${kvm[@]}" --append "${append[*]}" ---------------------------------------------------------------------------- --- 0-DAY kernel test infrastructure Open Source Technology Center https://lists.01.org/pipermail/lkp Intel Corporation