1:32 a.m.
FYI, we noticed the following commit (built with gcc-7):
commit: 07173c3ec276cbb18dc0e0687d37d310e98a1480 ("block: enable multipage
bvecs")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
in testcase: fio-basic
on test machine: 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G memory
with following parameters:
runtime: 300s
nr_task: 8t
disk: 1SSD
fs: xfs
rw: randread
bs: 2M
ioengine: sync
test_size: 512g
caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
fio: pid=0, err=5/file:filesetup.c:190, func=fsync, error=Input/output error
fio: pid=0, err=5/file:filesetup.c:190, func=fsync, error=Input/output error
To reproduce:
git clonehttps://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp run job.yaml
Thanks,
Rong Chen
3:47 a.m.
Hi rong.a.chen,
Thanks for your report!
On Tue, Jun 25, 2019 at 04:32:48PM +0800, kernel test robot wrote:
FYI, we noticed the following commit (built with gcc-7):
commit: 07173c3ec276cbb18dc0e0687d37d310e98a1480 ("block: enable multipage
bvecs")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
in testcase: fio-basic
on test machine: 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G memory
with following parameters:
runtime: 300s
nr_task: 8t
disk: 1SSD
fs: xfs
rw: randread
bs: 2M
ioengine: sync
test_size: 512g
caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
fio: pid=0, err=5/file:filesetup.c:190, func=fsync, error=Input/output error
fio: pid=0, err=5/file:filesetup.c:190, func=fsync, error=Input/output error
I just tried the fio(randread, bs:2M, sync, nvme, xfs and block device) test on
5.2-rc6 kernel, looks it can't be reproduced.
For multi-page bvec change, I'd suggest to run test on 5.1 at least,
since there are several follow-up fixes.
Also the dmesg log seems not intact, looks it only shows booting.
Please let us know if the issue can be triggered on 5.1 or 5.2.
Thanks,
Ming
6:18 p.m.
Hi, Ming Lei.
在 2019/6/25 下午6:47, Ming Lei 写道:
Hi rong.a.chen,
Thanks for your report!
On Tue, Jun 25, 2019 at 04:32:48PM +0800, kernel test robot wrote:
> FYI, we noticed the following commit (built with gcc-7):
>
> commit: 07173c3ec276cbb18dc0e0687d37d310e98a1480 ("block: enable multipage
bvecs")
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
>
> in testcase: fio-basic
> on test machine: 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G
memory
> with following parameters:
>
> runtime: 300s
> nr_task: 8t
> disk: 1SSD
> fs: xfs
> rw: randread
> bs: 2M
> ioengine: sync
> test_size: 512g
>
> caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
>
> fio: pid=0, err=5/file:filesetup.c:190, func=fsync, error=Input/output error
> fio: pid=0, err=5/file:filesetup.c:190, func=fsync, error=Input/output error
I just tried the fio(randread, bs:2M, sync, nvme, xfs and block device) test on
5.2-rc6 kernel, looks it can't be reproduced.
For multi-page bvec change, I'd suggest to run test on 5.1 at least,
since there are several follow-up fixes.
Also the dmesg log seems not intact, looks it only shows booting.
Please let us know if the issue can be triggered on 5.1 or 5.2.
This issue can be triggered on v5.1 and v5,2-rc* (include v5.2-rc6)
on v5.2-rc6 run 20 times, 10 times trigger out this issue.
Thanks,
Ming
_______________________________________________
LKP mailing list
LKP(a)lists.01.org
https://lists.01.org/mailman/listinfo/lkp
7:05 p.m.
On Wed, Jun 26, 2019 at 09:18:36AM +0800, Liu Yiding wrote:
Hi, Ming Lei.
在 2019/6/25 下午6:47, Ming Lei 写道:
> Hi rong.a.chen,
>
> Thanks for your report!
>
> On Tue, Jun 25, 2019 at 04:32:48PM +0800, kernel test robot wrote:
> > FYI, we noticed the following commit (built with gcc-7):
> >
> > commit: 07173c3ec276cbb18dc0e0687d37d310e98a1480 ("block: enable multipage
bvecs")
> > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> >
> > in testcase: fio-basic
> > on test machine: 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G
memory
> > with following parameters:
> >
> > runtime: 300s
> > nr_task: 8t
> > disk: 1SSD
> > fs: xfs
> > rw: randread
> > bs: 2M
> > ioengine: sync
> > test_size: 512g
> >
> > caused below changes (please refer to attached dmesg/kmsg for entire
log/backtrace):
> >
> > fio: pid=0, err=5/file:filesetup.c:190, func=fsync, error=Input/output error
> > fio: pid=0, err=5/file:filesetup.c:190, func=fsync, error=Input/output error
> I just tried the fio(randread, bs:2M, sync, nvme, xfs and block device) test on
> 5.2-rc6 kernel, looks it can't be reproduced.
>
> For multi-page bvec change, I'd suggest to run test on 5.1 at least,
> since there are several follow-up fixes.
>
> Also the dmesg log seems not intact, looks it only shows booting.
>
> Please let us know if the issue can be triggered on 5.1 or 5.2.
This issue can be triggered on v5.1 and v5,2-rc* (include v5.2-rc6)
on v5.2-rc6 run 20 times, 10 times trigger out this issue.
OK, thanks!
It is EIO from fsync(), which should have something dumped in dmesg log,
could you provide me the dmesg log after the issue is triggered?
BTW, can you provide me the exact fio command in your test? I need to
find a big machine like yours for reproducing it.
Thanks,
Ming
7:21 p.m.
在 2019/6/26 上午10:05, Ming Lei 写道:
On Wed, Jun 26, 2019 at 09:18:36AM +0800, Liu Yiding wrote:
> Hi, Ming Lei.
>
>
> 在 2019/6/25 下午6:47, Ming Lei 写道:
>> Hi rong.a.chen,
>>
>> Thanks for your report!
>>
>> On Tue, Jun 25, 2019 at 04:32:48PM +0800, kernel test robot wrote:
>>> FYI, we noticed the following commit (built with gcc-7):
>>>
>>> commit: 07173c3ec276cbb18dc0e0687d37d310e98a1480 ("block: enable
multipage bvecs")
>>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
>>>
>>> in testcase: fio-basic
>>> on test machine: 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with
256G memory
>>> with following parameters:
>>>
>>> runtime: 300s
>>> nr_task: 8t
>>> disk: 1SSD
>>> fs: xfs
>>> rw: randread
>>> bs: 2M
>>> ioengine: sync
>>> test_size: 512g
>>>
>>> caused below changes (please refer to attached dmesg/kmsg for entire
log/backtrace):
>>>
>>> fio: pid=0, err=5/file:filesetup.c:190, func=fsync, error=Input/output error
>>> fio: pid=0, err=5/file:filesetup.c:190, func=fsync, error=Input/output error
>> I just tried the fio(randread, bs:2M, sync, nvme, xfs and block device) test on
>> 5.2-rc6 kernel, looks it can't be reproduced.
>>
>> For multi-page bvec change, I'd suggest to run test on 5.1 at least,
>> since there are several follow-up fixes.
>>
>> Also the dmesg log seems not intact, looks it only shows booting.
>>
>> Please let us know if the issue can be triggered on 5.1 or 5.2.
>
> This issue can be triggered on v5.1 and v5,2-rc* (include v5.2-rc6)
>
> on v5.2-rc6 run 20 times, 10 times trigger out this issue.
OK, thanks!
It is EIO from fsync(), which should have something dumped in dmesg log,
could you provide me the dmesg log after the issue is triggered?
i have attatched the whole dmesg.xz on email.
BTW, can you provide me the exact fio command in your test? I need to
find a big machine like yours for reproducing it.
lkp will do some test env setting for fio test, it seems that this
error happend in mount.
------
[ 107.314810] 2019-06-19 03:37:09 mount -t xfs -o inode64 /dev/sdb1
/fs/sdb1
[ 107.314812]
[ 107.316017] SGI XFS with ACLs, security attributes, realtime, scrub,
debug enabled
[ 107.338674] XFS (sdb1): Mounting V5 Filesystem
[ 107.356339] XFS (sdb1): Ending clean mount
[ 683.136603] WARNING: CPU: 1 PID: 1484 at drivers/scsi/scsi_lib.c:1010
scsi_init_io+0x103/0x170
[ 683.148708] Modules linked in: xfs dm_mod btrfs xor zstd_decompress
zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac
x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm mgag200 ttm
irqbypass drm_kms_helper crct10dif_pclmul crc32_pclmul crc32c_intel
syscopyarea ghash_clmulni_intel snd_pcm sysfillrect ipmi_si snd_timer
sysimgblt aesni_intel ahci ipmi_devintf crypto_simd snd fb_sys_fops
libahci soundcore mei_me cryptd glue_helper ipmi_msghandler pcspkr drm
libata mei ioatdma joydev lpc_ich i2c_i801 dca wmi acpi_power_meter
acpi_pad pcc_cpufreq ip_tables
[ 683.207865] CPU: 1 PID: 1484 Comm: kworker/1:1H Tainted: G
I 5.2.0-rc5 #1
[ 683.217421] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
SE5C610.86B.01.01.0008.021120151325 02/11/2015
[ 683.229600] Workqueue: kblockd blk_mq_run_work_fn
[ 683.235364] RIP: 0010:scsi_init_io+0x103/0x170
[ 683.240834] Code: 5c 41 5d c3 be 01 00 00 00 e9 49 ff ff ff 8b 45 70
89 83 08 01 00 00 f6 45 1a 01 75 92 31 c0 48 83 c4 08 5b 5d 41 5c 41 5d
c3 <0f> 0b 48 83 c4 08 b8 0a 00 00 00 5b 5d 41 5c 41 5d c3 48 8b 75 38
[ 683.262867] RSP: 0018:ffffb8f1cf853c78 EFLAGS: 00010246
[ 683.269260] RAX: 000000001d338288 RBX: ffff8d2af2612690 RCX:
0000000000000000
[ 683.277768] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
ffff8d2af2612690
[ 683.286278] RBP: ffff8d2af2612580 R08: 0000000000000000 R09:
ffff8d2af26126c8
[ 683.294785] R10: 0000000000000000 R11: fefefefefefefeff R12:
0000000000000801
[ 683.303295] R13: 0000000000000000 R14: 0000000000000000 R15:
ffff8d2b0b99a800
[ 683.311804] FS: 0000000000000000(0000) GS:ffff8d2b3f240000(0000)
knlGS:0000000000000000
[ 683.321379] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 683.328331] CR2: 00007faa9d9697d0 CR3: 000000407a8a8003 CR4:
00000000001606e0
[ 683.336843] Call Trace:
[ 683.340122] sd_init_command+0x2ee/0xaf0 [sd_mod]
[ 683.345917] scsi_queue_rq+0x559/0xa80
[ 683.350639] blk_mq_dispatch_rq_list+0x3da/0x5b0
[ 683.356333] ? syscall_return_via_sysret+0x10/0x7f
[ 683.362276] ? __switch_to_asm+0x40/0x70
[ 683.367187] ? elv_rb_del+0x1f/0x30
[ 683.371607] ? deadline_remove_request+0x55/0xc0
[ 683.377290] blk_mq_do_dispatch_sched+0x76/0x120
[ 683.382990] blk_mq_sched_dispatch_requests+0x100/0x170
[ 683.389349] __blk_mq_run_hw_queue+0x60/0x130
[ 683.394745] process_one_work+0x19c/0x3c0
[ 683.399746] worker_thread+0x3c/0x3b0
[ 683.404392] ? process_one_work+0x3c0/0x3c0
[ 683.409580] kthread+0x11e/0x140
[ 683.413734] ? kthread_park+0x90/0x90
[ 683.418335] ret_from_fork+0x35/0x40
[ 683.422841] ---[ end trace f691cac020928424 ]---
[ 683.428614] print_req_error: I/O error, dev sdb, sector 489915016
flags 801
[ 683.561809] XFS (sdb1): writeback error on sector 489915016
[ 732.579401] fio: pid=0, err=5/file:filesetup.c:190, func=fsync,
error=Input/output error
[ 732.579403]
[ 1022.580119] print_req_error: I/O error, dev sdb, sector 795883128
flags 801
[ 1022.933676] XFS (sdb1): writeback error on sector 795883128
[ 1035.041938] fio: pid=0, err=5/file:filesetup.c:190, func=fsync,
error=Input/output error
------
Thanks,
Ming
7:28 p.m.
attatch reproduce.sh
在 2019/6/26 上午10:21, Liu Yiding 写道:
在 2019/6/26 上午10:05, Ming Lei 写道:
> On Wed, Jun 26, 2019 at 09:18:36AM +0800, Liu Yiding wrote:
>> Hi, Ming Lei.
>>
>>
>> 在 2019/6/25 下午6:47, Ming Lei 写道:
>>> Hi rong.a.chen,
>>>
>>> Thanks for your report!
>>>
>>> On Tue, Jun 25, 2019 at 04:32:48PM +0800, kernel test robot wrote:
>>>> FYI, we noticed the following commit (built with gcc-7):
>>>>
>>>> commit: 07173c3ec276cbb18dc0e0687d37d310e98a1480 ("block: enable
>>>> multipage bvecs")
>>>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git
>>>> master
>>>>
>>>> in testcase: fio-basic
>>>> on test machine: 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @
>>>> 2.30GHz with 256G memory
>>>> with following parameters:
>>>>
>>>> runtime: 300s
>>>> nr_task: 8t
>>>> disk: 1SSD
>>>> fs: xfs
>>>> rw: randread
>>>> bs: 2M
>>>> ioengine: sync
>>>> test_size: 512g
>>>>
>>>> caused below changes (please refer to attached dmesg/kmsg for
>>>> entire log/backtrace):
>>>>
>>>> fio: pid=0, err=5/file:filesetup.c:190, func=fsync,
>>>> error=Input/output error
>>>> fio: pid=0, err=5/file:filesetup.c:190, func=fsync,
>>>> error=Input/output error
>>> I just tried the fio(randread, bs:2M, sync, nvme, xfs and block
>>> device) test on
>>> 5.2-rc6 kernel, looks it can't be reproduced.
>>>
>>> For multi-page bvec change, I'd suggest to run test on 5.1 at least,
>>> since there are several follow-up fixes.
>>>
>>> Also the dmesg log seems not intact, looks it only shows booting.
>>>
>>> Please let us know if the issue can be triggered on 5.1 or 5.2.
>>
>> This issue can be triggered on v5.1 and v5,2-rc* (include v5.2-rc6)
>>
>> on v5.2-rc6 run 20 times, 10 times trigger out this issue.
> OK, thanks!
>
> It is EIO from fsync(), which should have something dumped in dmesg log,
> could you provide me the dmesg log after the issue is triggered?
i have attatched the whole dmesg.xz on email.
>
> BTW, can you provide me the exact fio command in your test? I need to
> find a big machine like yours for reproducing it.
lkp will do some test env setting for fio test, it seems that this
error happend in mount.
------
[ 107.314810] 2019-06-19 03:37:09 mount -t xfs -o inode64 /dev/sdb1
/fs/sdb1
[ 107.314812]
[ 107.316017] SGI XFS with ACLs, security attributes, realtime,
scrub, debug enabled
[ 107.338674] XFS (sdb1): Mounting V5 Filesystem
[ 107.356339] XFS (sdb1): Ending clean mount
[ 683.136603] WARNING: CPU: 1 PID: 1484 at
drivers/scsi/scsi_lib.c:1010 scsi_init_io+0x103/0x170
[ 683.148708] Modules linked in: xfs dm_mod btrfs xor zstd_decompress
zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac
x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm mgag200
ttm irqbypass drm_kms_helper crct10dif_pclmul crc32_pclmul
crc32c_intel syscopyarea ghash_clmulni_intel snd_pcm sysfillrect
ipmi_si snd_timer sysimgblt aesni_intel ahci ipmi_devintf crypto_simd
snd fb_sys_fops libahci soundcore mei_me cryptd glue_helper
ipmi_msghandler pcspkr drm libata mei ioatdma joydev lpc_ich i2c_i801
dca wmi acpi_power_meter acpi_pad pcc_cpufreq ip_tables
[ 683.207865] CPU: 1 PID: 1484 Comm: kworker/1:1H Tainted: G
I 5.2.0-rc5 #1
[ 683.217421] Hardware name: Intel Corporation S2600WTT/S2600WTT,
BIOS SE5C610.86B.01.01.0008.021120151325 02/11/2015
[ 683.229600] Workqueue: kblockd blk_mq_run_work_fn
[ 683.235364] RIP: 0010:scsi_init_io+0x103/0x170
[ 683.240834] Code: 5c 41 5d c3 be 01 00 00 00 e9 49 ff ff ff 8b 45
70 89 83 08 01 00 00 f6 45 1a 01 75 92 31 c0 48 83 c4 08 5b 5d 41 5c
41 5d c3 <0f> 0b 48 83 c4 08 b8 0a 00 00 00 5b 5d 41 5c 41 5d c3 48 8b
75 38
[ 683.262867] RSP: 0018:ffffb8f1cf853c78 EFLAGS: 00010246
[ 683.269260] RAX: 000000001d338288 RBX: ffff8d2af2612690 RCX:
0000000000000000
[ 683.277768] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
ffff8d2af2612690
[ 683.286278] RBP: ffff8d2af2612580 R08: 0000000000000000 R09:
ffff8d2af26126c8
[ 683.294785] R10: 0000000000000000 R11: fefefefefefefeff R12:
0000000000000801
[ 683.303295] R13: 0000000000000000 R14: 0000000000000000 R15:
ffff8d2b0b99a800
[ 683.311804] FS: 0000000000000000(0000) GS:ffff8d2b3f240000(0000)
knlGS:0000000000000000
[ 683.321379] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 683.328331] CR2: 00007faa9d9697d0 CR3: 000000407a8a8003 CR4:
00000000001606e0
[ 683.336843] Call Trace:
[ 683.340122] sd_init_command+0x2ee/0xaf0 [sd_mod]
[ 683.345917] scsi_queue_rq+0x559/0xa80
[ 683.350639] blk_mq_dispatch_rq_list+0x3da/0x5b0
[ 683.356333] ? syscall_return_via_sysret+0x10/0x7f
[ 683.362276] ? __switch_to_asm+0x40/0x70
[ 683.367187] ? elv_rb_del+0x1f/0x30
[ 683.371607] ? deadline_remove_request+0x55/0xc0
[ 683.377290] blk_mq_do_dispatch_sched+0x76/0x120
[ 683.382990] blk_mq_sched_dispatch_requests+0x100/0x170
[ 683.389349] __blk_mq_run_hw_queue+0x60/0x130
[ 683.394745] process_one_work+0x19c/0x3c0
[ 683.399746] worker_thread+0x3c/0x3b0
[ 683.404392] ? process_one_work+0x3c0/0x3c0
[ 683.409580] kthread+0x11e/0x140
[ 683.413734] ? kthread_park+0x90/0x90
[ 683.418335] ret_from_fork+0x35/0x40
[ 683.422841] ---[ end trace f691cac020928424 ]---
[ 683.428614] print_req_error: I/O error, dev sdb, sector 489915016
flags 801
[ 683.561809] XFS (sdb1): writeback error on sector 489915016
[ 732.579401] fio: pid=0, err=5/file:filesetup.c:190, func=fsync,
error=Input/output error
[ 732.579403]
[ 1022.580119] print_req_error: I/O error, dev sdb, sector 795883128
flags 801
[ 1022.933676] XFS (sdb1): writeback error on sector 795883128
[ 1035.041938] fio: pid=0, err=5/file:filesetup.c:190, func=fsync,
error=Input/output error
------
>
> Thanks,
> Ming
>
>
_______________________________________________
LKP mailing list
LKP(a)lists.01.org
https://lists.01.org/mailman/listinfo/lkp
7:42 p.m.
sorry, something wrong with attatched file.
------
wipefs -a --force /dev/sdb1
mkfs -t xfs /dev/sdb1
mkdir -p /fs/sdb1
mount -t xfs -o inode64 /dev/sdb1 /fs/sdb1
for cpu_dir in /sys/devices/system/cpu/cpu[0-9]*
do
online_file="$cpu_dir"/online
[ -f "$online_file" ] && [ "$(cat
"$online_file")" -eq 0 ] && continue
file="$cpu_dir"/cpufreq/scaling_governor
[ -f "$file" ] && echo "performance" >
"$file"
done
echo '[global]
bs=2M
ioengine=sync
iodepth=32
size=68719476736
direct=0
runtime=300
invalidate=1
fallocate=posix
group_reporting
[task_0]
rw=randread
directory=/fs/sdb1
numjobs=8' | fio --output-format=json -
umount /fs/sdb1
------
在 2019/6/26 上午10:28, Liu Yiding 写道:
attatch reproduce.sh
在 2019/6/26 上午10:21, Liu Yiding 写道:
>
> 在 2019/6/26 上午10:05, Ming Lei 写道:
>> On Wed, Jun 26, 2019 at 09:18:36AM +0800, Liu Yiding wrote:
>>> Hi, Ming Lei.
>>>
>>>
>>> 在 2019/6/25 下午6:47, Ming Lei 写道:
>>>> Hi rong.a.chen,
>>>>
>>>> Thanks for your report!
>>>>
>>>> On Tue, Jun 25, 2019 at 04:32:48PM +0800, kernel test robot wrote:
>>>>> FYI, we noticed the following commit (built with gcc-7):
>>>>>
>>>>> commit: 07173c3ec276cbb18dc0e0687d37d310e98a1480 ("block: enable
>>>>> multipage bvecs")
>>>>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git
>>>>> master
>>>>>
>>>>> in testcase: fio-basic
>>>>> on test machine: 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @
>>>>> 2.30GHz with 256G memory
>>>>> with following parameters:
>>>>>
>>>>> runtime: 300s
>>>>> nr_task: 8t
>>>>> disk: 1SSD
>>>>> fs: xfs
>>>>> rw: randread
>>>>> bs: 2M
>>>>> ioengine: sync
>>>>> test_size: 512g
>>>>>
>>>>> caused below changes (please refer to attached dmesg/kmsg for
>>>>> entire log/backtrace):
>>>>>
>>>>> fio: pid=0, err=5/file:filesetup.c:190, func=fsync,
>>>>> error=Input/output error
>>>>> fio: pid=0, err=5/file:filesetup.c:190, func=fsync,
>>>>> error=Input/output error
>>>> I just tried the fio(randread, bs:2M, sync, nvme, xfs and block
>>>> device) test on
>>>> 5.2-rc6 kernel, looks it can't be reproduced.
>>>>
>>>> For multi-page bvec change, I'd suggest to run test on 5.1 at least,
>>>> since there are several follow-up fixes.
>>>>
>>>> Also the dmesg log seems not intact, looks it only shows booting.
>>>>
>>>> Please let us know if the issue can be triggered on 5.1 or 5.2.
>>>
>>> This issue can be triggered on v5.1 and v5,2-rc* (include v5.2-rc6)
>>>
>>> on v5.2-rc6 run 20 times, 10 times trigger out this issue.
>> OK, thanks!
>>
>> It is EIO from fsync(), which should have something dumped in dmesg
>> log,
>> could you provide me the dmesg log after the issue is triggered?
>
>
> i have attatched the whole dmesg.xz on email.
>
>
>>
>> BTW, can you provide me the exact fio command in your test? I need to
>> find a big machine like yours for reproducing it.
>
> lkp will do some test env setting for fio test, it seems that this
> error happend in mount.
>
> ------
>
> [ 107.314810] 2019-06-19 03:37:09 mount -t xfs -o inode64 /dev/sdb1
> /fs/sdb1
> [ 107.314812]
> [ 107.316017] SGI XFS with ACLs, security attributes, realtime,
> scrub, debug enabled
> [ 107.338674] XFS (sdb1): Mounting V5 Filesystem
> [ 107.356339] XFS (sdb1): Ending clean mount
> [ 683.136603] WARNING: CPU: 1 PID: 1484 at
> drivers/scsi/scsi_lib.c:1010 scsi_init_io+0x103/0x170
> [ 683.148708] Modules linked in: xfs dm_mod btrfs xor
> zstd_decompress zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl
> sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm
> mgag200 ttm irqbypass drm_kms_helper crct10dif_pclmul crc32_pclmul
> crc32c_intel syscopyarea ghash_clmulni_intel snd_pcm sysfillrect
> ipmi_si snd_timer sysimgblt aesni_intel ahci ipmi_devintf crypto_simd
> snd fb_sys_fops libahci soundcore mei_me cryptd glue_helper
> ipmi_msghandler pcspkr drm libata mei ioatdma joydev lpc_ich i2c_i801
> dca wmi acpi_power_meter acpi_pad pcc_cpufreq ip_tables
> [ 683.207865] CPU: 1 PID: 1484 Comm: kworker/1:1H Tainted:
> G I 5.2.0-rc5 #1
> [ 683.217421] Hardware name: Intel Corporation S2600WTT/S2600WTT,
> BIOS SE5C610.86B.01.01.0008.021120151325 02/11/2015
> [ 683.229600] Workqueue: kblockd blk_mq_run_work_fn
> [ 683.235364] RIP: 0010:scsi_init_io+0x103/0x170
> [ 683.240834] Code: 5c 41 5d c3 be 01 00 00 00 e9 49 ff ff ff 8b 45
> 70 89 83 08 01 00 00 f6 45 1a 01 75 92 31 c0 48 83 c4 08 5b 5d 41 5c
> 41 5d c3 <0f> 0b 48 83 c4 08 b8 0a 00 00 00 5b 5d 41 5c 41 5d c3 48
> 8b 75 38
> [ 683.262867] RSP: 0018:ffffb8f1cf853c78 EFLAGS: 00010246
> [ 683.269260] RAX: 000000001d338288 RBX: ffff8d2af2612690 RCX:
> 0000000000000000
> [ 683.277768] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
> ffff8d2af2612690
> [ 683.286278] RBP: ffff8d2af2612580 R08: 0000000000000000 R09:
> ffff8d2af26126c8
> [ 683.294785] R10: 0000000000000000 R11: fefefefefefefeff R12:
> 0000000000000801
> [ 683.303295] R13: 0000000000000000 R14: 0000000000000000 R15:
> ffff8d2b0b99a800
> [ 683.311804] FS: 0000000000000000(0000) GS:ffff8d2b3f240000(0000)
> knlGS:0000000000000000
> [ 683.321379] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 683.328331] CR2: 00007faa9d9697d0 CR3: 000000407a8a8003 CR4:
> 00000000001606e0
> [ 683.336843] Call Trace:
> [ 683.340122] sd_init_command+0x2ee/0xaf0 [sd_mod]
> [ 683.345917] scsi_queue_rq+0x559/0xa80
> [ 683.350639] blk_mq_dispatch_rq_list+0x3da/0x5b0
> [ 683.356333] ? syscall_return_via_sysret+0x10/0x7f
> [ 683.362276] ? __switch_to_asm+0x40/0x70
> [ 683.367187] ? elv_rb_del+0x1f/0x30
> [ 683.371607] ? deadline_remove_request+0x55/0xc0
> [ 683.377290] blk_mq_do_dispatch_sched+0x76/0x120
> [ 683.382990] blk_mq_sched_dispatch_requests+0x100/0x170
> [ 683.389349] __blk_mq_run_hw_queue+0x60/0x130
> [ 683.394745] process_one_work+0x19c/0x3c0
> [ 683.399746] worker_thread+0x3c/0x3b0
> [ 683.404392] ? process_one_work+0x3c0/0x3c0
> [ 683.409580] kthread+0x11e/0x140
> [ 683.413734] ? kthread_park+0x90/0x90
> [ 683.418335] ret_from_fork+0x35/0x40
> [ 683.422841] ---[ end trace f691cac020928424 ]---
> [ 683.428614] print_req_error: I/O error, dev sdb, sector 489915016
> flags 801
> [ 683.561809] XFS (sdb1): writeback error on sector 489915016
> [ 732.579401] fio: pid=0, err=5/file:filesetup.c:190, func=fsync,
> error=Input/output error
> [ 732.579403]
> [ 1022.580119] print_req_error: I/O error, dev sdb, sector 795883128
> flags 801
> [ 1022.933676] XFS (sdb1): writeback error on sector 795883128
> [ 1035.041938] fio: pid=0, err=5/file:filesetup.c:190, func=fsync,
> error=Input/output error
>
> ------
>
>
>>
>> Thanks,
>> Ming
>>
>>
>
>
>
> _______________________________________________
> LKP mailing list
> LKP(a)lists.01.org
> https://lists.01.org/mailman/listinfo/lkp
_______________________________________________
LKP mailing list
LKP(a)lists.01.org
https://lists.01.org/mailman/listinfo/lkp
8:32 p.m.
On Wed, Jun 26, 2019 at 10:42:52AM +0800, Liu Yiding wrote:
sorry, something wrong with attatched file.
------
wipefs -a --force /dev/sdb1
mkfs -t xfs /dev/sdb1
mkdir -p /fs/sdb1
mount -t xfs -o inode64 /dev/sdb1 /fs/sdb1
for cpu_dir in /sys/devices/system/cpu/cpu[0-9]*
do
online_file="$cpu_dir"/online
[ -f "$online_file" ] && [ "$(cat
"$online_file")" -eq 0 ] && continue
file="$cpu_dir"/cpufreq/scaling_governor
[ -f "$file" ] && echo "performance" >
"$file"
done
echo '[global]
bs=2M
ioengine=sync
iodepth=32
size=68719476736
direct=0
runtime=300
invalidate=1
fallocate=posix
group_reporting
[task_0]
rw=randread
directory=/fs/sdb1
numjobs=8' | fio --output-format=json -
umount /fs/sdb1
Hi Liu Yiding,
Thanks for providing me the script.
It may take a while for me to reproduce the issue, so could you apply
the following debug patch and post the log first?
diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index 65d0a10c76ad..4b272677458f 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -991,6 +991,30 @@ static blk_status_t scsi_init_sgtable(struct request *req,
return BLK_STS_OK;
}
+static void scsi_dump_rq(const struct request *rq)
+{
+ int i = 0;
+ struct req_iterator rq_iter;
+ struct bio_vec bvec;
+ struct bio *bio;
+
+ printk("%s: rq %d(f:%x/%x, seg: %d)\n", __func__,
+ rq->tag, rq->cmd_flags, rq->rq_flags,
+ rq->nr_phys_segments);
+
+ __rq_for_each_bio(bio, rq)
+ printk("bio %d: %hx/%hx %u %llu %u\n", i++,
+ bio->bi_flags, bio->bi_opf,
+ bio->bi_phys_segments,
+ (unsigned long long)bio->bi_iter.bi_sector,
+ bio->bi_iter.bi_size);
+ i = 0;
+ rq_for_each_bvec(bvec, rq, rq_iter)
+ printk("%d: pfn %lu offset %u len %u\n", i++,
+ (unsigned long)page_to_pfn(bvec.bv_page),
+ bvec.bv_offset, bvec.bv_len);
+}
+
/*
* Function: scsi_init_io()
*
@@ -1007,8 +1031,10 @@ blk_status_t scsi_init_io(struct scsi_cmnd *cmd)
struct request *rq = cmd->request;
blk_status_t ret;
- if (WARN_ON_ONCE(!blk_rq_nr_phys_segments(rq)))
+ if (WARN_ON_ONCE(!blk_rq_nr_phys_segments(rq))) {
+ scsi_dump_rq(rq);
return BLK_STS_IOERR;
+ }
ret = scsi_init_sgtable(rq, &cmd->sdb);
if (ret)
Thanks,
Ming
5:59 p.m.
Hi, Ming.
在 2019/6/26 上午11:32, Ming Lei 写道:
On Wed, Jun 26, 2019 at 10:42:52AM +0800, Liu Yiding wrote:
> sorry, something wrong with attatched file.
>
> ------
>
> wipefs -a --force /dev/sdb1
> mkfs -t xfs /dev/sdb1
> mkdir -p /fs/sdb1
> mount -t xfs -o inode64 /dev/sdb1 /fs/sdb1
>
> for cpu_dir in /sys/devices/system/cpu/cpu[0-9]*
> do
> online_file="$cpu_dir"/online
> [ -f "$online_file" ] && [ "$(cat
"$online_file")" -eq 0 ] && continue
>
> file="$cpu_dir"/cpufreq/scaling_governor
> [ -f "$file" ] && echo "performance" >
"$file"
> done
>
> echo '[global]
> bs=2M
> ioengine=sync
> iodepth=32
> size=68719476736
> direct=0
> runtime=300
> invalidate=1
> fallocate=posix
> group_reporting
>
> [task_0]
> rw=randread
> directory=/fs/sdb1
> numjobs=8' | fio --output-format=json -
> umount /fs/sdb1
Hi Liu Yiding,
Thanks for providing me the script.
It may take a while for me to reproduce the issue, so could you apply
the following debug patch and post the log first?
------
2412 [ 112.558155] XFS (sdb1): Ending clean mount
2413 [ 1317.330820] WARNING: CPU: 63 PID: 1495 at
drivers/scsi/scsi_lib.c:1034 scsi_init_io+0x11a/0x2e0
2414 [ 1317.344000] Modules linked in: xfs dm_mod btrfs xor
zstd_decompress zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl
sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_i ntel
kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel mgag200 snd_pcm
ghash_clmulni_intel ttm snd_timer drm_kms_helper ipmi_si ahci snd
aesni_intel syscopyarea libahci ipmi_devi ntf sysfillrect sysimgblt
crypto_simd mei_me fb_sys_fops soundcore ioatdma cryptd ipmi_msghandler
glue_helper pcspkr drm libata mei joydev lpc_ich i2c_i801 dca wmi
acpi_power_meter acpi _pad pcc_cpufreq ip_tables
2415 [ 1317.408135] CPU: 63 PID: 1495 Comm: kworker/63:1H Tainted:
G I 5.2.0-rc5-00001-gc2d55e2 #1
2416 [ 1317.420350] Hardware name: Intel Corporation S2600WTT/S2600WTT,
BIOS SE5C610.86B.01.01.0008.021120151325 02/11/2015
2417 [ 1317.433545] Workqueue: kblockd blk_mq_run_work_fn
2418 [ 1317.440295] RIP: 0010:scsi_init_io+0x11a/0x2e0
2419 [ 1317.446746] Code: be 01 00 00 00 e9 44 ff ff ff 8b 45 70 89 83
08 01 00 00 f6 45 1a 01 75 8a 31 c0 48 83 c4 08 5b 5d 41 5c 41 5d 41 5e
41 5f c3 <0f> 0b 8b 4d 18 8b 55 28 45 31 c9 48 c7 c6 98 da 69 82 48
c7 c7 f0
2420 [ 1317.470863] RSP: 0018:ffffa5fa4f7e7c68 EFLAGS: 00010246
2421 [ 1317.478294] RAX: 000000003d17d1b8 RBX: ffff8ae6cb742690 RCX:
0000000000000000
2422 [ 1317.487878] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
ffff8ae6cb742690
2423 [ 1317.497469] RBP: ffff8ae6cb742580 R08: 0000000000023003 R09:
ffff8ae6cb7426c8
2424 [ 1317.507057] R10: ffffa5fa4c7d7da0 R11: fefefefefefefeff R12:
0000000000000801
2425 [ 1317.516650] R13: 0000000000000000 R14: 0000000000000000 R15:
ffff8ac800bd1800
2426 [ 1317.526253] FS: 0000000000000000(0000)
GS:ffff8b077fac0000(0000) knlGS:0000000000000000
2427 [ 1317.536918] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
2428 [ 1317.544952] CR2: 00007f39a5818004 CR3: 000000343440a006 CR4:
00000000001606e0
2429 [ 1317.554554] Call Trace:
2430 [ 1317.558921] sd_init_command+0x2ee/0xaf0 [sd_mod]
2431 [ 1317.565800] scsi_queue_rq+0x559/0xa80
2432 [ 1317.571607] blk_mq_dispatch_rq_list+0x3da/0x5b0
2433 [ 1317.578375] ? syscall_return_via_sysret+0x10/0x7f
2434 [ 1317.585339] ? __switch_to_asm+0x40/0x70
2435 [ 1317.591321] ? elv_rb_del+0x1f/0x30
2436 [ 1317.596807] ? deadline_remove_request+0x55/0xc0
2437 [ 1317.603560] blk_mq_do_dispatch_sched+0x76/0x120
2438 [ 1317.610302] blk_mq_sched_dispatch_requests+0x100/0x170
2439 [ 1317.617733] __blk_mq_run_hw_queue+0x60/0x130
2440 [ 1317.624200] process_one_work+0x19c/0x3c0
2441 [ 1317.630258] worker_thread+0x3c/0x3b0
2442 [ 1317.635937] ? process_one_work+0x3c0/0x3c0
2443 [ 1317.642183] kthread+0x11e/0x140
2444 [ 1317.647346] ? kthread_park+0x90/0x90
2445 [ 1317.652986] ret_from_fork+0x35/0x40
2446 [ 1317.658523] ---[ end trace 61026ff97261da6a ]---
2447 [ 1317.665327] scsi_dump_rq: rq 6(f:801/23003, seg: 0)
2448 [ 1317.671601] bio 0: d602/801 0 1024971192 0
2449 [ 1317.676991] print_req_error: I/O error, dev sdb, sector
1024971192 flags 801
2450 [ 1317.685722] scsi_dump_rq: rq 8(f:801/23003, seg: 0)
2451 [ 1317.691994] bio 0: d602/801 0 1033359800 0
2452 [ 1317.697383] print_req_error: I/O error, dev sdb, sector
1033359800 flags 801
2453 [ 1317.914433] XFS (sdb1): writeback error on sector 1024971192
2454 [ 1318.113497] XFS (sdb1): writeback error on sector 1033359800
------
diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index 65d0a10c76ad..4b272677458f 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -991,6 +991,30 @@ static blk_status_t scsi_init_sgtable(struct request *req,
return BLK_STS_OK;
}
+static void scsi_dump_rq(const struct request *rq)
+{
+ int i = 0;
+ struct req_iterator rq_iter;
+ struct bio_vec bvec;
+ struct bio *bio;
+
+ printk("%s: rq %d(f:%x/%x, seg: %d)\n", __func__,
+ rq->tag, rq->cmd_flags, rq->rq_flags,
+ rq->nr_phys_segments);
+
+ __rq_for_each_bio(bio, rq)
+ printk("bio %d: %hx/%hx %u %llu %u\n", i++,
+ bio->bi_flags, bio->bi_opf,
+ bio->bi_phys_segments,
+ (unsigned long long)bio->bi_iter.bi_sector,
+ bio->bi_iter.bi_size);
+ i = 0;
+ rq_for_each_bvec(bvec, rq, rq_iter)
+ printk("%d: pfn %lu offset %u len %u\n", i++,
+ (unsigned long)page_to_pfn(bvec.bv_page),
+ bvec.bv_offset, bvec.bv_len);
+}
+
/*
* Function: scsi_init_io()
*
@@ -1007,8 +1031,10 @@ blk_status_t scsi_init_io(struct scsi_cmnd *cmd)
struct request *rq = cmd->request;
blk_status_t ret;
- if (WARN_ON_ONCE(!blk_rq_nr_phys_segments(rq)))
+ if (WARN_ON_ONCE(!blk_rq_nr_phys_segments(rq))) {
+ scsi_dump_rq(rq);
return BLK_STS_IOERR;
+ }
ret = scsi_init_sgtable(rq, &cmd->sdb);
if (ret)
Thanks,
Ming
8:41 p.m.
On Thu, Jun 27, 2019 at 08:59:51AM +0800, Liu Yiding wrote:
Hi, Ming.
在 2019/6/26 上午11:32, Ming Lei 写道:
> On Wed, Jun 26, 2019 at 10:42:52AM +0800, Liu Yiding wrote:
> > sorry, something wrong with attatched file.
> >
> > ------
> >
> > wipefs -a --force /dev/sdb1
> > mkfs -t xfs /dev/sdb1
> > mkdir -p /fs/sdb1
> > mount -t xfs -o inode64 /dev/sdb1 /fs/sdb1
> >
> > for cpu_dir in /sys/devices/system/cpu/cpu[0-9]*
> > do
> > online_file="$cpu_dir"/online
> > [ -f "$online_file" ] && [ "$(cat
"$online_file")" -eq 0 ] && continue
> >
> > file="$cpu_dir"/cpufreq/scaling_governor
> > [ -f "$file" ] && echo "performance" >
"$file"
> > done
> >
> > echo '[global]
> > bs=2M
> > ioengine=sync
> > iodepth=32
> > size=68719476736
> > direct=0
> > runtime=300
> > invalidate=1
> > fallocate=posix
> > group_reporting
> >
> > [task_0]
> > rw=randread
> > directory=/fs/sdb1
> > numjobs=8' | fio --output-format=json -
> > umount /fs/sdb1
> Hi Liu Yiding,
>
> Thanks for providing me the script.
>
> It may take a while for me to reproduce the issue, so could you apply
> the following debug patch and post the log first?
------
2412 [ 112.558155] XFS (sdb1): Ending clean mount
2413 [ 1317.330820] WARNING: CPU: 63 PID: 1495 at
drivers/scsi/scsi_lib.c:1034 scsi_init_io+0x11a/0x2e0
2414 [ 1317.344000] Modules linked in: xfs dm_mod btrfs xor zstd_decompress
zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac
x86_pkg_temp_thermal intel_powerclamp coretemp kvm_i ntel kvm irqbypass
crct10dif_pclmul crc32_pclmul crc32c_intel mgag200 snd_pcm
ghash_clmulni_intel ttm snd_timer drm_kms_helper ipmi_si ahci snd
aesni_intel syscopyarea libahci ipmi_devi ntf sysfillrect sysimgblt
crypto_simd mei_me fb_sys_fops soundcore ioatdma cryptd ipmi_msghandler
glue_helper pcspkr drm libata mei joydev lpc_ich i2c_i801 dca wmi
acpi_power_meter acpi _pad pcc_cpufreq ip_tables
2415 [ 1317.408135] CPU: 63 PID: 1495 Comm: kworker/63:1H Tainted:
G I 5.2.0-rc5-00001-gc2d55e2 #1
2416 [ 1317.420350] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
SE5C610.86B.01.01.0008.021120151325 02/11/2015
2417 [ 1317.433545] Workqueue: kblockd blk_mq_run_work_fn
2418 [ 1317.440295] RIP: 0010:scsi_init_io+0x11a/0x2e0
2419 [ 1317.446746] Code: be 01 00 00 00 e9 44 ff ff ff 8b 45 70 89 83 08 01
00 00 f6 45 1a 01 75 8a 31 c0 48 83 c4 08 5b 5d 41 5c 41 5d 41 5e 41 5f c3
<0f> 0b 8b 4d 18 8b 55 28 45 31 c9 48 c7 c6 98 da 69 82 48 c7 c7 f0
2420 [ 1317.470863] RSP: 0018:ffffa5fa4f7e7c68 EFLAGS: 00010246
2421 [ 1317.478294] RAX: 000000003d17d1b8 RBX: ffff8ae6cb742690 RCX:
0000000000000000
2422 [ 1317.487878] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
ffff8ae6cb742690
2423 [ 1317.497469] RBP: ffff8ae6cb742580 R08: 0000000000023003 R09:
ffff8ae6cb7426c8
2424 [ 1317.507057] R10: ffffa5fa4c7d7da0 R11: fefefefefefefeff R12:
0000000000000801
2425 [ 1317.516650] R13: 0000000000000000 R14: 0000000000000000 R15:
ffff8ac800bd1800
2426 [ 1317.526253] FS: 0000000000000000(0000) GS:ffff8b077fac0000(0000)
knlGS:0000000000000000
2427 [ 1317.536918] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
2428 [ 1317.544952] CR2: 00007f39a5818004 CR3: 000000343440a006 CR4:
00000000001606e0
2429 [ 1317.554554] Call Trace:
2430 [ 1317.558921] sd_init_command+0x2ee/0xaf0 [sd_mod]
2431 [ 1317.565800] scsi_queue_rq+0x559/0xa80
2432 [ 1317.571607] blk_mq_dispatch_rq_list+0x3da/0x5b0
2433 [ 1317.578375] ? syscall_return_via_sysret+0x10/0x7f
2434 [ 1317.585339] ? __switch_to_asm+0x40/0x70
2435 [ 1317.591321] ? elv_rb_del+0x1f/0x30
2436 [ 1317.596807] ? deadline_remove_request+0x55/0xc0
2437 [ 1317.603560] blk_mq_do_dispatch_sched+0x76/0x120
2438 [ 1317.610302] blk_mq_sched_dispatch_requests+0x100/0x170
2439 [ 1317.617733] __blk_mq_run_hw_queue+0x60/0x130
2440 [ 1317.624200] process_one_work+0x19c/0x3c0
2441 [ 1317.630258] worker_thread+0x3c/0x3b0
2442 [ 1317.635937] ? process_one_work+0x3c0/0x3c0
2443 [ 1317.642183] kthread+0x11e/0x140
2444 [ 1317.647346] ? kthread_park+0x90/0x90
2445 [ 1317.652986] ret_from_fork+0x35/0x40
2446 [ 1317.658523] ---[ end trace 61026ff97261da6a ]---
2447 [ 1317.665327] scsi_dump_rq: rq 6(f:801/23003, seg: 0)
2448 [ 1317.671601] bio 0: d602/801 0 1024971192 0
2449 [ 1317.676991] print_req_error: I/O error, dev sdb, sector 1024971192
flags 801
2450 [ 1317.685722] scsi_dump_rq: rq 8(f:801/23003, seg: 0)
2451 [ 1317.691994] bio 0: d602/801 0 1033359800 0
The above line shows the bio is zero-size, and this bio is submitted
from bio, since the bio->bi_flags indicates that BIO_CLONED isn't set.
Please apply the following debug patch, and we may see where is wrong.
diff --git a/block/blk-core.c b/block/blk-core.c
index 8340f69670d8..86bb61b0df83 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -1125,6 +1125,7 @@ blk_qc_t submit_bio(struct bio *bio)
bio_devname(bio, b), count);
}
}
+ WARN_ON(bio->bi_iter.bi_size == 0 && bio_op(bio) == REQ_OP_WRITE);
return generic_make_request(bio);
}
thanks,
Ming
5:59 p.m.
在 2019/6/27 上午11:41, Ming Lei 写道:
On Thu, Jun 27, 2019 at 08:59:51AM +0800, Liu Yiding wrote:
> Hi, Ming.
>
> 在 2019/6/26 上午11:32, Ming Lei 写道:
>> On Wed, Jun 26, 2019 at 10:42:52AM +0800, Liu Yiding wrote:
>>> sorry, something wrong with attatched file.
>>>
>>> ------
>>>
>>> wipefs -a --force /dev/sdb1
>>> mkfs -t xfs /dev/sdb1
>>> mkdir -p /fs/sdb1
>>> mount -t xfs -o inode64 /dev/sdb1 /fs/sdb1
>>>
>>> for cpu_dir in /sys/devices/system/cpu/cpu[0-9]*
>>> do
>>> online_file="$cpu_dir"/online
>>> [ -f "$online_file" ] && [ "$(cat
"$online_file")" -eq 0 ] && continue
>>>
>>> file="$cpu_dir"/cpufreq/scaling_governor
>>> [ -f "$file" ] && echo "performance" >
"$file"
>>> done
>>>
>>> echo '[global]
>>> bs=2M
>>> ioengine=sync
>>> iodepth=32
>>> size=68719476736
>>> direct=0
>>> runtime=300
>>> invalidate=1
>>> fallocate=posix
>>> group_reporting
>>>
>>> [task_0]
>>> rw=randread
>>> directory=/fs/sdb1
>>> numjobs=8' | fio --output-format=json -
>>> umount /fs/sdb1
>> Hi Liu Yiding,
>>
>> Thanks for providing me the script.
>>
>> It may take a while for me to reproduce the issue, so could you apply
>> the following debug patch and post the log first?
>
>
> ------
>
> 2412 [ 112.558155] XFS (sdb1): Ending clean mount
> 2413 [ 1317.330820] WARNING: CPU: 63 PID: 1495 at
> drivers/scsi/scsi_lib.c:1034 scsi_init_io+0x11a/0x2e0
> 2414 [ 1317.344000] Modules linked in: xfs dm_mod btrfs xor zstd_decompress
> zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac
> x86_pkg_temp_thermal intel_powerclamp coretemp kvm_i ntel kvm irqbypass
> crct10dif_pclmul crc32_pclmul crc32c_intel mgag200 snd_pcm
> ghash_clmulni_intel ttm snd_timer drm_kms_helper ipmi_si ahci snd
> aesni_intel syscopyarea libahci ipmi_devi ntf sysfillrect sysimgblt
> crypto_simd mei_me fb_sys_fops soundcore ioatdma cryptd ipmi_msghandler
> glue_helper pcspkr drm libata mei joydev lpc_ich i2c_i801 dca wmi
> acpi_power_meter acpi _pad pcc_cpufreq ip_tables
> 2415 [ 1317.408135] CPU: 63 PID: 1495 Comm: kworker/63:1H Tainted:
> G I 5.2.0-rc5-00001-gc2d55e2 #1
> 2416 [ 1317.420350] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
> SE5C610.86B.01.01.0008.021120151325 02/11/2015
> 2417 [ 1317.433545] Workqueue: kblockd blk_mq_run_work_fn
> 2418 [ 1317.440295] RIP: 0010:scsi_init_io+0x11a/0x2e0
> 2419 [ 1317.446746] Code: be 01 00 00 00 e9 44 ff ff ff 8b 45 70 89 83 08 01
> 00 00 f6 45 1a 01 75 8a 31 c0 48 83 c4 08 5b 5d 41 5c 41 5d 41 5e 41 5f c3
> <0f> 0b 8b 4d 18 8b 55 28 45 31 c9 48 c7 c6 98 da 69 82 48 c7 c7 f0
> 2420 [ 1317.470863] RSP: 0018:ffffa5fa4f7e7c68 EFLAGS: 00010246
> 2421 [ 1317.478294] RAX: 000000003d17d1b8 RBX: ffff8ae6cb742690 RCX:
> 0000000000000000
> 2422 [ 1317.487878] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
> ffff8ae6cb742690
> 2423 [ 1317.497469] RBP: ffff8ae6cb742580 R08: 0000000000023003 R09:
> ffff8ae6cb7426c8
> 2424 [ 1317.507057] R10: ffffa5fa4c7d7da0 R11: fefefefefefefeff R12:
> 0000000000000801
> 2425 [ 1317.516650] R13: 0000000000000000 R14: 0000000000000000 R15:
> ffff8ac800bd1800
> 2426 [ 1317.526253] FS: 0000000000000000(0000) GS:ffff8b077fac0000(0000)
> knlGS:0000000000000000
> 2427 [ 1317.536918] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> 2428 [ 1317.544952] CR2: 00007f39a5818004 CR3: 000000343440a006 CR4:
> 00000000001606e0
> 2429 [ 1317.554554] Call Trace:
> 2430 [ 1317.558921] sd_init_command+0x2ee/0xaf0 [sd_mod]
> 2431 [ 1317.565800] scsi_queue_rq+0x559/0xa80
> 2432 [ 1317.571607] blk_mq_dispatch_rq_list+0x3da/0x5b0
> 2433 [ 1317.578375] ? syscall_return_via_sysret+0x10/0x7f
> 2434 [ 1317.585339] ? __switch_to_asm+0x40/0x70
> 2435 [ 1317.591321] ? elv_rb_del+0x1f/0x30
> 2436 [ 1317.596807] ? deadline_remove_request+0x55/0xc0
> 2437 [ 1317.603560] blk_mq_do_dispatch_sched+0x76/0x120
> 2438 [ 1317.610302] blk_mq_sched_dispatch_requests+0x100/0x170
> 2439 [ 1317.617733] __blk_mq_run_hw_queue+0x60/0x130
> 2440 [ 1317.624200] process_one_work+0x19c/0x3c0
> 2441 [ 1317.630258] worker_thread+0x3c/0x3b0
> 2442 [ 1317.635937] ? process_one_work+0x3c0/0x3c0
> 2443 [ 1317.642183] kthread+0x11e/0x140
> 2444 [ 1317.647346] ? kthread_park+0x90/0x90
> 2445 [ 1317.652986] ret_from_fork+0x35/0x40
> 2446 [ 1317.658523] ---[ end trace 61026ff97261da6a ]---
> 2447 [ 1317.665327] scsi_dump_rq: rq 6(f:801/23003, seg: 0)
> 2448 [ 1317.671601] bio 0: d602/801 0 1024971192 0
> 2449 [ 1317.676991] print_req_error: I/O error, dev sdb, sector 1024971192
> flags 801
> 2450 [ 1317.685722] scsi_dump_rq: rq 8(f:801/23003, seg: 0)
> 2451 [ 1317.691994] bio 0: d602/801 0 1033359800 0
The above line shows the bio is zero-size, and this bio is submitted
from bio, since the bio->bi_flags indicates that BIO_CLONED isn't set.
Please apply the following debug patch, and we may see where is wrong.
diff --git a/block/blk-core.c b/block/blk-core.c
index 8340f69670d8..86bb61b0df83 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -1125,6 +1125,7 @@ blk_qc_t submit_bio(struct bio *bio)
bio_devname(bio, b), count);
}
}
+ WARN_ON(bio->bi_iter.bi_size == 0 && bio_op(bio) == REQ_OP_WRITE);
return generic_make_request(bio);
Sorry for delay,
------
[ 116.619136] 2019-06-29 16:12:41 mkdir -p /fs/sdb1
[ 116.619138]
[ 116.628743] 2019-06-29 16:12:41 mount -t xfs -o inode64 /dev/sdb1
/fs/sdb1
[ 116.628745]
[ 116.634699] SGI XFS with ACLs, security attributes, realtime, scrub,
debug enabled
[ 116.653110] XFS (sdb1): Mounting V5 Filesystem
[ 116.670081] XFS (sdb1): Ending clean mount
[ 690.106779] WARNING: CPU: 53 PID: 1477 at block/blk-core.c:1128
submit_bio+0x93/0x150
[ 690.117972] Modules linked in: xfs dm_mod btrfs xor zstd_decompress
zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac
x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
mgag200 crct10dif_pclmul crc32_pclmul ttm crc32c_intel snd_pcm
ghash_clmulni_intel drm_kms_helper snd_timer ahci syscopyarea
sysfillrect snd aesni_intel sysimgblt libahci ipmi_si soundcore
fb_sys_fops mei_me crypto_simd ipmi_devintf ioatdma cryptd pcspkr drm
ipmi_msghandler glue_helper libata mei joydev lpc_ich i2c_i801 dca wmi
acpi_power_meter acpi_pad pcc_cpufreq ip_tables
[ 690.176967] CPU: 53 PID: 1477 Comm: fio Tainted: G W I
5.2.0-rc5-00002-g9fa09bd #1
[ 690.187198] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
SE5C610.86B.01.01.0008.021120151325 02/11/2015
[ 690.199375] RIP: 0010:submit_bio+0x93/0x150
[ 690.204583] Code: ee 74 45 83 e1 01 74 26 65 48 01 35 bf 39 3f 45 8b
0d ad 96 c4 01 85 c9 75 56 8b 43 28 85 c0 75 b3 0f b6 53 10 83 fa 01 75
aa <0f> 0b eb a6 65 48 8b 14 25 c0 6b 01 00 65 48 01 35 88 39 3f 45 48
[ 690.226668] RSP: 0018:ffffb2d90efc3b40 EFLAGS: 00010246
[ 690.233072] RAX: 0000000000000000 RBX: ffff9654c6ad9f40 RCX:
0000000000000801
[ 690.241617] RDX: 0000000000000001 RSI: ffff9654c6ad9f00 RDI:
ffff9654c6ad9f40
[ 690.250146] RBP: 0000000000000000 R08: 0000000000000002 R09:
0000000000000000
[ 690.258667] R10: 00000000000303a0 R11: ffff96757ffd4000 R12:
ffffb2d90efc3e08
[ 690.267183] R13: ffff9654c6ad9f00 R14: ffffb2d90efc3e08 R15:
0000000000000000
[ 690.275696] FS: 00007fa8e005da80(0000) GS:ffff9654ffac0000(0000)
knlGS:0000000000000000
[ 690.285282] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 690.292252] CR2: 00007fa8cc05b3c0 CR3: 0000001fc8518001 CR4:
00000000001606e0
[ 690.300770] Call Trace:
[ 690.304123] xfs_submit_ioend+0x84/0x1e0 [xfs]
[ 690.309668] xfs_do_writepage+0x246/0x5a0 [xfs]
[ 690.315279] write_cache_pages+0x16b/0x440
[ 690.320420] ? xfs_vm_releasepage+0x80/0x80 [xfs]
[ 690.326238] xfs_vm_writepages+0x6b/0xa0 [xfs]
[ 690.331742] do_writepages+0x4b/0xf0
[ 690.336275] ? iomap_file_buffered_write+0x62/0x90
[ 690.342162] ? iomap_write_end+0x240/0x240
[ 690.347295] ? xfs_iunlock+0x165/0x220 [xfs]
[ 690.352595] ? __filemap_fdatawrite_range+0xc1/0x100
[ 690.358673] __filemap_fdatawrite_range+0xc1/0x100
[ 690.364571] ? __switch_to_asm+0x34/0x70
[ 690.369484] file_write_and_wait_range+0x4c/0xa0
[ 690.375198] xfs_file_fsync+0x5d/0x230 [xfs]
[ 690.380499] do_fsync+0x38/0x60
[ 690.384534] __x64_sys_fsync+0x10/0x20
[ 690.389243] do_syscall_64+0x5b/0x1e0
[ 690.393857] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 690.400020] RIP: 0033:0x7fa8cbe4f70d
[ 690.404533] Code: bc 20 00 00 75 10 b8 4a 00 00 00 0f 05 48 3d 01 f0
ff ff 73 31 c3 48 83 ec 08 e8 6e f7 ff ff 48 89 04 24 b8 4a 00 00 00 0f
05 <48> 8b 3c 24 48 89 c2 e8 b7 f7 ff ff 48 89 d0 48 83 c4 08 48 3d 01
[ 690.426581] RSP: 002b:00007ffed6954160 EFLAGS: 00000293 ORIG_RAX:
000000000000004a
[ 690.435583] RAX: ffffffffffffffda RBX: 00005573363790f0 RCX:
00007fa8cbe4f70d
[ 690.444102] RDX: 0000000000200000 RSI: 00005573363790f0 RDI:
0000000000000003
[ 690.452615] RBP: 00007fa8b337f418 R08: 0000000000000008 R09:
0000000000000001
[ 690.461124] R10: 00000000897b0000 R11: 0000000000000293 R12:
0000000000000000
[ 690.469631] R13: 0000001000000000 R14: 00007fa8c81dda50 R15:
0000000000000000
[ 690.478130] ---[ end trace 90dd4ee6d5cdfcb8 ]---
[ 690.483850] WARNING: CPU: 53 PID: 1562 at
drivers/scsi/scsi_lib.c:1034 scsi_init_io+0x11a/0x2e0
[ 690.495957] Modules linked in: xfs dm_mod btrfs xor zstd_decompress
zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac
x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
mgag200 crct10dif_pclmul crc32_pclmul ttm crc32c_intel snd_pcm
ghash_clmulni_intel drm_kms_helper snd_timer ahci syscopyarea
sysfillrect snd aesni_intel sysimgblt libahci ipmi_si soundcore
fb_sys_fops mei_me crypto_simd ipmi_devintf ioatdma cryptd pcspkr drm
ipmi_msghandler glue_helper libata mei joydev lpc_ich i2c_i801 dca wmi
acpi_power_meter acpi_pad pcc_cpufreq ip_tables
[ 690.555277] CPU: 53 PID: 1562 Comm: kworker/53:1H Tainted: G W
I 5.2.0-rc5-00002-g9fa09bd #1
[ 690.566507] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
SE5C610.86B.01.01.0008.021120151325 02/11/2015
[ 690.578729] Workqueue: kblockd blk_mq_run_work_fn
[ 690.584527] RIP: 0010:scsi_init_io+0x11a/0x2e0
[ 690.590031] Code: be 01 00 00 00 e9 44 ff ff ff 8b 45 70 89 83 08 01
00 00 f6 45 1a 01 75 8a 31 c0 48 83 c4 08 5b 5d 41 5c 41 5d 41 5e 41 5f
c3 <0f> 0b 8b 4d 18 8b 55 28 45 31 c9 48 c7 c6 98 da 89 bb 48 c7 c7 f0
[ 690.612092] RSP: 0018:ffffb2d90f513c68 EFLAGS: 00010246
[ 690.618484] RAX: 000000001d0c2838 RBX: ffff9654be21a9d0 RCX:
0000000000000000
[ 690.627018] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
ffff9654be21a9d0
[ 690.635558] RBP: ffff9654be21a8c0 R08: 0000000000023003 R09:
ffff9654be21aa08
[ 690.644080] R10: 0000000000000000 R11: fefefefefefefeff R12:
0000000000000801
[ 690.652615] R13: 0000000000000000 R14: 0000000000000000 R15:
ffff967572532000
[ 690.661142] FS: 0000000000000000(0000) GS:ffff9654ffac0000(0000)
knlGS:0000000000000000
[ 690.670748] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 690.677760] CR2: 00007fa8cc05b3c0 CR3: 0000001fc8518001 CR4:
00000000001606e0
[ 690.686268] Call Trace:
[ 690.689574] sd_init_command+0x2ee/0xaf0 [sd_mod]
[ 690.695387] scsi_queue_rq+0x559/0xa80
[ 690.700148] blk_mq_dispatch_rq_list+0x3da/0x5b0
[ 690.705889] ? syscall_return_via_sysret+0x10/0x7f
[ 690.711840] ? __switch_to_asm+0x40/0x70
[ 690.716827] ? elv_rb_del+0x1f/0x30
[ 690.721275] ? deadline_remove_request+0x55/0xc0
[ 690.726966] blk_mq_do_dispatch_sched+0x76/0x120
[ 690.732657] blk_mq_sched_dispatch_requests+0x100/0x170
[ 690.739030] __blk_mq_run_hw_queue+0x60/0x130
[ 690.744454] process_one_work+0x19c/0x3c0
[ 690.749484] worker_thread+0x3c/0x3b0
[ 690.754097] ? process_one_work+0x3c0/0x3c0
[ 690.759304] kthread+0x11e/0x140
[ 690.763452] ? kthread_park+0x90/0x90
[ 690.768092] ret_from_fork+0x35/0x40
[ 690.772613] ---[ end trace 90dd4ee6d5cdfcb9 ]---
[ 690.778316] scsi_dump_rq: rq 6(f:801/23003, seg: 0)
[ 690.784295] bio 0: d602/801 0 487335992 0
[ 690.789280] print_req_error: I/O error, dev sdb, sector 487335992
flags 801
[ 690.923742] XFS (sdb1): writeback error on sector 487335992
[ 744.116327] fio: pid=0, err=5/file:filesetup.c:190, func=fsync,
error=Input/output error
[ 744.116330]
------
}
thanks,
Ming
7:47 p.m.
Hi Yiding,
Thanks for your test & providing the log.
On Mon, Jul 01, 2019 at 08:59:06AM +0800, Liu Yiding wrote:
在 2019/6/27 上午11:41, Ming Lei 写道:
> On Thu, Jun 27, 2019 at 08:59:51AM +0800, Liu Yiding wrote:
> > Hi, Ming.
> >
> > 在 2019/6/26 上午11:32, Ming Lei 写道:
> > > On Wed, Jun 26, 2019 at 10:42:52AM +0800, Liu Yiding wrote:
> > > > sorry, something wrong with attatched file.
> > > >
> > > > ------
> > > >
> > > > wipefs -a --force /dev/sdb1
> > > > mkfs -t xfs /dev/sdb1
> > > > mkdir -p /fs/sdb1
> > > > mount -t xfs -o inode64 /dev/sdb1 /fs/sdb1
> > > >
> > > > for cpu_dir in /sys/devices/system/cpu/cpu[0-9]*
> > > > do
> > > > online_file="$cpu_dir"/online
> > > > [ -f "$online_file" ] && [ "$(cat
"$online_file")" -eq 0 ] && continue
> > > >
> > > > file="$cpu_dir"/cpufreq/scaling_governor
> > > > [ -f "$file" ] && echo
"performance" > "$file"
> > > > done
> > > >
> > > > echo '[global]
> > > > bs=2M
> > > > ioengine=sync
> > > > iodepth=32
> > > > size=68719476736
> > > > direct=0
> > > > runtime=300
> > > > invalidate=1
> > > > fallocate=posix
> > > > group_reporting
> > > >
> > > > [task_0]
> > > > rw=randread
> > > > directory=/fs/sdb1
> > > > numjobs=8' | fio --output-format=json -
> > > > umount /fs/sdb1
> > > Hi Liu Yiding,
> > >
> > > Thanks for providing me the script.
> > >
> > > It may take a while for me to reproduce the issue, so could you apply
> > > the following debug patch and post the log first?
> >
> >
> > ------
> >
> > 2412 [ 112.558155] XFS (sdb1): Ending clean mount
> > 2413 [ 1317.330820] WARNING: CPU: 63 PID: 1495 at
> > drivers/scsi/scsi_lib.c:1034 scsi_init_io+0x11a/0x2e0
> > 2414 [ 1317.344000] Modules linked in: xfs dm_mod btrfs xor zstd_decompress
> > zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac
> > x86_pkg_temp_thermal intel_powerclamp coretemp kvm_i ntel kvm irqbypass
> > crct10dif_pclmul crc32_pclmul crc32c_intel mgag200 snd_pcm
> > ghash_clmulni_intel ttm snd_timer drm_kms_helper ipmi_si ahci snd
> > aesni_intel syscopyarea libahci ipmi_devi ntf sysfillrect sysimgblt
> > crypto_simd mei_me fb_sys_fops soundcore ioatdma cryptd ipmi_msghandler
> > glue_helper pcspkr drm libata mei joydev lpc_ich i2c_i801 dca wmi
> > acpi_power_meter acpi _pad pcc_cpufreq ip_tables
> > 2415 [ 1317.408135] CPU: 63 PID: 1495 Comm: kworker/63:1H Tainted:
> > G I 5.2.0-rc5-00001-gc2d55e2 #1
> > 2416 [ 1317.420350] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
> > SE5C610.86B.01.01.0008.021120151325 02/11/2015
> > 2417 [ 1317.433545] Workqueue: kblockd blk_mq_run_work_fn
> > 2418 [ 1317.440295] RIP: 0010:scsi_init_io+0x11a/0x2e0
> > 2419 [ 1317.446746] Code: be 01 00 00 00 e9 44 ff ff ff 8b 45 70 89 83 08 01
> > 00 00 f6 45 1a 01 75 8a 31 c0 48 83 c4 08 5b 5d 41 5c 41 5d 41 5e 41 5f c3
> > <0f> 0b 8b 4d 18 8b 55 28 45 31 c9 48 c7 c6 98 da 69 82 48 c7 c7 f0
> > 2420 [ 1317.470863] RSP: 0018:ffffa5fa4f7e7c68 EFLAGS: 00010246
> > 2421 [ 1317.478294] RAX: 000000003d17d1b8 RBX: ffff8ae6cb742690 RCX:
> > 0000000000000000
> > 2422 [ 1317.487878] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
> > ffff8ae6cb742690
> > 2423 [ 1317.497469] RBP: ffff8ae6cb742580 R08: 0000000000023003 R09:
> > ffff8ae6cb7426c8
> > 2424 [ 1317.507057] R10: ffffa5fa4c7d7da0 R11: fefefefefefefeff R12:
> > 0000000000000801
> > 2425 [ 1317.516650] R13: 0000000000000000 R14: 0000000000000000 R15:
> > ffff8ac800bd1800
> > 2426 [ 1317.526253] FS: 0000000000000000(0000) GS:ffff8b077fac0000(0000)
> > knlGS:0000000000000000
> > 2427 [ 1317.536918] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > 2428 [ 1317.544952] CR2: 00007f39a5818004 CR3: 000000343440a006 CR4:
> > 00000000001606e0
> > 2429 [ 1317.554554] Call Trace:
> > 2430 [ 1317.558921] sd_init_command+0x2ee/0xaf0 [sd_mod]
> > 2431 [ 1317.565800] scsi_queue_rq+0x559/0xa80
> > 2432 [ 1317.571607] blk_mq_dispatch_rq_list+0x3da/0x5b0
> > 2433 [ 1317.578375] ? syscall_return_via_sysret+0x10/0x7f
> > 2434 [ 1317.585339] ? __switch_to_asm+0x40/0x70
> > 2435 [ 1317.591321] ? elv_rb_del+0x1f/0x30
> > 2436 [ 1317.596807] ? deadline_remove_request+0x55/0xc0
> > 2437 [ 1317.603560] blk_mq_do_dispatch_sched+0x76/0x120
> > 2438 [ 1317.610302] blk_mq_sched_dispatch_requests+0x100/0x170
> > 2439 [ 1317.617733] __blk_mq_run_hw_queue+0x60/0x130
> > 2440 [ 1317.624200] process_one_work+0x19c/0x3c0
> > 2441 [ 1317.630258] worker_thread+0x3c/0x3b0
> > 2442 [ 1317.635937] ? process_one_work+0x3c0/0x3c0
> > 2443 [ 1317.642183] kthread+0x11e/0x140
> > 2444 [ 1317.647346] ? kthread_park+0x90/0x90
> > 2445 [ 1317.652986] ret_from_fork+0x35/0x40
> > 2446 [ 1317.658523] ---[ end trace 61026ff97261da6a ]---
> > 2447 [ 1317.665327] scsi_dump_rq: rq 6(f:801/23003, seg: 0)
> > 2448 [ 1317.671601] bio 0: d602/801 0 1024971192 0
> > 2449 [ 1317.676991] print_req_error: I/O error, dev sdb, sector 1024971192
> > flags 801
> > 2450 [ 1317.685722] scsi_dump_rq: rq 8(f:801/23003, seg: 0)
> > 2451 [ 1317.691994] bio 0: d602/801 0 1033359800 0
> The above line shows the bio is zero-size, and this bio is submitted
> from bio, since the bio->bi_flags indicates that BIO_CLONED isn't set.
>
> Please apply the following debug patch, and we may see where is wrong.
>
> diff --git a/block/blk-core.c b/block/blk-core.c
> index 8340f69670d8..86bb61b0df83 100644
> --- a/block/blk-core.c
> +++ b/block/blk-core.c
> @@ -1125,6 +1125,7 @@ blk_qc_t submit_bio(struct bio *bio)
> bio_devname(bio, b), count);
> }
> }
> + WARN_ON(bio->bi_iter.bi_size == 0 && bio_op(bio) == REQ_OP_WRITE);
> return generic_make_request(bio);
Sorry for delay,
------
[ 116.619136] 2019-06-29 16:12:41 mkdir -p /fs/sdb1
[ 116.619138]
[ 116.628743] 2019-06-29 16:12:41 mount -t xfs -o inode64 /dev/sdb1
/fs/sdb1
[ 116.628745]
[ 116.634699] SGI XFS with ACLs, security attributes, realtime, scrub,
debug enabled
[ 116.653110] XFS (sdb1): Mounting V5 Filesystem
[ 116.670081] XFS (sdb1): Ending clean mount
[ 690.106779] WARNING: CPU: 53 PID: 1477 at block/blk-core.c:1128
submit_bio+0x93/0x150
[ 690.117972] Modules linked in: xfs dm_mod btrfs xor zstd_decompress
zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac
x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
mgag200 crct10dif_pclmul crc32_pclmul ttm crc32c_intel snd_pcm
ghash_clmulni_intel drm_kms_helper snd_timer ahci syscopyarea sysfillrect
snd aesni_intel sysimgblt libahci ipmi_si soundcore fb_sys_fops mei_me
crypto_simd ipmi_devintf ioatdma cryptd pcspkr drm ipmi_msghandler
glue_helper libata mei joydev lpc_ich i2c_i801 dca wmi acpi_power_meter
acpi_pad pcc_cpufreq ip_tables
[ 690.176967] CPU: 53 PID: 1477 Comm: fio Tainted: G W I
5.2.0-rc5-00002-g9fa09bd #1
[ 690.187198] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
SE5C610.86B.01.01.0008.021120151325 02/11/2015
[ 690.199375] RIP: 0010:submit_bio+0x93/0x150
[ 690.204583] Code: ee 74 45 83 e1 01 74 26 65 48 01 35 bf 39 3f 45 8b 0d
ad 96 c4 01 85 c9 75 56 8b 43 28 85 c0 75 b3 0f b6 53 10 83 fa 01 75 aa <0f>
0b eb a6 65 48 8b 14 25 c0 6b 01 00 65 48 01 35 88 39 3f 45 48
[ 690.226668] RSP: 0018:ffffb2d90efc3b40 EFLAGS: 00010246
[ 690.233072] RAX: 0000000000000000 RBX: ffff9654c6ad9f40 RCX:
0000000000000801
[ 690.241617] RDX: 0000000000000001 RSI: ffff9654c6ad9f00 RDI:
ffff9654c6ad9f40
[ 690.250146] RBP: 0000000000000000 R08: 0000000000000002 R09:
0000000000000000
[ 690.258667] R10: 00000000000303a0 R11: ffff96757ffd4000 R12:
ffffb2d90efc3e08
[ 690.267183] R13: ffff9654c6ad9f00 R14: ffffb2d90efc3e08 R15:
0000000000000000
[ 690.275696] FS: 00007fa8e005da80(0000) GS:ffff9654ffac0000(0000)
knlGS:0000000000000000
[ 690.285282] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 690.292252] CR2: 00007fa8cc05b3c0 CR3: 0000001fc8518001 CR4:
00000000001606e0
[ 690.300770] Call Trace:
[ 690.304123] xfs_submit_ioend+0x84/0x1e0 [xfs]
[ 690.309668] xfs_do_writepage+0x246/0x5a0 [xfs]
[ 690.315279] write_cache_pages+0x16b/0x440
[ 690.320420] ? xfs_vm_releasepage+0x80/0x80 [xfs]
[ 690.326238] xfs_vm_writepages+0x6b/0xa0 [xfs]
[ 690.331742] do_writepages+0x4b/0xf0
[ 690.336275] ? iomap_file_buffered_write+0x62/0x90
[ 690.342162] ? iomap_write_end+0x240/0x240
[ 690.347295] ? xfs_iunlock+0x165/0x220 [xfs]
[ 690.352595] ? __filemap_fdatawrite_range+0xc1/0x100
[ 690.358673] __filemap_fdatawrite_range+0xc1/0x100
[ 690.364571] ? __switch_to_asm+0x34/0x70
[ 690.369484] file_write_and_wait_range+0x4c/0xa0
[ 690.375198] xfs_file_fsync+0x5d/0x230 [xfs]
[ 690.380499] do_fsync+0x38/0x60
[ 690.384534] __x64_sys_fsync+0x10/0x20
[ 690.389243] do_syscall_64+0x5b/0x1e0
[ 690.393857] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 690.400020] RIP: 0033:0x7fa8cbe4f70d
[ 690.404533] Code: bc 20 00 00 75 10 b8 4a 00 00 00 0f 05 48 3d 01 f0 ff
ff 73 31 c3 48 83 ec 08 e8 6e f7 ff ff 48 89 04 24 b8 4a 00 00 00 0f 05 <48>
8b 3c 24 48 89 c2 e8 b7 f7 ff ff 48 89 d0 48 83 c4 08 48 3d 01
[ 690.426581] RSP: 002b:00007ffed6954160 EFLAGS: 00000293 ORIG_RAX:
000000000000004a
[ 690.435583] RAX: ffffffffffffffda RBX: 00005573363790f0 RCX:
00007fa8cbe4f70d
[ 690.444102] RDX: 0000000000200000 RSI: 00005573363790f0 RDI:
0000000000000003
[ 690.452615] RBP: 00007fa8b337f418 R08: 0000000000000008 R09:
0000000000000001
[ 690.461124] R10: 00000000897b0000 R11: 0000000000000293 R12:
0000000000000000
[ 690.469631] R13: 0000001000000000 R14: 00007fa8c81dda50 R15:
0000000000000000
[ 690.478130] ---[ end trace 90dd4ee6d5cdfcb8 ]---
I have just checked xfs code, and looks the only reason is that bio size
is overflowed, that means the whole bio may reach 4GB. That might be
possible in case of writeback. But I have to double check if that is
the case, could you test the following patch and provide the log?
diff --git a/block/blk-core.c b/block/blk-core.c
index 8340f69670d8..9d61e408c608 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -1125,6 +1125,19 @@ blk_qc_t submit_bio(struct bio *bio)
bio_devname(bio, b), count);
}
}
+ if (WARN_ON(bio->bi_iter.bi_size == 0 && bio_op(bio) == REQ_OP_WRITE)) {
+ int i;
+ printk("bio: %hx/%hx %u %llu %u\n", bio->bi_flags, bio->bi_opf,
+ bio->bi_vcnt,
+ (unsigned long long)bio->bi_iter.bi_sector,
+ bio->bi_iter.bi_size);
+ for (i = 0; i < bio->bi_vcnt; i++) {
+ struct bio_vec *bvec = &bio->bi_io_vec[i];
+ printk("%d: pfn %lu offset %u len %u\n", i++,
+ (unsigned long)page_to_pfn(bvec->bv_page),
+ bvec->bv_offset, bvec->bv_len);
+ }
+ }
return generic_make_request(bio);
}
Thanks,
Ming
9:49 p.m.
在 2019/7/1 上午10:47, Ming Lei 写道:
[snip]
>>>
>>> ------
>>>
>>> 2412 [ 112.558155] XFS (sdb1): Ending clean mount
>>> 2413 [ 1317.330820] WARNING: CPU: 63 PID: 1495 at
>>> drivers/scsi/scsi_lib.c:1034 scsi_init_io+0x11a/0x2e0
>>> 2414 [ 1317.344000] Modules linked in: xfs dm_mod btrfs xor zstd_decompress
>>> zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac
>>> x86_pkg_temp_thermal intel_powerclamp coretemp kvm_i ntel kvm irqbypass
>>> crct10dif_pclmul crc32_pclmul crc32c_intel mgag200 snd_pcm
>>> ghash_clmulni_intel ttm snd_timer drm_kms_helper ipmi_si ahci snd
>>> aesni_intel syscopyarea libahci ipmi_devi ntf sysfillrect sysimgblt
>>> crypto_simd mei_me fb_sys_fops soundcore ioatdma cryptd ipmi_msghandler
>>> glue_helper pcspkr drm libata mei joydev lpc_ich i2c_i801 dca wmi
>>> acpi_power_meter acpi _pad pcc_cpufreq ip_tables
>>> 2415 [ 1317.408135] CPU: 63 PID: 1495 Comm: kworker/63:1H Tainted:
>>> G I 5.2.0-rc5-00001-gc2d55e2 #1
>>> 2416 [ 1317.420350] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
>>> SE5C610.86B.01.01.0008.021120151325 02/11/2015
>>> 2417 [ 1317.433545] Workqueue: kblockd blk_mq_run_work_fn
>>> 2418 [ 1317.440295] RIP: 0010:scsi_init_io+0x11a/0x2e0
>>> 2419 [ 1317.446746] Code: be 01 00 00 00 e9 44 ff ff ff 8b 45 70 89 83 08 01
>>> 00 00 f6 45 1a 01 75 8a 31 c0 48 83 c4 08 5b 5d 41 5c 41 5d 41 5e 41 5f c3
>>> <0f> 0b 8b 4d 18 8b 55 28 45 31 c9 48 c7 c6 98 da 69 82 48 c7 c7
f0
>>> 2420 [ 1317.470863] RSP: 0018:ffffa5fa4f7e7c68 EFLAGS: 00010246
>>> 2421 [ 1317.478294] RAX: 000000003d17d1b8 RBX: ffff8ae6cb742690 RCX:
>>> 0000000000000000
>>> 2422 [ 1317.487878] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
>>> ffff8ae6cb742690
>>> 2423 [ 1317.497469] RBP: ffff8ae6cb742580 R08: 0000000000023003 R09:
>>> ffff8ae6cb7426c8
>>> 2424 [ 1317.507057] R10: ffffa5fa4c7d7da0 R11: fefefefefefefeff R12:
>>> 0000000000000801
>>> 2425 [ 1317.516650] R13: 0000000000000000 R14: 0000000000000000 R15:
>>> ffff8ac800bd1800
>>> 2426 [ 1317.526253] FS: 0000000000000000(0000) GS:ffff8b077fac0000(0000)
>>> knlGS:0000000000000000
>>> 2427 [ 1317.536918] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> 2428 [ 1317.544952] CR2: 00007f39a5818004 CR3: 000000343440a006 CR4:
>>> 00000000001606e0
>>> 2429 [ 1317.554554] Call Trace:
>>> 2430 [ 1317.558921] sd_init_command+0x2ee/0xaf0 [sd_mod]
>>> 2431 [ 1317.565800] scsi_queue_rq+0x559/0xa80
>>> 2432 [ 1317.571607] blk_mq_dispatch_rq_list+0x3da/0x5b0
>>> 2433 [ 1317.578375] ? syscall_return_via_sysret+0x10/0x7f
>>> 2434 [ 1317.585339] ? __switch_to_asm+0x40/0x70
>>> 2435 [ 1317.591321] ? elv_rb_del+0x1f/0x30
>>> 2436 [ 1317.596807] ? deadline_remove_request+0x55/0xc0
>>> 2437 [ 1317.603560] blk_mq_do_dispatch_sched+0x76/0x120
>>> 2438 [ 1317.610302] blk_mq_sched_dispatch_requests+0x100/0x170
>>> 2439 [ 1317.617733] __blk_mq_run_hw_queue+0x60/0x130
>>> 2440 [ 1317.624200] process_one_work+0x19c/0x3c0
>>> 2441 [ 1317.630258] worker_thread+0x3c/0x3b0
>>> 2442 [ 1317.635937] ? process_one_work+0x3c0/0x3c0
>>> 2443 [ 1317.642183] kthread+0x11e/0x140
>>> 2444 [ 1317.647346] ? kthread_park+0x90/0x90
>>> 2445 [ 1317.652986] ret_from_fork+0x35/0x40
>>> 2446 [ 1317.658523] ---[ end trace 61026ff97261da6a ]---
>>> 2447 [ 1317.665327] scsi_dump_rq: rq 6(f:801/23003, seg: 0)
>>> 2448 [ 1317.671601] bio 0: d602/801 0 1024971192 0
>>> 2449 [ 1317.676991] print_req_error: I/O error, dev sdb, sector 1024971192
>>> flags 801
>>> 2450 [ 1317.685722] scsi_dump_rq: rq 8(f:801/23003, seg: 0)
>>> 2451 [ 1317.691994] bio 0: d602/801 0 1033359800 0
>> The above line shows the bio is zero-size, and this bio is submitted
>> from bio, since the bio->bi_flags indicates that BIO_CLONED isn't set.
>>
>> Please apply the following debug patch, and we may see where is wrong.
>>
>> diff --git a/block/blk-core.c b/block/blk-core.c
>> index 8340f69670d8..86bb61b0df83 100644
>> --- a/block/blk-core.c
>> +++ b/block/blk-core.c
>> @@ -1125,6 +1125,7 @@ blk_qc_t submit_bio(struct bio *bio)
>> bio_devname(bio, b), count);
>> }
>> }
>> + WARN_ON(bio->bi_iter.bi_size == 0 && bio_op(bio) == REQ_OP_WRITE);
>> return generic_make_request(bio);
>
> Sorry for delay,
>
> ------
>
> [ 116.619136] 2019-06-29 16:12:41 mkdir -p /fs/sdb1
> [ 116.619138]
> [ 116.628743] 2019-06-29 16:12:41 mount -t xfs -o inode64 /dev/sdb1
> /fs/sdb1
> [ 116.628745]
> [ 116.634699] SGI XFS with ACLs, security attributes, realtime, scrub,
> debug enabled
> [ 116.653110] XFS (sdb1): Mounting V5 Filesystem
> [ 116.670081] XFS (sdb1): Ending clean mount
> [ 690.106779] WARNING: CPU: 53 PID: 1477 at block/blk-core.c:1128
> submit_bio+0x93/0x150
> [ 690.117972] Modules linked in: xfs dm_mod btrfs xor zstd_decompress
> zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac
> x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
> mgag200 crct10dif_pclmul crc32_pclmul ttm crc32c_intel snd_pcm
> ghash_clmulni_intel drm_kms_helper snd_timer ahci syscopyarea sysfillrect
> snd aesni_intel sysimgblt libahci ipmi_si soundcore fb_sys_fops mei_me
> crypto_simd ipmi_devintf ioatdma cryptd pcspkr drm ipmi_msghandler
> glue_helper libata mei joydev lpc_ich i2c_i801 dca wmi acpi_power_meter
> acpi_pad pcc_cpufreq ip_tables
> [ 690.176967] CPU: 53 PID: 1477 Comm: fio Tainted: G W I
> 5.2.0-rc5-00002-g9fa09bd #1
> [ 690.187198] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
> SE5C610.86B.01.01.0008.021120151325 02/11/2015
> [ 690.199375] RIP: 0010:submit_bio+0x93/0x150
> [ 690.204583] Code: ee 74 45 83 e1 01 74 26 65 48 01 35 bf 39 3f 45 8b 0d
> ad 96 c4 01 85 c9 75 56 8b 43 28 85 c0 75 b3 0f b6 53 10 83 fa 01 75 aa <0f>
> 0b eb a6 65 48 8b 14 25 c0 6b 01 00 65 48 01 35 88 39 3f 45 48
> [ 690.226668] RSP: 0018:ffffb2d90efc3b40 EFLAGS: 00010246
> [ 690.233072] RAX: 0000000000000000 RBX: ffff9654c6ad9f40 RCX:
> 0000000000000801
> [ 690.241617] RDX: 0000000000000001 RSI: ffff9654c6ad9f00 RDI:
> ffff9654c6ad9f40
> [ 690.250146] RBP: 0000000000000000 R08: 0000000000000002 R09:
> 0000000000000000
> [ 690.258667] R10: 00000000000303a0 R11: ffff96757ffd4000 R12:
> ffffb2d90efc3e08
> [ 690.267183] R13: ffff9654c6ad9f00 R14: ffffb2d90efc3e08 R15:
> 0000000000000000
> [ 690.275696] FS: 00007fa8e005da80(0000) GS:ffff9654ffac0000(0000)
> knlGS:0000000000000000
> [ 690.285282] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 690.292252] CR2: 00007fa8cc05b3c0 CR3: 0000001fc8518001 CR4:
> 00000000001606e0
> [ 690.300770] Call Trace:
> [ 690.304123] xfs_submit_ioend+0x84/0x1e0 [xfs]
> [ 690.309668] xfs_do_writepage+0x246/0x5a0 [xfs]
> [ 690.315279] write_cache_pages+0x16b/0x440
> [ 690.320420] ? xfs_vm_releasepage+0x80/0x80 [xfs]
> [ 690.326238] xfs_vm_writepages+0x6b/0xa0 [xfs]
> [ 690.331742] do_writepages+0x4b/0xf0
> [ 690.336275] ? iomap_file_buffered_write+0x62/0x90
> [ 690.342162] ? iomap_write_end+0x240/0x240
> [ 690.347295] ? xfs_iunlock+0x165/0x220 [xfs]
> [ 690.352595] ? __filemap_fdatawrite_range+0xc1/0x100
> [ 690.358673] __filemap_fdatawrite_range+0xc1/0x100
> [ 690.364571] ? __switch_to_asm+0x34/0x70
> [ 690.369484] file_write_and_wait_range+0x4c/0xa0
> [ 690.375198] xfs_file_fsync+0x5d/0x230 [xfs]
> [ 690.380499] do_fsync+0x38/0x60
> [ 690.384534] __x64_sys_fsync+0x10/0x20
> [ 690.389243] do_syscall_64+0x5b/0x1e0
> [ 690.393857] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [ 690.400020] RIP: 0033:0x7fa8cbe4f70d
> [ 690.404533] Code: bc 20 00 00 75 10 b8 4a 00 00 00 0f 05 48 3d 01 f0 ff
> ff 73 31 c3 48 83 ec 08 e8 6e f7 ff ff 48 89 04 24 b8 4a 00 00 00 0f 05 <48>
> 8b 3c 24 48 89 c2 e8 b7 f7 ff ff 48 89 d0 48 83 c4 08 48 3d 01
> [ 690.426581] RSP: 002b:00007ffed6954160 EFLAGS: 00000293 ORIG_RAX:
> 000000000000004a
> [ 690.435583] RAX: ffffffffffffffda RBX: 00005573363790f0 RCX:
> 00007fa8cbe4f70d
> [ 690.444102] RDX: 0000000000200000 RSI: 00005573363790f0 RDI:
> 0000000000000003
> [ 690.452615] RBP: 00007fa8b337f418 R08: 0000000000000008 R09:
> 0000000000000001
> [ 690.461124] R10: 00000000897b0000 R11: 0000000000000293 R12:
> 0000000000000000
> [ 690.469631] R13: 0000001000000000 R14: 00007fa8c81dda50 R15:
> 0000000000000000
> [ 690.478130] ---[ end trace 90dd4ee6d5cdfcb8 ]---
I have just checked xfs code, and looks the only reason is that bio size
is overflowed, that means the whole bio may reach 4GB. That might be
possible in case of writeback. But I have to double check if that is
the case, could you test the following patch and provide the log?
diff --git a/block/blk-core.c b/block/blk-core.c
index 8340f69670d8..9d61e408c608 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -1125,6 +1125,19 @@ blk_qc_t submit_bio(struct bio *bio)
bio_devname(bio, b), count);
}
}
+ if (WARN_ON(bio->bi_iter.bi_size == 0 && bio_op(bio) == REQ_OP_WRITE)) {
+ int i;
+ printk("bio: %hx/%hx %u %llu %u\n", bio->bi_flags, bio->bi_opf,
+ bio->bi_vcnt,
+ (unsigned long long)bio->bi_iter.bi_sector,
+ bio->bi_iter.bi_size);
+ for (i = 0; i < bio->bi_vcnt; i++) {
+ struct bio_vec *bvec = &bio->bi_io_vec[i];
+ printk("%d: pfn %lu offset %u len %u\n", i++,
+ (unsigned long)page_to_pfn(bvec->bv_page),
+ bvec->bv_offset, bvec->bv_len);
+ }
+ }
return generic_make_request(bio);
}
the newest log and i also see you fix patch, i will test your fix patch
more times to check it.
------
[ 54.656016] 2019-07-01 11:48:52 mkfs -t xfs /dev/sdb1
[ 54.656017]
[ 119.228467] WARNING: CPU: 5 PID: 1530 at block/blk-core.c:1128
submit_bio+0xa5/0x1d0
[ 119.240810] Modules linked in: dm_mod btrfs xor zstd_decompress
zstd_compress raid6_pq libcrc32c sd_mod intel_rapl sg sb_edac
x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
mgag200 crct10dif_pclmul snd_pcm crc32_pclmul ttm crc32c_intel snd_timer
drm_kms_helper ghash_clmulni_intel snd syscopyarea ipmi_si sysfillrect
soundcore aesni_intel sysimgblt ipmi_devintf crypto_simd fb_sys_fops
ahci mei_me libahci cryptd ipmi_msghandler pcspkr glue_helper drm mei
libata joydev ioatdma lpc_ich i2c_i801 dca wmi acpi_power_meter acpi_pad
pcc_cpufreq ip_tables
[ 119.306252] CPU: 5 PID: 1530 Comm: mkfs.xfs Tainted: G W
I 5.2.0-rc5-00001-g82f8097 #1
[ 119.318274] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
SE5C610.86B.01.01.0008.021120151325 02/11/2015
[ 119.331852] RIP: 0010:submit_bio+0xa5/0x1d0
[ 119.338422] Code: 00 00 65 48 01 35 b3 39 5f 76 8b 0d a1 96 c4 01 85
c9 0f 85 9b 00 00 00 8b 43 28 85 c0 75 a5 8b 53 10 0f b6 ca 83 f9 01 75
9a <0f> 0b 0f b7 4b 68 0f b7 73 14 45 31 c9 4c 8b 43 20 48 c7 c7 f1 99
[ 119.363296] RSP: 0018:ffff940c4fb17e58 EFLAGS: 00010246
[ 119.371107] RAX: 0000000000000000 RBX: ffff88a53e3d6900 RCX:
0000000000000001
[ 119.381068] RDX: 0000000000040801 RSI: ffff88868138c000 RDI:
ffff88a53e3d6900
[ 119.391041] RBP: ffff88a5448b0000 R08: ffff88a53e3d6900 R09:
0000000000000000
[ 119.401003] R10: 0000000000000000 R11: 0000000000000000 R12:
0000000000000000
[ 119.410967] R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
[ 119.420881] FS: 00007f37eba81780(0000) GS:ffff88a57f340000(0000)
knlGS:0000000000000000
[ 119.431839] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 119.440161] CR2: 00005630eb09d000 CR3: 0000001fc711c002 CR4:
00000000001606e0
[ 119.450017] Call Trace:
[ 119.454595] submit_bio_wait+0x53/0x80
[ 119.460636] blkdev_issue_flush+0x81/0xb0
[ 119.466930] blkdev_fsync+0x2f/0x40
[ 119.472590] do_fsync+0x38/0x60
[ 119.477810] __x64_sys_fsync+0x10/0x20
[ 119.483722] do_syscall_64+0x5b/0x1e0
[ 119.489516] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 119.496857] RIP: 0033:0x7f37eb2146f0
[ 119.502502] Code: 48 3d 01 f0 ff ff 73 01 c3 48 8b 0d b2 78 20 00 f7
d8 64 89 01 48 83 c8 ff c3 83 3d d9 bc 20 00 00 75 10 b8 4a 00 00 00 0f
05 <48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 6e f7 ff ff 48 89 04 24
[ 119.526842] RSP: 002b:00007fffb27f4878 EFLAGS: 00000246 ORIG_RAX:
000000000000004a
[ 119.536975] RAX: ffffffffffffffda RBX: 0000000000000811 RCX:
00007f37eb2146f0
[ 119.546592] RDX: 00005630e91ba8e0 RSI: 0000000000000000 RDI:
0000000000000004
[ 119.556171] RBP: 0000000000000004 R08: 00005630eb02b8a0 R09:
00007fffb27f4e00
[ 119.565753] R10: 000000000000009b R11: 0000000000000246 R12:
00007fffb27f4c60
[ 119.575339] R13: 00007fffb27f4a30 R14: 00005630eb048800 R15:
00005630eb05a800
[ 119.584940] ---[ end trace 46a2e1214c930529 ]---
[ 119.591765] bio: 0/801 0 0 0
------
Thanks,
Ming
11:29 p.m.
On Mon, Jul 01, 2019 at 12:49:51PM +0800, Liu Yiding wrote:
在 2019/7/1 上午10:47, Ming Lei 写道:
> [snip]
> > > >
> > > > ------
> > > >
> > > > 2412 [ 112.558155] XFS (sdb1): Ending clean mount
> > > > 2413 [ 1317.330820] WARNING: CPU: 63 PID: 1495 at
> > > > drivers/scsi/scsi_lib.c:1034 scsi_init_io+0x11a/0x2e0
> > > > 2414 [ 1317.344000] Modules linked in: xfs dm_mod btrfs xor
zstd_decompress
> > > > zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac
> > > > x86_pkg_temp_thermal intel_powerclamp coretemp kvm_i ntel kvm
irqbypass
> > > > crct10dif_pclmul crc32_pclmul crc32c_intel mgag200 snd_pcm
> > > > ghash_clmulni_intel ttm snd_timer drm_kms_helper ipmi_si ahci snd
> > > > aesni_intel syscopyarea libahci ipmi_devi ntf sysfillrect
sysimgblt
> > > > crypto_simd mei_me fb_sys_fops soundcore ioatdma cryptd
ipmi_msghandler
> > > > glue_helper pcspkr drm libata mei joydev lpc_ich i2c_i801 dca wmi
> > > > acpi_power_meter acpi _pad pcc_cpufreq ip_tables
> > > > 2415 [ 1317.408135] CPU: 63 PID: 1495 Comm: kworker/63:1H Tainted:
> > > > G I 5.2.0-rc5-00001-gc2d55e2 #1
> > > > 2416 [ 1317.420350] Hardware name: Intel Corporation
S2600WTT/S2600WTT, BIOS
> > > > SE5C610.86B.01.01.0008.021120151325 02/11/2015
> > > > 2417 [ 1317.433545] Workqueue: kblockd blk_mq_run_work_fn
> > > > 2418 [ 1317.440295] RIP: 0010:scsi_init_io+0x11a/0x2e0
> > > > 2419 [ 1317.446746] Code: be 01 00 00 00 e9 44 ff ff ff 8b 45 70 89
83 08 01
> > > > 00 00 f6 45 1a 01 75 8a 31 c0 48 83 c4 08 5b 5d 41 5c 41 5d 41 5e 41
5f c3
> > > > <0f> 0b 8b 4d 18 8b 55 28 45 31 c9 48 c7 c6 98 da 69 82 48
c7 c7 f0
> > > > 2420 [ 1317.470863] RSP: 0018:ffffa5fa4f7e7c68 EFLAGS: 00010246
> > > > 2421 [ 1317.478294] RAX: 000000003d17d1b8 RBX: ffff8ae6cb742690 RCX:
> > > > 0000000000000000
> > > > 2422 [ 1317.487878] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
> > > > ffff8ae6cb742690
> > > > 2423 [ 1317.497469] RBP: ffff8ae6cb742580 R08: 0000000000023003 R09:
> > > > ffff8ae6cb7426c8
> > > > 2424 [ 1317.507057] R10: ffffa5fa4c7d7da0 R11: fefefefefefefeff R12:
> > > > 0000000000000801
> > > > 2425 [ 1317.516650] R13: 0000000000000000 R14: 0000000000000000 R15:
> > > > ffff8ac800bd1800
> > > > 2426 [ 1317.526253] FS: 0000000000000000(0000)
GS:ffff8b077fac0000(0000)
> > > > knlGS:0000000000000000
> > > > 2427 [ 1317.536918] CS: 0010 DS: 0000 ES: 0000 CR0:
0000000080050033
> > > > 2428 [ 1317.544952] CR2: 00007f39a5818004 CR3: 000000343440a006 CR4:
> > > > 00000000001606e0
> > > > 2429 [ 1317.554554] Call Trace:
> > > > 2430 [ 1317.558921] sd_init_command+0x2ee/0xaf0 [sd_mod]
> > > > 2431 [ 1317.565800] scsi_queue_rq+0x559/0xa80
> > > > 2432 [ 1317.571607] blk_mq_dispatch_rq_list+0x3da/0x5b0
> > > > 2433 [ 1317.578375] ? syscall_return_via_sysret+0x10/0x7f
> > > > 2434 [ 1317.585339] ? __switch_to_asm+0x40/0x70
> > > > 2435 [ 1317.591321] ? elv_rb_del+0x1f/0x30
> > > > 2436 [ 1317.596807] ? deadline_remove_request+0x55/0xc0
> > > > 2437 [ 1317.603560] blk_mq_do_dispatch_sched+0x76/0x120
> > > > 2438 [ 1317.610302] blk_mq_sched_dispatch_requests+0x100/0x170
> > > > 2439 [ 1317.617733] __blk_mq_run_hw_queue+0x60/0x130
> > > > 2440 [ 1317.624200] process_one_work+0x19c/0x3c0
> > > > 2441 [ 1317.630258] worker_thread+0x3c/0x3b0
> > > > 2442 [ 1317.635937] ? process_one_work+0x3c0/0x3c0
> > > > 2443 [ 1317.642183] kthread+0x11e/0x140
> > > > 2444 [ 1317.647346] ? kthread_park+0x90/0x90
> > > > 2445 [ 1317.652986] ret_from_fork+0x35/0x40
> > > > 2446 [ 1317.658523] ---[ end trace 61026ff97261da6a ]---
> > > > 2447 [ 1317.665327] scsi_dump_rq: rq 6(f:801/23003, seg: 0)
> > > > 2448 [ 1317.671601] bio 0: d602/801 0 1024971192 0
> > > > 2449 [ 1317.676991] print_req_error: I/O error, dev sdb, sector
1024971192
> > > > flags 801
> > > > 2450 [ 1317.685722] scsi_dump_rq: rq 8(f:801/23003, seg: 0)
> > > > 2451 [ 1317.691994] bio 0: d602/801 0 1033359800 0
> > > The above line shows the bio is zero-size, and this bio is submitted
> > > from bio, since the bio->bi_flags indicates that BIO_CLONED isn't
set.
> > >
> > > Please apply the following debug patch, and we may see where is wrong.
> > >
> > > diff --git a/block/blk-core.c b/block/blk-core.c
> > > index 8340f69670d8..86bb61b0df83 100644
> > > --- a/block/blk-core.c
> > > +++ b/block/blk-core.c
> > > @@ -1125,6 +1125,7 @@ blk_qc_t submit_bio(struct bio *bio)
> > > bio_devname(bio, b), count);
> > > }
> > > }
> > > + WARN_ON(bio->bi_iter.bi_size == 0 && bio_op(bio) ==
REQ_OP_WRITE);
> > > return generic_make_request(bio);
> >
> > Sorry for delay,
> >
> > ------
> >
> > [ 116.619136] 2019-06-29 16:12:41 mkdir -p /fs/sdb1
> > [ 116.619138]
> > [ 116.628743] 2019-06-29 16:12:41 mount -t xfs -o inode64 /dev/sdb1
> > /fs/sdb1
> > [ 116.628745]
> > [ 116.634699] SGI XFS with ACLs, security attributes, realtime, scrub,
> > debug enabled
> > [ 116.653110] XFS (sdb1): Mounting V5 Filesystem
> > [ 116.670081] XFS (sdb1): Ending clean mount
> > [ 690.106779] WARNING: CPU: 53 PID: 1477 at block/blk-core.c:1128
> > submit_bio+0x93/0x150
> > [ 690.117972] Modules linked in: xfs dm_mod btrfs xor zstd_decompress
> > zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac
> > x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
> > mgag200 crct10dif_pclmul crc32_pclmul ttm crc32c_intel snd_pcm
> > ghash_clmulni_intel drm_kms_helper snd_timer ahci syscopyarea sysfillrect
> > snd aesni_intel sysimgblt libahci ipmi_si soundcore fb_sys_fops mei_me
> > crypto_simd ipmi_devintf ioatdma cryptd pcspkr drm ipmi_msghandler
> > glue_helper libata mei joydev lpc_ich i2c_i801 dca wmi acpi_power_meter
> > acpi_pad pcc_cpufreq ip_tables
> > [ 690.176967] CPU: 53 PID: 1477 Comm: fio Tainted: G W I
> > 5.2.0-rc5-00002-g9fa09bd #1
> > [ 690.187198] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
> > SE5C610.86B.01.01.0008.021120151325 02/11/2015
> > [ 690.199375] RIP: 0010:submit_bio+0x93/0x150
> > [ 690.204583] Code: ee 74 45 83 e1 01 74 26 65 48 01 35 bf 39 3f 45 8b 0d
> > ad 96 c4 01 85 c9 75 56 8b 43 28 85 c0 75 b3 0f b6 53 10 83 fa 01 75 aa
<0f>
> > 0b eb a6 65 48 8b 14 25 c0 6b 01 00 65 48 01 35 88 39 3f 45 48
> > [ 690.226668] RSP: 0018:ffffb2d90efc3b40 EFLAGS: 00010246
> > [ 690.233072] RAX: 0000000000000000 RBX: ffff9654c6ad9f40 RCX:
> > 0000000000000801
> > [ 690.241617] RDX: 0000000000000001 RSI: ffff9654c6ad9f00 RDI:
> > ffff9654c6ad9f40
> > [ 690.250146] RBP: 0000000000000000 R08: 0000000000000002 R09:
> > 0000000000000000
> > [ 690.258667] R10: 00000000000303a0 R11: ffff96757ffd4000 R12:
> > ffffb2d90efc3e08
> > [ 690.267183] R13: ffff9654c6ad9f00 R14: ffffb2d90efc3e08 R15:
> > 0000000000000000
> > [ 690.275696] FS: 00007fa8e005da80(0000) GS:ffff9654ffac0000(0000)
> > knlGS:0000000000000000
> > [ 690.285282] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [ 690.292252] CR2: 00007fa8cc05b3c0 CR3: 0000001fc8518001 CR4:
> > 00000000001606e0
> > [ 690.300770] Call Trace:
> > [ 690.304123] xfs_submit_ioend+0x84/0x1e0 [xfs]
> > [ 690.309668] xfs_do_writepage+0x246/0x5a0 [xfs]
> > [ 690.315279] write_cache_pages+0x16b/0x440
> > [ 690.320420] ? xfs_vm_releasepage+0x80/0x80 [xfs]
> > [ 690.326238] xfs_vm_writepages+0x6b/0xa0 [xfs]
> > [ 690.331742] do_writepages+0x4b/0xf0
> > [ 690.336275] ? iomap_file_buffered_write+0x62/0x90
> > [ 690.342162] ? iomap_write_end+0x240/0x240
> > [ 690.347295] ? xfs_iunlock+0x165/0x220 [xfs]
> > [ 690.352595] ? __filemap_fdatawrite_range+0xc1/0x100
> > [ 690.358673] __filemap_fdatawrite_range+0xc1/0x100
> > [ 690.364571] ? __switch_to_asm+0x34/0x70
> > [ 690.369484] file_write_and_wait_range+0x4c/0xa0
> > [ 690.375198] xfs_file_fsync+0x5d/0x230 [xfs]
> > [ 690.380499] do_fsync+0x38/0x60
> > [ 690.384534] __x64_sys_fsync+0x10/0x20
> > [ 690.389243] do_syscall_64+0x5b/0x1e0
> > [ 690.393857] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > [ 690.400020] RIP: 0033:0x7fa8cbe4f70d
> > [ 690.404533] Code: bc 20 00 00 75 10 b8 4a 00 00 00 0f 05 48 3d 01 f0 ff
> > ff 73 31 c3 48 83 ec 08 e8 6e f7 ff ff 48 89 04 24 b8 4a 00 00 00 0f 05
<48>
> > 8b 3c 24 48 89 c2 e8 b7 f7 ff ff 48 89 d0 48 83 c4 08 48 3d 01
> > [ 690.426581] RSP: 002b:00007ffed6954160 EFLAGS: 00000293 ORIG_RAX:
> > 000000000000004a
> > [ 690.435583] RAX: ffffffffffffffda RBX: 00005573363790f0 RCX:
> > 00007fa8cbe4f70d
> > [ 690.444102] RDX: 0000000000200000 RSI: 00005573363790f0 RDI:
> > 0000000000000003
> > [ 690.452615] RBP: 00007fa8b337f418 R08: 0000000000000008 R09:
> > 0000000000000001
> > [ 690.461124] R10: 00000000897b0000 R11: 0000000000000293 R12:
> > 0000000000000000
> > [ 690.469631] R13: 0000001000000000 R14: 00007fa8c81dda50 R15:
> > 0000000000000000
> > [ 690.478130] ---[ end trace 90dd4ee6d5cdfcb8 ]---
> I have just checked xfs code, and looks the only reason is that bio size
> is overflowed, that means the whole bio may reach 4GB. That might be
> possible in case of writeback. But I have to double check if that is
> the case, could you test the following patch and provide the log?
>
> diff --git a/block/blk-core.c b/block/blk-core.c
> index 8340f69670d8..9d61e408c608 100644
> --- a/block/blk-core.c
> +++ b/block/blk-core.c
> @@ -1125,6 +1125,19 @@ blk_qc_t submit_bio(struct bio *bio)
> bio_devname(bio, b), count);
> }
> }
> + if (WARN_ON(bio->bi_iter.bi_size == 0 && bio_op(bio) == REQ_OP_WRITE))
{
> + int i;
> + printk("bio: %hx/%hx %u %llu %u\n", bio->bi_flags, bio->bi_opf,
> + bio->bi_vcnt,
> + (unsigned long long)bio->bi_iter.bi_sector,
> + bio->bi_iter.bi_size);
> + for (i = 0; i < bio->bi_vcnt; i++) {
> + struct bio_vec *bvec = &bio->bi_io_vec[i];
> + printk("%d: pfn %lu offset %u len %u\n", i++,
> + (unsigned long)page_to_pfn(bvec->bv_page),
> + bvec->bv_offset, bvec->bv_len);
> + }
> + }
> return generic_make_request(bio);
> }
the newest log and i also see you fix patch, i will test your fix patch
more times to check it.
------
[ 54.656016] 2019-07-01 11:48:52 mkfs -t xfs /dev/sdb1
[ 54.656017]
[ 119.228467] WARNING: CPU: 5 PID: 1530 at block/blk-core.c:1128
submit_bio+0xa5/0x1d0
[ 119.240810] Modules linked in: dm_mod btrfs xor zstd_decompress
zstd_compress raid6_pq libcrc32c sd_mod intel_rapl sg sb_edac
x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
mgag200 crct10dif_pclmul snd_pcm crc32_pclmul ttm crc32c_intel snd_timer
drm_kms_helper ghash_clmulni_intel snd syscopyarea ipmi_si sysfillrect
soundcore aesni_intel sysimgblt ipmi_devintf crypto_simd fb_sys_fops ahci
mei_me libahci cryptd ipmi_msghandler pcspkr glue_helper drm mei libata
joydev ioatdma lpc_ich i2c_i801 dca wmi acpi_power_meter acpi_pad
pcc_cpufreq ip_tables
[ 119.306252] CPU: 5 PID: 1530 Comm: mkfs.xfs Tainted: G W I
5.2.0-rc5-00001-g82f8097 #1
[ 119.318274] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
SE5C610.86B.01.01.0008.021120151325 02/11/2015
[ 119.331852] RIP: 0010:submit_bio+0xa5/0x1d0
[ 119.338422] Code: 00 00 65 48 01 35 b3 39 5f 76 8b 0d a1 96 c4 01 85 c9
0f 85 9b 00 00 00 8b 43 28 85 c0 75 a5 8b 53 10 0f b6 ca 83 f9 01 75 9a <0f>
0b 0f b7 4b 68 0f b7 73 14 45 31 c9 4c 8b 43 20 48 c7 c7 f1 99
[ 119.363296] RSP: 0018:ffff940c4fb17e58 EFLAGS: 00010246
[ 119.371107] RAX: 0000000000000000 RBX: ffff88a53e3d6900 RCX:
0000000000000001
[ 119.381068] RDX: 0000000000040801 RSI: ffff88868138c000 RDI:
ffff88a53e3d6900
[ 119.391041] RBP: ffff88a5448b0000 R08: ffff88a53e3d6900 R09:
0000000000000000
[ 119.401003] R10: 0000000000000000 R11: 0000000000000000 R12:
0000000000000000
[ 119.410967] R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
[ 119.420881] FS: 00007f37eba81780(0000) GS:ffff88a57f340000(0000)
knlGS:0000000000000000
[ 119.431839] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 119.440161] CR2: 00005630eb09d000 CR3: 0000001fc711c002 CR4:
00000000001606e0
[ 119.450017] Call Trace:
[ 119.454595] submit_bio_wait+0x53/0x80
[ 119.460636] blkdev_issue_flush+0x81/0xb0
[ 119.466930] blkdev_fsync+0x2f/0x40
[ 119.472590] do_fsync+0x38/0x60
[ 119.477810] __x64_sys_fsync+0x10/0x20
[ 119.483722] do_syscall_64+0x5b/0x1e0
[ 119.489516] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 119.496857] RIP: 0033:0x7f37eb2146f0
[ 119.502502] Code: 48 3d 01 f0 ff ff 73 01 c3 48 8b 0d b2 78 20 00 f7 d8
64 89 01 48 83 c8 ff c3 83 3d d9 bc 20 00 00 75 10 b8 4a 00 00 00 0f 05 <48>
3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 6e f7 ff ff 48 89 04 24
[ 119.526842] RSP: 002b:00007fffb27f4878 EFLAGS: 00000246 ORIG_RAX:
000000000000004a
[ 119.536975] RAX: ffffffffffffffda RBX: 0000000000000811 RCX:
00007f37eb2146f0
[ 119.546592] RDX: 00005630e91ba8e0 RSI: 0000000000000000 RDI:
0000000000000004
[ 119.556171] RBP: 0000000000000004 R08: 00005630eb02b8a0 R09:
00007fffb27f4e00
[ 119.565753] R10: 000000000000009b R11: 0000000000000246 R12:
00007fffb27f4c60
[ 119.575339] R13: 00007fffb27f4a30 R14: 00005630eb048800 R15:
00005630eb05a800
[ 119.584940] ---[ end trace 46a2e1214c930529 ]---
[ 119.591765] bio: 0/801 0 0 0
The above log means that the bio is really empty bio from xfs, so
the patch I posted can't fix the issue.
I will take a close look at xfs code.
Thanks,
Ming
11:44 p.m.
On Mon, Jul 01, 2019 at 12:49:51PM +0800, Liu Yiding wrote:
在 2019/7/1 上午10:47, Ming Lei 写道:
> [snip]
> > > >
> > > > ------
> > > >
> > > > 2412 [ 112.558155] XFS (sdb1): Ending clean mount
> > > > 2413 [ 1317.330820] WARNING: CPU: 63 PID: 1495 at
> > > > drivers/scsi/scsi_lib.c:1034 scsi_init_io+0x11a/0x2e0
> > > > 2414 [ 1317.344000] Modules linked in: xfs dm_mod btrfs xor
zstd_decompress
> > > > zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac
> > > > x86_pkg_temp_thermal intel_powerclamp coretemp kvm_i ntel kvm
irqbypass
> > > > crct10dif_pclmul crc32_pclmul crc32c_intel mgag200 snd_pcm
> > > > ghash_clmulni_intel ttm snd_timer drm_kms_helper ipmi_si ahci snd
> > > > aesni_intel syscopyarea libahci ipmi_devi ntf sysfillrect
sysimgblt
> > > > crypto_simd mei_me fb_sys_fops soundcore ioatdma cryptd
ipmi_msghandler
> > > > glue_helper pcspkr drm libata mei joydev lpc_ich i2c_i801 dca wmi
> > > > acpi_power_meter acpi _pad pcc_cpufreq ip_tables
> > > > 2415 [ 1317.408135] CPU: 63 PID: 1495 Comm: kworker/63:1H Tainted:
> > > > G I 5.2.0-rc5-00001-gc2d55e2 #1
> > > > 2416 [ 1317.420350] Hardware name: Intel Corporation
S2600WTT/S2600WTT, BIOS
> > > > SE5C610.86B.01.01.0008.021120151325 02/11/2015
> > > > 2417 [ 1317.433545] Workqueue: kblockd blk_mq_run_work_fn
> > > > 2418 [ 1317.440295] RIP: 0010:scsi_init_io+0x11a/0x2e0
> > > > 2419 [ 1317.446746] Code: be 01 00 00 00 e9 44 ff ff ff 8b 45 70 89
83 08 01
> > > > 00 00 f6 45 1a 01 75 8a 31 c0 48 83 c4 08 5b 5d 41 5c 41 5d 41 5e 41
5f c3
> > > > <0f> 0b 8b 4d 18 8b 55 28 45 31 c9 48 c7 c6 98 da 69 82 48
c7 c7 f0
> > > > 2420 [ 1317.470863] RSP: 0018:ffffa5fa4f7e7c68 EFLAGS: 00010246
> > > > 2421 [ 1317.478294] RAX: 000000003d17d1b8 RBX: ffff8ae6cb742690 RCX:
> > > > 0000000000000000
> > > > 2422 [ 1317.487878] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
> > > > ffff8ae6cb742690
> > > > 2423 [ 1317.497469] RBP: ffff8ae6cb742580 R08: 0000000000023003 R09:
> > > > ffff8ae6cb7426c8
> > > > 2424 [ 1317.507057] R10: ffffa5fa4c7d7da0 R11: fefefefefefefeff R12:
> > > > 0000000000000801
> > > > 2425 [ 1317.516650] R13: 0000000000000000 R14: 0000000000000000 R15:
> > > > ffff8ac800bd1800
> > > > 2426 [ 1317.526253] FS: 0000000000000000(0000)
GS:ffff8b077fac0000(0000)
> > > > knlGS:0000000000000000
> > > > 2427 [ 1317.536918] CS: 0010 DS: 0000 ES: 0000 CR0:
0000000080050033
> > > > 2428 [ 1317.544952] CR2: 00007f39a5818004 CR3: 000000343440a006 CR4:
> > > > 00000000001606e0
> > > > 2429 [ 1317.554554] Call Trace:
> > > > 2430 [ 1317.558921] sd_init_command+0x2ee/0xaf0 [sd_mod]
> > > > 2431 [ 1317.565800] scsi_queue_rq+0x559/0xa80
> > > > 2432 [ 1317.571607] blk_mq_dispatch_rq_list+0x3da/0x5b0
> > > > 2433 [ 1317.578375] ? syscall_return_via_sysret+0x10/0x7f
> > > > 2434 [ 1317.585339] ? __switch_to_asm+0x40/0x70
> > > > 2435 [ 1317.591321] ? elv_rb_del+0x1f/0x30
> > > > 2436 [ 1317.596807] ? deadline_remove_request+0x55/0xc0
> > > > 2437 [ 1317.603560] blk_mq_do_dispatch_sched+0x76/0x120
> > > > 2438 [ 1317.610302] blk_mq_sched_dispatch_requests+0x100/0x170
> > > > 2439 [ 1317.617733] __blk_mq_run_hw_queue+0x60/0x130
> > > > 2440 [ 1317.624200] process_one_work+0x19c/0x3c0
> > > > 2441 [ 1317.630258] worker_thread+0x3c/0x3b0
> > > > 2442 [ 1317.635937] ? process_one_work+0x3c0/0x3c0
> > > > 2443 [ 1317.642183] kthread+0x11e/0x140
> > > > 2444 [ 1317.647346] ? kthread_park+0x90/0x90
> > > > 2445 [ 1317.652986] ret_from_fork+0x35/0x40
> > > > 2446 [ 1317.658523] ---[ end trace 61026ff97261da6a ]---
> > > > 2447 [ 1317.665327] scsi_dump_rq: rq 6(f:801/23003, seg: 0)
> > > > 2448 [ 1317.671601] bio 0: d602/801 0 1024971192 0
> > > > 2449 [ 1317.676991] print_req_error: I/O error, dev sdb, sector
1024971192
> > > > flags 801
> > > > 2450 [ 1317.685722] scsi_dump_rq: rq 8(f:801/23003, seg: 0)
> > > > 2451 [ 1317.691994] bio 0: d602/801 0 1033359800 0
> > > The above line shows the bio is zero-size, and this bio is submitted
> > > from bio, since the bio->bi_flags indicates that BIO_CLONED isn't
set.
> > >
> > > Please apply the following debug patch, and we may see where is wrong.
> > >
> > > diff --git a/block/blk-core.c b/block/blk-core.c
> > > index 8340f69670d8..86bb61b0df83 100644
> > > --- a/block/blk-core.c
> > > +++ b/block/blk-core.c
> > > @@ -1125,6 +1125,7 @@ blk_qc_t submit_bio(struct bio *bio)
> > > bio_devname(bio, b), count);
> > > }
> > > }
> > > + WARN_ON(bio->bi_iter.bi_size == 0 && bio_op(bio) ==
REQ_OP_WRITE);
> > > return generic_make_request(bio);
> >
> > Sorry for delay,
> >
> > ------
> >
> > [ 116.619136] 2019-06-29 16:12:41 mkdir -p /fs/sdb1
> > [ 116.619138]
> > [ 116.628743] 2019-06-29 16:12:41 mount -t xfs -o inode64 /dev/sdb1
> > /fs/sdb1
> > [ 116.628745]
> > [ 116.634699] SGI XFS with ACLs, security attributes, realtime, scrub,
> > debug enabled
> > [ 116.653110] XFS (sdb1): Mounting V5 Filesystem
> > [ 116.670081] XFS (sdb1): Ending clean mount
> > [ 690.106779] WARNING: CPU: 53 PID: 1477 at block/blk-core.c:1128
> > submit_bio+0x93/0x150
> > [ 690.117972] Modules linked in: xfs dm_mod btrfs xor zstd_decompress
> > zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac
> > x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
> > mgag200 crct10dif_pclmul crc32_pclmul ttm crc32c_intel snd_pcm
> > ghash_clmulni_intel drm_kms_helper snd_timer ahci syscopyarea sysfillrect
> > snd aesni_intel sysimgblt libahci ipmi_si soundcore fb_sys_fops mei_me
> > crypto_simd ipmi_devintf ioatdma cryptd pcspkr drm ipmi_msghandler
> > glue_helper libata mei joydev lpc_ich i2c_i801 dca wmi acpi_power_meter
> > acpi_pad pcc_cpufreq ip_tables
> > [ 690.176967] CPU: 53 PID: 1477 Comm: fio Tainted: G W I
> > 5.2.0-rc5-00002-g9fa09bd #1
> > [ 690.187198] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
> > SE5C610.86B.01.01.0008.021120151325 02/11/2015
> > [ 690.199375] RIP: 0010:submit_bio+0x93/0x150
> > [ 690.204583] Code: ee 74 45 83 e1 01 74 26 65 48 01 35 bf 39 3f 45 8b 0d
> > ad 96 c4 01 85 c9 75 56 8b 43 28 85 c0 75 b3 0f b6 53 10 83 fa 01 75 aa
<0f>
> > 0b eb a6 65 48 8b 14 25 c0 6b 01 00 65 48 01 35 88 39 3f 45 48
> > [ 690.226668] RSP: 0018:ffffb2d90efc3b40 EFLAGS: 00010246
> > [ 690.233072] RAX: 0000000000000000 RBX: ffff9654c6ad9f40 RCX:
> > 0000000000000801
> > [ 690.241617] RDX: 0000000000000001 RSI: ffff9654c6ad9f00 RDI:
> > ffff9654c6ad9f40
> > [ 690.250146] RBP: 0000000000000000 R08: 0000000000000002 R09:
> > 0000000000000000
> > [ 690.258667] R10: 00000000000303a0 R11: ffff96757ffd4000 R12:
> > ffffb2d90efc3e08
> > [ 690.267183] R13: ffff9654c6ad9f00 R14: ffffb2d90efc3e08 R15:
> > 0000000000000000
> > [ 690.275696] FS: 00007fa8e005da80(0000) GS:ffff9654ffac0000(0000)
> > knlGS:0000000000000000
> > [ 690.285282] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [ 690.292252] CR2: 00007fa8cc05b3c0 CR3: 0000001fc8518001 CR4:
> > 00000000001606e0
> > [ 690.300770] Call Trace:
> > [ 690.304123] xfs_submit_ioend+0x84/0x1e0 [xfs]
> > [ 690.309668] xfs_do_writepage+0x246/0x5a0 [xfs]
> > [ 690.315279] write_cache_pages+0x16b/0x440
> > [ 690.320420] ? xfs_vm_releasepage+0x80/0x80 [xfs]
> > [ 690.326238] xfs_vm_writepages+0x6b/0xa0 [xfs]
> > [ 690.331742] do_writepages+0x4b/0xf0
> > [ 690.336275] ? iomap_file_buffered_write+0x62/0x90
> > [ 690.342162] ? iomap_write_end+0x240/0x240
> > [ 690.347295] ? xfs_iunlock+0x165/0x220 [xfs]
> > [ 690.352595] ? __filemap_fdatawrite_range+0xc1/0x100
> > [ 690.358673] __filemap_fdatawrite_range+0xc1/0x100
> > [ 690.364571] ? __switch_to_asm+0x34/0x70
> > [ 690.369484] file_write_and_wait_range+0x4c/0xa0
> > [ 690.375198] xfs_file_fsync+0x5d/0x230 [xfs]
> > [ 690.380499] do_fsync+0x38/0x60
> > [ 690.384534] __x64_sys_fsync+0x10/0x20
> > [ 690.389243] do_syscall_64+0x5b/0x1e0
> > [ 690.393857] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > [ 690.400020] RIP: 0033:0x7fa8cbe4f70d
> > [ 690.404533] Code: bc 20 00 00 75 10 b8 4a 00 00 00 0f 05 48 3d 01 f0 ff
> > ff 73 31 c3 48 83 ec 08 e8 6e f7 ff ff 48 89 04 24 b8 4a 00 00 00 0f 05
<48>
> > 8b 3c 24 48 89 c2 e8 b7 f7 ff ff 48 89 d0 48 83 c4 08 48 3d 01
> > [ 690.426581] RSP: 002b:00007ffed6954160 EFLAGS: 00000293 ORIG_RAX:
> > 000000000000004a
> > [ 690.435583] RAX: ffffffffffffffda RBX: 00005573363790f0 RCX:
> > 00007fa8cbe4f70d
> > [ 690.444102] RDX: 0000000000200000 RSI: 00005573363790f0 RDI:
> > 0000000000000003
> > [ 690.452615] RBP: 00007fa8b337f418 R08: 0000000000000008 R09:
> > 0000000000000001
> > [ 690.461124] R10: 00000000897b0000 R11: 0000000000000293 R12:
> > 0000000000000000
> > [ 690.469631] R13: 0000001000000000 R14: 00007fa8c81dda50 R15:
> > 0000000000000000
> > [ 690.478130] ---[ end trace 90dd4ee6d5cdfcb8 ]---
> I have just checked xfs code, and looks the only reason is that bio size
> is overflowed, that means the whole bio may reach 4GB. That might be
> possible in case of writeback. But I have to double check if that is
> the case, could you test the following patch and provide the log?
>
> diff --git a/block/blk-core.c b/block/blk-core.c
> index 8340f69670d8..9d61e408c608 100644
> --- a/block/blk-core.c
> +++ b/block/blk-core.c
> @@ -1125,6 +1125,19 @@ blk_qc_t submit_bio(struct bio *bio)
> bio_devname(bio, b), count);
> }
> }
> + if (WARN_ON(bio->bi_iter.bi_size == 0 && bio_op(bio) == REQ_OP_WRITE))
{
> + int i;
> + printk("bio: %hx/%hx %u %llu %u\n", bio->bi_flags, bio->bi_opf,
> + bio->bi_vcnt,
> + (unsigned long long)bio->bi_iter.bi_sector,
> + bio->bi_iter.bi_size);
> + for (i = 0; i < bio->bi_vcnt; i++) {
> + struct bio_vec *bvec = &bio->bi_io_vec[i];
> + printk("%d: pfn %lu offset %u len %u\n", i++,
> + (unsigned long)page_to_pfn(bvec->bv_page),
> + bvec->bv_offset, bvec->bv_len);
> + }
> + }
> return generic_make_request(bio);
> }
the newest log and i also see you fix patch, i will test your fix patch
more times to check it.
------
[ 54.656016] 2019-07-01 11:48:52 mkfs -t xfs /dev/sdb1
[ 54.656017]
[ 119.228467] WARNING: CPU: 5 PID: 1530 at block/blk-core.c:1128
submit_bio+0xa5/0x1d0
[ 119.240810] Modules linked in: dm_mod btrfs xor zstd_decompress
zstd_compress raid6_pq libcrc32c sd_mod intel_rapl sg sb_edac
x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
mgag200 crct10dif_pclmul snd_pcm crc32_pclmul ttm crc32c_intel snd_timer
drm_kms_helper ghash_clmulni_intel snd syscopyarea ipmi_si sysfillrect
soundcore aesni_intel sysimgblt ipmi_devintf crypto_simd fb_sys_fops ahci
mei_me libahci cryptd ipmi_msghandler pcspkr glue_helper drm mei libata
joydev ioatdma lpc_ich i2c_i801 dca wmi acpi_power_meter acpi_pad
pcc_cpufreq ip_tables
[ 119.306252] CPU: 5 PID: 1530 Comm: mkfs.xfs Tainted: G W I
5.2.0-rc5-00001-g82f8097 #1
[ 119.318274] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
SE5C610.86B.01.01.0008.021120151325 02/11/2015
[ 119.331852] RIP: 0010:submit_bio+0xa5/0x1d0
[ 119.338422] Code: 00 00 65 48 01 35 b3 39 5f 76 8b 0d a1 96 c4 01 85 c9
0f 85 9b 00 00 00 8b 43 28 85 c0 75 a5 8b 53 10 0f b6 ca 83 f9 01 75 9a <0f>
0b 0f b7 4b 68 0f b7 73 14 45 31 c9 4c 8b 43 20 48 c7 c7 f1 99
[ 119.363296] RSP: 0018:ffff940c4fb17e58 EFLAGS: 00010246
[ 119.371107] RAX: 0000000000000000 RBX: ffff88a53e3d6900 RCX:
0000000000000001
[ 119.381068] RDX: 0000000000040801 RSI: ffff88868138c000 RDI:
ffff88a53e3d6900
[ 119.391041] RBP: ffff88a5448b0000 R08: ffff88a53e3d6900 R09:
0000000000000000
[ 119.401003] R10: 0000000000000000 R11: 0000000000000000 R12:
0000000000000000
[ 119.410967] R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
[ 119.420881] FS: 00007f37eba81780(0000) GS:ffff88a57f340000(0000)
knlGS:0000000000000000
[ 119.431839] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 119.440161] CR2: 00005630eb09d000 CR3: 0000001fc711c002 CR4:
00000000001606e0
[ 119.450017] Call Trace:
[ 119.454595] submit_bio_wait+0x53/0x80
[ 119.460636] blkdev_issue_flush+0x81/0xb0
[ 119.466930] blkdev_fsync+0x2f/0x40
[ 119.472590] do_fsync+0x38/0x60
[ 119.477810] __x64_sys_fsync+0x10/0x20
[ 119.483722] do_syscall_64+0x5b/0x1e0
[ 119.489516] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 119.496857] RIP: 0033:0x7f37eb2146f0
[ 119.502502] Code: 48 3d 01 f0 ff ff 73 01 c3 48 8b 0d b2 78 20 00 f7 d8
64 89 01 48 83 c8 ff c3 83 3d d9 bc 20 00 00 75 10 b8 4a 00 00 00 0f 05 <48>
3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 6e f7 ff ff 48 89 04 24
[ 119.526842] RSP: 002b:00007fffb27f4878 EFLAGS: 00000246 ORIG_RAX:
000000000000004a
[ 119.536975] RAX: ffffffffffffffda RBX: 0000000000000811 RCX:
00007f37eb2146f0
[ 119.546592] RDX: 00005630e91ba8e0 RSI: 0000000000000000 RDI:
0000000000000004
[ 119.556171] RBP: 0000000000000004 R08: 00005630eb02b8a0 R09:
00007fffb27f4e00
[ 119.565753] R10: 000000000000009b R11: 0000000000000246 R12:
00007fffb27f4c60
[ 119.575339] R13: 00007fffb27f4a30 R14: 00005630eb048800 R15:
00005630eb05a800
[ 119.584940] ---[ end trace 46a2e1214c930529 ]---
[ 119.591765] bio: 0/801 0 0 0
Hi Yiding,
The above warning is false positive, and it is flush request, and
it isn't same with your original report in which the following warning
is triggered.
"drivers/scsi/scsi_lib.c:1034 scsi_init_io+0x11a/0x2e0"
So please run the test kernel with the debug patch I sent to you today
and try to trigger the warning in scsi_init_io().
Thanks,
Ming
11:54 p.m.
在 2019/7/1 下午2:44, Ming Lei 写道:
On Mon, Jul 01, 2019 at 12:49:51PM +0800, Liu Yiding wrote:
> 在 2019/7/1 上午10:47, Ming Lei 写道:
>> [snip]
>>>>> ------
>>>>>
>>>>> 2412 [ 112.558155] XFS (sdb1): Ending clean mount
>>>>> 2413 [ 1317.330820] WARNING: CPU: 63 PID: 1495 at
>>>>> drivers/scsi/scsi_lib.c:1034 scsi_init_io+0x11a/0x2e0
>>>>> 2414 [ 1317.344000] Modules linked in: xfs dm_mod btrfs xor
zstd_decompress
>>>>> zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac
>>>>> x86_pkg_temp_thermal intel_powerclamp coretemp kvm_i ntel kvm
irqbypass
>>>>> crct10dif_pclmul crc32_pclmul crc32c_intel mgag200 snd_pcm
>>>>> ghash_clmulni_intel ttm snd_timer drm_kms_helper ipmi_si ahci snd
>>>>> aesni_intel syscopyarea libahci ipmi_devi ntf sysfillrect
sysimgblt
>>>>> crypto_simd mei_me fb_sys_fops soundcore ioatdma cryptd
ipmi_msghandler
>>>>> glue_helper pcspkr drm libata mei joydev lpc_ich i2c_i801 dca wmi
>>>>> acpi_power_meter acpi _pad pcc_cpufreq ip_tables
>>>>> 2415 [ 1317.408135] CPU: 63 PID: 1495 Comm: kworker/63:1H Tainted:
>>>>> G I 5.2.0-rc5-00001-gc2d55e2 #1
>>>>> 2416 [ 1317.420350] Hardware name: Intel Corporation
S2600WTT/S2600WTT, BIOS
>>>>> SE5C610.86B.01.01.0008.021120151325 02/11/2015
>>>>> 2417 [ 1317.433545] Workqueue: kblockd blk_mq_run_work_fn
>>>>> 2418 [ 1317.440295] RIP: 0010:scsi_init_io+0x11a/0x2e0
>>>>> 2419 [ 1317.446746] Code: be 01 00 00 00 e9 44 ff ff ff 8b 45 70 89
83 08 01
>>>>> 00 00 f6 45 1a 01 75 8a 31 c0 48 83 c4 08 5b 5d 41 5c 41 5d 41 5e 41
5f c3
>>>>> <0f> 0b 8b 4d 18 8b 55 28 45 31 c9 48 c7 c6 98 da 69 82 48
c7 c7 f0
>>>>> 2420 [ 1317.470863] RSP: 0018:ffffa5fa4f7e7c68 EFLAGS: 00010246
>>>>> 2421 [ 1317.478294] RAX: 000000003d17d1b8 RBX: ffff8ae6cb742690 RCX:
>>>>> 0000000000000000
>>>>> 2422 [ 1317.487878] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
>>>>> ffff8ae6cb742690
>>>>> 2423 [ 1317.497469] RBP: ffff8ae6cb742580 R08: 0000000000023003 R09:
>>>>> ffff8ae6cb7426c8
>>>>> 2424 [ 1317.507057] R10: ffffa5fa4c7d7da0 R11: fefefefefefefeff R12:
>>>>> 0000000000000801
>>>>> 2425 [ 1317.516650] R13: 0000000000000000 R14: 0000000000000000 R15:
>>>>> ffff8ac800bd1800
>>>>> 2426 [ 1317.526253] FS: 0000000000000000(0000)
GS:ffff8b077fac0000(0000)
>>>>> knlGS:0000000000000000
>>>>> 2427 [ 1317.536918] CS: 0010 DS: 0000 ES: 0000 CR0:
0000000080050033
>>>>> 2428 [ 1317.544952] CR2: 00007f39a5818004 CR3: 000000343440a006 CR4:
>>>>> 00000000001606e0
>>>>> 2429 [ 1317.554554] Call Trace:
>>>>> 2430 [ 1317.558921] sd_init_command+0x2ee/0xaf0 [sd_mod]
>>>>> 2431 [ 1317.565800] scsi_queue_rq+0x559/0xa80
>>>>> 2432 [ 1317.571607] blk_mq_dispatch_rq_list+0x3da/0x5b0
>>>>> 2433 [ 1317.578375] ? syscall_return_via_sysret+0x10/0x7f
>>>>> 2434 [ 1317.585339] ? __switch_to_asm+0x40/0x70
>>>>> 2435 [ 1317.591321] ? elv_rb_del+0x1f/0x30
>>>>> 2436 [ 1317.596807] ? deadline_remove_request+0x55/0xc0
>>>>> 2437 [ 1317.603560] blk_mq_do_dispatch_sched+0x76/0x120
>>>>> 2438 [ 1317.610302] blk_mq_sched_dispatch_requests+0x100/0x170
>>>>> 2439 [ 1317.617733] __blk_mq_run_hw_queue+0x60/0x130
>>>>> 2440 [ 1317.624200] process_one_work+0x19c/0x3c0
>>>>> 2441 [ 1317.630258] worker_thread+0x3c/0x3b0
>>>>> 2442 [ 1317.635937] ? process_one_work+0x3c0/0x3c0
>>>>> 2443 [ 1317.642183] kthread+0x11e/0x140
>>>>> 2444 [ 1317.647346] ? kthread_park+0x90/0x90
>>>>> 2445 [ 1317.652986] ret_from_fork+0x35/0x40
>>>>> 2446 [ 1317.658523] ---[ end trace 61026ff97261da6a ]---
>>>>> 2447 [ 1317.665327] scsi_dump_rq: rq 6(f:801/23003, seg: 0)
>>>>> 2448 [ 1317.671601] bio 0: d602/801 0 1024971192 0
>>>>> 2449 [ 1317.676991] print_req_error: I/O error, dev sdb, sector
1024971192
>>>>> flags 801
>>>>> 2450 [ 1317.685722] scsi_dump_rq: rq 8(f:801/23003, seg: 0)
>>>>> 2451 [ 1317.691994] bio 0: d602/801 0 1033359800 0
>>>> The above line shows the bio is zero-size, and this bio is submitted
>>>> from bio, since the bio->bi_flags indicates that BIO_CLONED isn't
set.
>>>>
>>>> Please apply the following debug patch, and we may see where is wrong.
>>>>
>>>> diff --git a/block/blk-core.c b/block/blk-core.c
>>>> index 8340f69670d8..86bb61b0df83 100644
>>>> --- a/block/blk-core.c
>>>> +++ b/block/blk-core.c
>>>> @@ -1125,6 +1125,7 @@ blk_qc_t submit_bio(struct bio *bio)
>>>> bio_devname(bio, b), count);
>>>> }
>>>> }
>>>> + WARN_ON(bio->bi_iter.bi_size == 0 && bio_op(bio) ==
REQ_OP_WRITE);
>>>> return generic_make_request(bio);
>>> Sorry for delay,
>>>
>>> ------
>>>
>>> [ 116.619136] 2019-06-29 16:12:41 mkdir -p /fs/sdb1
>>> [ 116.619138]
>>> [ 116.628743] 2019-06-29 16:12:41 mount -t xfs -o inode64 /dev/sdb1
>>> /fs/sdb1
>>> [ 116.628745]
>>> [ 116.634699] SGI XFS with ACLs, security attributes, realtime, scrub,
>>> debug enabled
>>> [ 116.653110] XFS (sdb1): Mounting V5 Filesystem
>>> [ 116.670081] XFS (sdb1): Ending clean mount
>>> [ 690.106779] WARNING: CPU: 53 PID: 1477 at block/blk-core.c:1128
>>> submit_bio+0x93/0x150
>>> [ 690.117972] Modules linked in: xfs dm_mod btrfs xor zstd_decompress
>>> zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac
>>> x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
>>> mgag200 crct10dif_pclmul crc32_pclmul ttm crc32c_intel snd_pcm
>>> ghash_clmulni_intel drm_kms_helper snd_timer ahci syscopyarea sysfillrect
>>> snd aesni_intel sysimgblt libahci ipmi_si soundcore fb_sys_fops mei_me
>>> crypto_simd ipmi_devintf ioatdma cryptd pcspkr drm ipmi_msghandler
>>> glue_helper libata mei joydev lpc_ich i2c_i801 dca wmi acpi_power_meter
>>> acpi_pad pcc_cpufreq ip_tables
>>> [ 690.176967] CPU: 53 PID: 1477 Comm: fio Tainted: G W I
>>> 5.2.0-rc5-00002-g9fa09bd #1
>>> [ 690.187198] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
>>> SE5C610.86B.01.01.0008.021120151325 02/11/2015
>>> [ 690.199375] RIP: 0010:submit_bio+0x93/0x150
>>> [ 690.204583] Code: ee 74 45 83 e1 01 74 26 65 48 01 35 bf 39 3f 45 8b 0d
>>> ad 96 c4 01 85 c9 75 56 8b 43 28 85 c0 75 b3 0f b6 53 10 83 fa 01 75 aa
<0f>
>>> 0b eb a6 65 48 8b 14 25 c0 6b 01 00 65 48 01 35 88 39 3f 45 48
>>> [ 690.226668] RSP: 0018:ffffb2d90efc3b40 EFLAGS: 00010246
>>> [ 690.233072] RAX: 0000000000000000 RBX: ffff9654c6ad9f40 RCX:
>>> 0000000000000801
>>> [ 690.241617] RDX: 0000000000000001 RSI: ffff9654c6ad9f00 RDI:
>>> ffff9654c6ad9f40
>>> [ 690.250146] RBP: 0000000000000000 R08: 0000000000000002 R09:
>>> 0000000000000000
>>> [ 690.258667] R10: 00000000000303a0 R11: ffff96757ffd4000 R12:
>>> ffffb2d90efc3e08
>>> [ 690.267183] R13: ffff9654c6ad9f00 R14: ffffb2d90efc3e08 R15:
>>> 0000000000000000
>>> [ 690.275696] FS: 00007fa8e005da80(0000) GS:ffff9654ffac0000(0000)
>>> knlGS:0000000000000000
>>> [ 690.285282] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> [ 690.292252] CR2: 00007fa8cc05b3c0 CR3: 0000001fc8518001 CR4:
>>> 00000000001606e0
>>> [ 690.300770] Call Trace:
>>> [ 690.304123] xfs_submit_ioend+0x84/0x1e0 [xfs]
>>> [ 690.309668] xfs_do_writepage+0x246/0x5a0 [xfs]
>>> [ 690.315279] write_cache_pages+0x16b/0x440
>>> [ 690.320420] ? xfs_vm_releasepage+0x80/0x80 [xfs]
>>> [ 690.326238] xfs_vm_writepages+0x6b/0xa0 [xfs]
>>> [ 690.331742] do_writepages+0x4b/0xf0
>>> [ 690.336275] ? iomap_file_buffered_write+0x62/0x90
>>> [ 690.342162] ? iomap_write_end+0x240/0x240
>>> [ 690.347295] ? xfs_iunlock+0x165/0x220 [xfs]
>>> [ 690.352595] ? __filemap_fdatawrite_range+0xc1/0x100
>>> [ 690.358673] __filemap_fdatawrite_range+0xc1/0x100
>>> [ 690.364571] ? __switch_to_asm+0x34/0x70
>>> [ 690.369484] file_write_and_wait_range+0x4c/0xa0
>>> [ 690.375198] xfs_file_fsync+0x5d/0x230 [xfs]
>>> [ 690.380499] do_fsync+0x38/0x60
>>> [ 690.384534] __x64_sys_fsync+0x10/0x20
>>> [ 690.389243] do_syscall_64+0x5b/0x1e0
>>> [ 690.393857] entry_SYSCALL_64_after_hwframe+0x44/0xa9
>>> [ 690.400020] RIP: 0033:0x7fa8cbe4f70d
>>> [ 690.404533] Code: bc 20 00 00 75 10 b8 4a 00 00 00 0f 05 48 3d 01 f0 ff
>>> ff 73 31 c3 48 83 ec 08 e8 6e f7 ff ff 48 89 04 24 b8 4a 00 00 00 0f 05
<48>
>>> 8b 3c 24 48 89 c2 e8 b7 f7 ff ff 48 89 d0 48 83 c4 08 48 3d 01
>>> [ 690.426581] RSP: 002b:00007ffed6954160 EFLAGS: 00000293 ORIG_RAX:
>>> 000000000000004a
>>> [ 690.435583] RAX: ffffffffffffffda RBX: 00005573363790f0 RCX:
>>> 00007fa8cbe4f70d
>>> [ 690.444102] RDX: 0000000000200000 RSI: 00005573363790f0 RDI:
>>> 0000000000000003
>>> [ 690.452615] RBP: 00007fa8b337f418 R08: 0000000000000008 R09:
>>> 0000000000000001
>>> [ 690.461124] R10: 00000000897b0000 R11: 0000000000000293 R12:
>>> 0000000000000000
>>> [ 690.469631] R13: 0000001000000000 R14: 00007fa8c81dda50 R15:
>>> 0000000000000000
>>> [ 690.478130] ---[ end trace 90dd4ee6d5cdfcb8 ]---
>> I have just checked xfs code, and looks the only reason is that bio size
>> is overflowed, that means the whole bio may reach 4GB. That might be
>> possible in case of writeback. But I have to double check if that is
>> the case, could you test the following patch and provide the log?
>>
>> diff --git a/block/blk-core.c b/block/blk-core.c
>> index 8340f69670d8..9d61e408c608 100644
>> --- a/block/blk-core.c
>> +++ b/block/blk-core.c
>> @@ -1125,6 +1125,19 @@ blk_qc_t submit_bio(struct bio *bio)
>> bio_devname(bio, b), count);
>> }
>> }
>> + if (WARN_ON(bio->bi_iter.bi_size == 0 && bio_op(bio) ==
REQ_OP_WRITE)) {
>> + int i;
>> + printk("bio: %hx/%hx %u %llu %u\n", bio->bi_flags,
bio->bi_opf,
>> + bio->bi_vcnt,
>> + (unsigned long long)bio->bi_iter.bi_sector,
>> + bio->bi_iter.bi_size);
>> + for (i = 0; i < bio->bi_vcnt; i++) {
>> + struct bio_vec *bvec = &bio->bi_io_vec[i];
>> + printk("%d: pfn %lu offset %u len %u\n", i++,
>> + (unsigned long)page_to_pfn(bvec->bv_page),
>> + bvec->bv_offset, bvec->bv_len);
>> + }
>> + }
>> return generic_make_request(bio);
>> }
> the newest log and i also see you fix patch, i will test your fix patch
> more times to check it.
>
> ------
>
> [ 54.656016] 2019-07-01 11:48:52 mkfs -t xfs /dev/sdb1
> [ 54.656017]
> [ 119.228467] WARNING: CPU: 5 PID: 1530 at block/blk-core.c:1128
> submit_bio+0xa5/0x1d0
> [ 119.240810] Modules linked in: dm_mod btrfs xor zstd_decompress
> zstd_compress raid6_pq libcrc32c sd_mod intel_rapl sg sb_edac
> x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
> mgag200 crct10dif_pclmul snd_pcm crc32_pclmul ttm crc32c_intel snd_timer
> drm_kms_helper ghash_clmulni_intel snd syscopyarea ipmi_si sysfillrect
> soundcore aesni_intel sysimgblt ipmi_devintf crypto_simd fb_sys_fops ahci
> mei_me libahci cryptd ipmi_msghandler pcspkr glue_helper drm mei libata
> joydev ioatdma lpc_ich i2c_i801 dca wmi acpi_power_meter acpi_pad
> pcc_cpufreq ip_tables
> [ 119.306252] CPU: 5 PID: 1530 Comm: mkfs.xfs Tainted: G W I
> 5.2.0-rc5-00001-g82f8097 #1
> [ 119.318274] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
> SE5C610.86B.01.01.0008.021120151325 02/11/2015
> [ 119.331852] RIP: 0010:submit_bio+0xa5/0x1d0
> [ 119.338422] Code: 00 00 65 48 01 35 b3 39 5f 76 8b 0d a1 96 c4 01 85 c9
> 0f 85 9b 00 00 00 8b 43 28 85 c0 75 a5 8b 53 10 0f b6 ca 83 f9 01 75 9a <0f>
> 0b 0f b7 4b 68 0f b7 73 14 45 31 c9 4c 8b 43 20 48 c7 c7 f1 99
> [ 119.363296] RSP: 0018:ffff940c4fb17e58 EFLAGS: 00010246
> [ 119.371107] RAX: 0000000000000000 RBX: ffff88a53e3d6900 RCX:
> 0000000000000001
> [ 119.381068] RDX: 0000000000040801 RSI: ffff88868138c000 RDI:
> ffff88a53e3d6900
> [ 119.391041] RBP: ffff88a5448b0000 R08: ffff88a53e3d6900 R09:
> 0000000000000000
> [ 119.401003] R10: 0000000000000000 R11: 0000000000000000 R12:
> 0000000000000000
> [ 119.410967] R13: 0000000000000000 R14: 0000000000000000 R15:
> 0000000000000000
> [ 119.420881] FS: 00007f37eba81780(0000) GS:ffff88a57f340000(0000)
> knlGS:0000000000000000
> [ 119.431839] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 119.440161] CR2: 00005630eb09d000 CR3: 0000001fc711c002 CR4:
> 00000000001606e0
> [ 119.450017] Call Trace:
> [ 119.454595] submit_bio_wait+0x53/0x80
> [ 119.460636] blkdev_issue_flush+0x81/0xb0
> [ 119.466930] blkdev_fsync+0x2f/0x40
> [ 119.472590] do_fsync+0x38/0x60
> [ 119.477810] __x64_sys_fsync+0x10/0x20
> [ 119.483722] do_syscall_64+0x5b/0x1e0
> [ 119.489516] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [ 119.496857] RIP: 0033:0x7f37eb2146f0
> [ 119.502502] Code: 48 3d 01 f0 ff ff 73 01 c3 48 8b 0d b2 78 20 00 f7 d8
> 64 89 01 48 83 c8 ff c3 83 3d d9 bc 20 00 00 75 10 b8 4a 00 00 00 0f 05 <48>
> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 6e f7 ff ff 48 89 04 24
> [ 119.526842] RSP: 002b:00007fffb27f4878 EFLAGS: 00000246 ORIG_RAX:
> 000000000000004a
> [ 119.536975] RAX: ffffffffffffffda RBX: 0000000000000811 RCX:
> 00007f37eb2146f0
> [ 119.546592] RDX: 00005630e91ba8e0 RSI: 0000000000000000 RDI:
> 0000000000000004
> [ 119.556171] RBP: 0000000000000004 R08: 00005630eb02b8a0 R09:
> 00007fffb27f4e00
> [ 119.565753] R10: 000000000000009b R11: 0000000000000246 R12:
> 00007fffb27f4c60
> [ 119.575339] R13: 00007fffb27f4a30 R14: 00005630eb048800 R15:
> 00005630eb05a800
> [ 119.584940] ---[ end trace 46a2e1214c930529 ]---
> [ 119.591765] bio: 0/801 0 0 0
>
Hi Yiding,
The above warning is false positive, and it is flush request, and
it isn't same with your original report in which the following warning
is triggered.
"drivers/scsi/scsi_lib.c:1034 scsi_init_io+0x11a/0x2e0"
So please run the test kernel with the debug patch I sent to you today
and try to trigger the warning in scsi_init_io().
------
[ 728.863669] 230: pfn 66784668 offset 0 len 17031168
[ 728.869319] 232: pfn 66788826 offset 0 len 22708224
[ 728.874946] 234: pfn 66794370 offset 0 len 29933568
[ 728.880563] 236: pfn 66801678 offset 0 len 4136960
[ 728.886086] 238: pfn 66820096 offset 0 len 2506752
[ 729.336992] WARNING: CPU: 40 PID: 1312 at
drivers/scsi/scsi_lib.c:1010 scsi_init_io+0x103/0x170
[ 729.349150] Modules linked in: xfs dm_mod btrfs xor zstd_decompress
zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac
x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
crct10dif_pclmul mgag200 crc32_pclmul ttm crc32c_intel
ghash_clmulni_intel drm_kms_helper snd_pcm syscopyarea sysfillrect
sysimgblt aesni_intel snd_timer ahci crypto_simd ipmi_si fb_sys_fops
mei_me libahci snd ipmi_devintf cryptd soundcore glue_helper
ipmi_msghandler pcspkr drm ioatdma mei libata joydev lpc_ich i2c_i801
dca wmi acpi_pad acpi_power_meter pcc_cpufreq ip_tables
[ 729.409451] CPU: 40 PID: 1312 Comm: kworker/40:1H Tainted: G W
I 5.2.0-rc5-00001-g82f8097 #1
[ 729.421013] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
SE5C610.86B.01.01.0008.021120151325 02/11/2015
[ 729.433604] Workqueue: kblockd blk_mq_run_work_fn
[ 729.439805] RIP: 0010:scsi_init_io+0x103/0x170
[ 729.445726] Code: 5c 41 5d c3 be 01 00 00 00 e9 49 ff ff ff 8b 45 70
89 83 08 01 00 00 f6 45 1a 01 75 92 31 c0 48 83 c4 08 5b 5d 41 5c 41 5d
c3 <0f> 0b 48 83 c4 08 b8 0a 00 00 00 5b 5d 41 5c 41 5d c3 48 8b 75 38
[ 729.468787] RSP: 0018:ffffac02cea47c78 EFLAGS: 00010246
[ 729.475705] RAX: 000000001eafb2a8 RBX: ffff95df3e7a3950 RCX:
0000000000000000
[ 729.484790] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
ffff95df3e7a3950
[ 729.493889] RBP: ffff95df3e7a3840 R08: 0000000000000000 R09:
ffff95df3e7a3988
[ 729.502995] R10: ffffac02cc71fda0 R11: fefefefefefefeff R12:
0000000000000801
[ 729.512123] R13: 0000000000000000 R14: 0000000000000000 R15:
ffff95df3ba8e800
[ 729.521261] FS: 0000000000000000(0000) GS:ffff95df7f780000(0000)
knlGS:0000000000000000
[ 729.531505] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 729.539140] CR2: 00007f7465922814 CR3: 0000000a80c0a006 CR4:
00000000001606e0
[ 729.548359] Call Trace:
[ 729.552355] sd_init_command+0x2ee/0xaf0 [sd_mod]
[ 729.558882] scsi_queue_rq+0x559/0xa80
[ 729.564354] blk_mq_dispatch_rq_list+0x3da/0x5b0
[ 729.570820] ? syscall_return_via_sysret+0x10/0x7f
[ 729.577488] ? __switch_to_asm+0x40/0x70
[ 729.583192] ? elv_rb_del+0x1f/0x30
[ 729.588417] ? deadline_remove_request+0x55/0xc0
[ 729.594915] blk_mq_do_dispatch_sched+0x76/0x120
[ 729.601424] blk_mq_sched_dispatch_requests+0x100/0x170
[ 729.608626] __blk_mq_run_hw_queue+0x60/0x130
[ 729.614873] process_one_work+0x19c/0x3c0
[ 729.620734] worker_thread+0x3c/0x3b0
[ 729.626210] ? process_one_work+0x3c0/0x3c0
[ 729.632280] kthread+0x11e/0x140
[ 729.637281] ? kthread_park+0x90/0x90
[ 729.642773] ret_from_fork+0x35/0x40
[ 729.648172] ---[ end trace 929c66c4a933d3e4 ]---
[ 729.654870] print_req_error: I/O error, dev sdb, sector 514831016
flags 801
------
Thanks,
Ming
12:14 a.m.
On Mon, Jul 01, 2019 at 02:54:31PM +0800, Liu Yiding wrote:
在 2019/7/1 下午2:44, Ming Lei 写道:
> On Mon, Jul 01, 2019 at 12:49:51PM +0800, Liu Yiding wrote:
> > 在 2019/7/1 上午10:47, Ming Lei 写道:
> > > [snip]
> > > > > > ------
> > > > > >
> > > > > > 2412 [ 112.558155] XFS (sdb1): Ending clean mount
> > > > > > 2413 [ 1317.330820] WARNING: CPU: 63 PID: 1495 at
> > > > > > drivers/scsi/scsi_lib.c:1034 scsi_init_io+0x11a/0x2e0
> > > > > > 2414 [ 1317.344000] Modules linked in: xfs dm_mod btrfs xor
zstd_decompress
> > > > > > zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl
sb_edac
> > > > > > x86_pkg_temp_thermal intel_powerclamp coretemp kvm_i
ntel kvm irqbypass
> > > > > > crct10dif_pclmul crc32_pclmul crc32c_intel mgag200 snd_pcm
> > > > > > ghash_clmulni_intel ttm snd_timer drm_kms_helper ipmi_si
ahci snd
> > > > > > aesni_intel syscopyarea libahci ipmi_devi ntf
sysfillrect sysimgblt
> > > > > > crypto_simd mei_me fb_sys_fops soundcore ioatdma cryptd
ipmi_msghandler
> > > > > > glue_helper pcspkr drm libata mei joydev lpc_ich i2c_i801
dca wmi
> > > > > > acpi_power_meter acpi _pad pcc_cpufreq ip_tables
> > > > > > 2415 [ 1317.408135] CPU: 63 PID: 1495 Comm: kworker/63:1H
Tainted:
> > > > > > G I 5.2.0-rc5-00001-gc2d55e2 #1
> > > > > > 2416 [ 1317.420350] Hardware name: Intel Corporation
S2600WTT/S2600WTT, BIOS
> > > > > > SE5C610.86B.01.01.0008.021120151325 02/11/2015
> > > > > > 2417 [ 1317.433545] Workqueue: kblockd blk_mq_run_work_fn
> > > > > > 2418 [ 1317.440295] RIP: 0010:scsi_init_io+0x11a/0x2e0
> > > > > > 2419 [ 1317.446746] Code: be 01 00 00 00 e9 44 ff ff ff 8b
45 70 89 83 08 01
> > > > > > 00 00 f6 45 1a 01 75 8a 31 c0 48 83 c4 08 5b 5d 41 5c 41 5d
41 5e 41 5f c3
> > > > > > <0f> 0b 8b 4d 18 8b 55 28 45 31 c9 48 c7 c6 98
da 69 82 48 c7 c7 f0
> > > > > > 2420 [ 1317.470863] RSP: 0018:ffffa5fa4f7e7c68 EFLAGS:
00010246
> > > > > > 2421 [ 1317.478294] RAX: 000000003d17d1b8 RBX:
ffff8ae6cb742690 RCX:
> > > > > > 0000000000000000
> > > > > > 2422 [ 1317.487878] RDX: 0000000000000000 RSI:
0000000000000000 RDI:
> > > > > > ffff8ae6cb742690
> > > > > > 2423 [ 1317.497469] RBP: ffff8ae6cb742580 R08:
0000000000023003 R09:
> > > > > > ffff8ae6cb7426c8
> > > > > > 2424 [ 1317.507057] R10: ffffa5fa4c7d7da0 R11:
fefefefefefefeff R12:
> > > > > > 0000000000000801
> > > > > > 2425 [ 1317.516650] R13: 0000000000000000 R14:
0000000000000000 R15:
> > > > > > ffff8ac800bd1800
> > > > > > 2426 [ 1317.526253] FS: 0000000000000000(0000)
GS:ffff8b077fac0000(0000)
> > > > > > knlGS:0000000000000000
> > > > > > 2427 [ 1317.536918] CS: 0010 DS: 0000 ES: 0000 CR0:
0000000080050033
> > > > > > 2428 [ 1317.544952] CR2: 00007f39a5818004 CR3:
000000343440a006 CR4:
> > > > > > 00000000001606e0
> > > > > > 2429 [ 1317.554554] Call Trace:
> > > > > > 2430 [ 1317.558921] sd_init_command+0x2ee/0xaf0 [sd_mod]
> > > > > > 2431 [ 1317.565800] scsi_queue_rq+0x559/0xa80
> > > > > > 2432 [ 1317.571607] blk_mq_dispatch_rq_list+0x3da/0x5b0
> > > > > > 2433 [ 1317.578375] ? syscall_return_via_sysret+0x10/0x7f
> > > > > > 2434 [ 1317.585339] ? __switch_to_asm+0x40/0x70
> > > > > > 2435 [ 1317.591321] ? elv_rb_del+0x1f/0x30
> > > > > > 2436 [ 1317.596807] ? deadline_remove_request+0x55/0xc0
> > > > > > 2437 [ 1317.603560] blk_mq_do_dispatch_sched+0x76/0x120
> > > > > > 2438 [ 1317.610302]
blk_mq_sched_dispatch_requests+0x100/0x170
> > > > > > 2439 [ 1317.617733] __blk_mq_run_hw_queue+0x60/0x130
> > > > > > 2440 [ 1317.624200] process_one_work+0x19c/0x3c0
> > > > > > 2441 [ 1317.630258] worker_thread+0x3c/0x3b0
> > > > > > 2442 [ 1317.635937] ? process_one_work+0x3c0/0x3c0
> > > > > > 2443 [ 1317.642183] kthread+0x11e/0x140
> > > > > > 2444 [ 1317.647346] ? kthread_park+0x90/0x90
> > > > > > 2445 [ 1317.652986] ret_from_fork+0x35/0x40
> > > > > > 2446 [ 1317.658523] ---[ end trace 61026ff97261da6a ]---
> > > > > > 2447 [ 1317.665327] scsi_dump_rq: rq 6(f:801/23003, seg:
0)
> > > > > > 2448 [ 1317.671601] bio 0: d602/801 0 1024971192 0
> > > > > > 2449 [ 1317.676991] print_req_error: I/O error, dev sdb,
sector 1024971192
> > > > > > flags 801
> > > > > > 2450 [ 1317.685722] scsi_dump_rq: rq 8(f:801/23003, seg:
0)
> > > > > > 2451 [ 1317.691994] bio 0: d602/801 0 1033359800 0
> > > > > The above line shows the bio is zero-size, and this bio is
submitted
> > > > > from bio, since the bio->bi_flags indicates that BIO_CLONED
isn't set.
> > > > >
> > > > > Please apply the following debug patch, and we may see where is
wrong.
> > > > >
> > > > > diff --git a/block/blk-core.c b/block/blk-core.c
> > > > > index 8340f69670d8..86bb61b0df83 100644
> > > > > --- a/block/blk-core.c
> > > > > +++ b/block/blk-core.c
> > > > > @@ -1125,6 +1125,7 @@ blk_qc_t submit_bio(struct bio *bio)
> > > > > bio_devname(bio, b), count);
> > > > > }
> > > > > }
> > > > > + WARN_ON(bio->bi_iter.bi_size == 0 && bio_op(bio) ==
REQ_OP_WRITE);
> > > > > return generic_make_request(bio);
> > > > Sorry for delay,
> > > >
> > > > ------
> > > >
> > > > [ 116.619136] 2019-06-29 16:12:41 mkdir -p /fs/sdb1
> > > > [ 116.619138]
> > > > [ 116.628743] 2019-06-29 16:12:41 mount -t xfs -o inode64 /dev/sdb1
> > > > /fs/sdb1
> > > > [ 116.628745]
> > > > [ 116.634699] SGI XFS with ACLs, security attributes, realtime,
scrub,
> > > > debug enabled
> > > > [ 116.653110] XFS (sdb1): Mounting V5 Filesystem
> > > > [ 116.670081] XFS (sdb1): Ending clean mount
> > > > [ 690.106779] WARNING: CPU: 53 PID: 1477 at block/blk-core.c:1128
> > > > submit_bio+0x93/0x150
> > > > [ 690.117972] Modules linked in: xfs dm_mod btrfs xor
zstd_decompress
> > > > zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac
> > > > x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm
irqbypass
> > > > mgag200 crct10dif_pclmul crc32_pclmul ttm crc32c_intel snd_pcm
> > > > ghash_clmulni_intel drm_kms_helper snd_timer ahci syscopyarea
sysfillrect
> > > > snd aesni_intel sysimgblt libahci ipmi_si soundcore fb_sys_fops
mei_me
> > > > crypto_simd ipmi_devintf ioatdma cryptd pcspkr drm ipmi_msghandler
> > > > glue_helper libata mei joydev lpc_ich i2c_i801 dca wmi
acpi_power_meter
> > > > acpi_pad pcc_cpufreq ip_tables
> > > > [ 690.176967] CPU: 53 PID: 1477 Comm: fio Tainted: G W I
> > > > 5.2.0-rc5-00002-g9fa09bd #1
> > > > [ 690.187198] Hardware name: Intel Corporation S2600WTT/S2600WTT,
BIOS
> > > > SE5C610.86B.01.01.0008.021120151325 02/11/2015
> > > > [ 690.199375] RIP: 0010:submit_bio+0x93/0x150
> > > > [ 690.204583] Code: ee 74 45 83 e1 01 74 26 65 48 01 35 bf 39 3f 45
8b 0d
> > > > ad 96 c4 01 85 c9 75 56 8b 43 28 85 c0 75 b3 0f b6 53 10 83 fa 01 75
aa <0f>
> > > > 0b eb a6 65 48 8b 14 25 c0 6b 01 00 65 48 01 35 88 39 3f 45 48
> > > > [ 690.226668] RSP: 0018:ffffb2d90efc3b40 EFLAGS: 00010246
> > > > [ 690.233072] RAX: 0000000000000000 RBX: ffff9654c6ad9f40 RCX:
> > > > 0000000000000801
> > > > [ 690.241617] RDX: 0000000000000001 RSI: ffff9654c6ad9f00 RDI:
> > > > ffff9654c6ad9f40
> > > > [ 690.250146] RBP: 0000000000000000 R08: 0000000000000002 R09:
> > > > 0000000000000000
> > > > [ 690.258667] R10: 00000000000303a0 R11: ffff96757ffd4000 R12:
> > > > ffffb2d90efc3e08
> > > > [ 690.267183] R13: ffff9654c6ad9f00 R14: ffffb2d90efc3e08 R15:
> > > > 0000000000000000
> > > > [ 690.275696] FS: 00007fa8e005da80(0000) GS:ffff9654ffac0000(0000)
> > > > knlGS:0000000000000000
> > > > [ 690.285282] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > [ 690.292252] CR2: 00007fa8cc05b3c0 CR3: 0000001fc8518001 CR4:
> > > > 00000000001606e0
> > > > [ 690.300770] Call Trace:
> > > > [ 690.304123] xfs_submit_ioend+0x84/0x1e0 [xfs]
> > > > [ 690.309668] xfs_do_writepage+0x246/0x5a0 [xfs]
> > > > [ 690.315279] write_cache_pages+0x16b/0x440
> > > > [ 690.320420] ? xfs_vm_releasepage+0x80/0x80 [xfs]
> > > > [ 690.326238] xfs_vm_writepages+0x6b/0xa0 [xfs]
> > > > [ 690.331742] do_writepages+0x4b/0xf0
> > > > [ 690.336275] ? iomap_file_buffered_write+0x62/0x90
> > > > [ 690.342162] ? iomap_write_end+0x240/0x240
> > > > [ 690.347295] ? xfs_iunlock+0x165/0x220 [xfs]
> > > > [ 690.352595] ? __filemap_fdatawrite_range+0xc1/0x100
> > > > [ 690.358673] __filemap_fdatawrite_range+0xc1/0x100
> > > > [ 690.364571] ? __switch_to_asm+0x34/0x70
> > > > [ 690.369484] file_write_and_wait_range+0x4c/0xa0
> > > > [ 690.375198] xfs_file_fsync+0x5d/0x230 [xfs]
> > > > [ 690.380499] do_fsync+0x38/0x60
> > > > [ 690.384534] __x64_sys_fsync+0x10/0x20
> > > > [ 690.389243] do_syscall_64+0x5b/0x1e0
> > > > [ 690.393857] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > > > [ 690.400020] RIP: 0033:0x7fa8cbe4f70d
> > > > [ 690.404533] Code: bc 20 00 00 75 10 b8 4a 00 00 00 0f 05 48 3d 01
f0 ff
> > > > ff 73 31 c3 48 83 ec 08 e8 6e f7 ff ff 48 89 04 24 b8 4a 00 00 00 0f
05 <48>
> > > > 8b 3c 24 48 89 c2 e8 b7 f7 ff ff 48 89 d0 48 83 c4 08 48 3d 01
> > > > [ 690.426581] RSP: 002b:00007ffed6954160 EFLAGS: 00000293 ORIG_RAX:
> > > > 000000000000004a
> > > > [ 690.435583] RAX: ffffffffffffffda RBX: 00005573363790f0 RCX:
> > > > 00007fa8cbe4f70d
> > > > [ 690.444102] RDX: 0000000000200000 RSI: 00005573363790f0 RDI:
> > > > 0000000000000003
> > > > [ 690.452615] RBP: 00007fa8b337f418 R08: 0000000000000008 R09:
> > > > 0000000000000001
> > > > [ 690.461124] R10: 00000000897b0000 R11: 0000000000000293 R12:
> > > > 0000000000000000
> > > > [ 690.469631] R13: 0000001000000000 R14: 00007fa8c81dda50 R15:
> > > > 0000000000000000
> > > > [ 690.478130] ---[ end trace 90dd4ee6d5cdfcb8 ]---
> > > I have just checked xfs code, and looks the only reason is that bio size
> > > is overflowed, that means the whole bio may reach 4GB. That might be
> > > possible in case of writeback. But I have to double check if that is
> > > the case, could you test the following patch and provide the log?
> > >
> > > diff --git a/block/blk-core.c b/block/blk-core.c
> > > index 8340f69670d8..9d61e408c608 100644
> > > --- a/block/blk-core.c
> > > +++ b/block/blk-core.c
> > > @@ -1125,6 +1125,19 @@ blk_qc_t submit_bio(struct bio *bio)
> > > bio_devname(bio, b), count);
> > > }
> > > }
> > > + if (WARN_ON(bio->bi_iter.bi_size == 0 && bio_op(bio) ==
REQ_OP_WRITE)) {
> > > + int i;
> > > + printk("bio: %hx/%hx %u %llu %u\n", bio->bi_flags,
bio->bi_opf,
> > > + bio->bi_vcnt,
> > > + (unsigned long long)bio->bi_iter.bi_sector,
> > > + bio->bi_iter.bi_size);
> > > + for (i = 0; i < bio->bi_vcnt; i++) {
> > > + struct bio_vec *bvec = &bio->bi_io_vec[i];
> > > + printk("%d: pfn %lu offset %u len %u\n", i++,
> > > + (unsigned
long)page_to_pfn(bvec->bv_page),
> > > + bvec->bv_offset, bvec->bv_len);
> > > + }
> > > + }
> > > return generic_make_request(bio);
> > > }
> > the newest log and i also see you fix patch, i will test your fix patch
> > more times to check it.
> >
> > ------
> >
> > [ 54.656016] 2019-07-01 11:48:52 mkfs -t xfs /dev/sdb1
> > [ 54.656017]
> > [ 119.228467] WARNING: CPU: 5 PID: 1530 at block/blk-core.c:1128
> > submit_bio+0xa5/0x1d0
> > [ 119.240810] Modules linked in: dm_mod btrfs xor zstd_decompress
> > zstd_compress raid6_pq libcrc32c sd_mod intel_rapl sg sb_edac
> > x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
> > mgag200 crct10dif_pclmul snd_pcm crc32_pclmul ttm crc32c_intel snd_timer
> > drm_kms_helper ghash_clmulni_intel snd syscopyarea ipmi_si sysfillrect
> > soundcore aesni_intel sysimgblt ipmi_devintf crypto_simd fb_sys_fops ahci
> > mei_me libahci cryptd ipmi_msghandler pcspkr glue_helper drm mei libata
> > joydev ioatdma lpc_ich i2c_i801 dca wmi acpi_power_meter acpi_pad
> > pcc_cpufreq ip_tables
> > [ 119.306252] CPU: 5 PID: 1530 Comm: mkfs.xfs Tainted: G W I
> > 5.2.0-rc5-00001-g82f8097 #1
> > [ 119.318274] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
> > SE5C610.86B.01.01.0008.021120151325 02/11/2015
> > [ 119.331852] RIP: 0010:submit_bio+0xa5/0x1d0
> > [ 119.338422] Code: 00 00 65 48 01 35 b3 39 5f 76 8b 0d a1 96 c4 01 85 c9
> > 0f 85 9b 00 00 00 8b 43 28 85 c0 75 a5 8b 53 10 0f b6 ca 83 f9 01 75 9a
<0f>
> > 0b 0f b7 4b 68 0f b7 73 14 45 31 c9 4c 8b 43 20 48 c7 c7 f1 99
> > [ 119.363296] RSP: 0018:ffff940c4fb17e58 EFLAGS: 00010246
> > [ 119.371107] RAX: 0000000000000000 RBX: ffff88a53e3d6900 RCX:
> > 0000000000000001
> > [ 119.381068] RDX: 0000000000040801 RSI: ffff88868138c000 RDI:
> > ffff88a53e3d6900
> > [ 119.391041] RBP: ffff88a5448b0000 R08: ffff88a53e3d6900 R09:
> > 0000000000000000
> > [ 119.401003] R10: 0000000000000000 R11: 0000000000000000 R12:
> > 0000000000000000
> > [ 119.410967] R13: 0000000000000000 R14: 0000000000000000 R15:
> > 0000000000000000
> > [ 119.420881] FS: 00007f37eba81780(0000) GS:ffff88a57f340000(0000)
> > knlGS:0000000000000000
> > [ 119.431839] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [ 119.440161] CR2: 00005630eb09d000 CR3: 0000001fc711c002 CR4:
> > 00000000001606e0
> > [ 119.450017] Call Trace:
> > [ 119.454595] submit_bio_wait+0x53/0x80
> > [ 119.460636] blkdev_issue_flush+0x81/0xb0
> > [ 119.466930] blkdev_fsync+0x2f/0x40
> > [ 119.472590] do_fsync+0x38/0x60
> > [ 119.477810] __x64_sys_fsync+0x10/0x20
> > [ 119.483722] do_syscall_64+0x5b/0x1e0
> > [ 119.489516] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > [ 119.496857] RIP: 0033:0x7f37eb2146f0
> > [ 119.502502] Code: 48 3d 01 f0 ff ff 73 01 c3 48 8b 0d b2 78 20 00 f7 d8
> > 64 89 01 48 83 c8 ff c3 83 3d d9 bc 20 00 00 75 10 b8 4a 00 00 00 0f 05
<48>
> > 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 6e f7 ff ff 48 89 04 24
> > [ 119.526842] RSP: 002b:00007fffb27f4878 EFLAGS: 00000246 ORIG_RAX:
> > 000000000000004a
> > [ 119.536975] RAX: ffffffffffffffda RBX: 0000000000000811 RCX:
> > 00007f37eb2146f0
> > [ 119.546592] RDX: 00005630e91ba8e0 RSI: 0000000000000000 RDI:
> > 0000000000000004
> > [ 119.556171] RBP: 0000000000000004 R08: 00005630eb02b8a0 R09:
> > 00007fffb27f4e00
> > [ 119.565753] R10: 000000000000009b R11: 0000000000000246 R12:
> > 00007fffb27f4c60
> > [ 119.575339] R13: 00007fffb27f4a30 R14: 00005630eb048800 R15:
> > 00005630eb05a800
> > [ 119.584940] ---[ end trace 46a2e1214c930529 ]---
> > [ 119.591765] bio: 0/801 0 0 0
> >
> Hi Yiding,
>
> The above warning is false positive, and it is flush request, and
> it isn't same with your original report in which the following warning
> is triggered.
>
> "drivers/scsi/scsi_lib.c:1034 scsi_init_io+0x11a/0x2e0"
>
> So please run the test kernel with the debug patch I sent to you today
> and try to trigger the warning in scsi_init_io().
------
[ 728.863669] 230: pfn 66784668 offset 0 len 17031168
[ 728.869319] 232: pfn 66788826 offset 0 len 22708224
[ 728.874946] 234: pfn 66794370 offset 0 len 29933568
[ 728.880563] 236: pfn 66801678 offset 0 len 4136960
[ 728.886086] 238: pfn 66820096 offset 0 len 2506752
Hi Yiding,
Could you include all logs dumped by the debug patch?
+ if (WARN_ON(bio->bi_iter.bi_size == 0 && bio_op(bio) == REQ_OP_WRITE))
{
+ int i;
+ printk("bio: %hx/%hx %u %llu %u\n", bio->bi_flags,
bio->bi_opf,
+ bio->bi_vcnt,
+ (unsigned long long)bio->bi_iter.bi_sector,
+ bio->bi_iter.bi_size);
+ for (i = 0; i < bio->bi_vcnt; i++) {
+ struct bio_vec *bvec = &bio->bi_io_vec[i];
+ printk("%d: pfn %lu offset %u len %u\n", i++,
+ (unsigned long)page_to_pfn(bvec->bv_page),
+ bvec->bv_offset, bvec->bv_len);
+ }
+ }
Thanks,
Ming
12:18 a.m.
在 2019/7/1 下午3:14, Ming Lei 写道:
On Mon, Jul 01, 2019 at 02:54:31PM +0800, Liu Yiding wrote:
> 在 2019/7/1 下午2:44, Ming Lei 写道:
>> On Mon, Jul 01, 2019 at 12:49:51PM +0800, Liu Yiding wrote:
>>> 在 2019/7/1 上午10:47, Ming Lei 写道:
>>>> [snip]
>>>>>>> ------
>>>>>>>
>>>>>>> 2412 [ 112.558155] XFS (sdb1): Ending clean mount
>>>>>>> 2413 [ 1317.330820] WARNING: CPU: 63 PID: 1495 at
>>>>>>> drivers/scsi/scsi_lib.c:1034 scsi_init_io+0x11a/0x2e0
>>>>>>> 2414 [ 1317.344000] Modules linked in: xfs dm_mod btrfs xor
zstd_decompress
>>>>>>> zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl
sb_edac
>>>>>>> x86_pkg_temp_thermal intel_powerclamp coretemp kvm_i ntel
kvm irqbypass
>>>>>>> crct10dif_pclmul crc32_pclmul crc32c_intel mgag200 snd_pcm
>>>>>>> ghash_clmulni_intel ttm snd_timer drm_kms_helper ipmi_si ahci
snd
>>>>>>> aesni_intel syscopyarea libahci ipmi_devi ntf sysfillrect
sysimgblt
>>>>>>> crypto_simd mei_me fb_sys_fops soundcore ioatdma cryptd
ipmi_msghandler
>>>>>>> glue_helper pcspkr drm libata mei joydev lpc_ich i2c_i801 dca
wmi
>>>>>>> acpi_power_meter acpi _pad pcc_cpufreq ip_tables
>>>>>>> 2415 [ 1317.408135] CPU: 63 PID: 1495 Comm: kworker/63:1H
Tainted:
>>>>>>> G I 5.2.0-rc5-00001-gc2d55e2 #1
>>>>>>> 2416 [ 1317.420350] Hardware name: Intel Corporation
S2600WTT/S2600WTT, BIOS
>>>>>>> SE5C610.86B.01.01.0008.021120151325 02/11/2015
>>>>>>> 2417 [ 1317.433545] Workqueue: kblockd blk_mq_run_work_fn
>>>>>>> 2418 [ 1317.440295] RIP: 0010:scsi_init_io+0x11a/0x2e0
>>>>>>> 2419 [ 1317.446746] Code: be 01 00 00 00 e9 44 ff ff ff 8b 45
70 89 83 08 01
>>>>>>> 00 00 f6 45 1a 01 75 8a 31 c0 48 83 c4 08 5b 5d 41 5c 41 5d
41 5e 41 5f c3
>>>>>>> <0f> 0b 8b 4d 18 8b 55 28 45 31 c9 48 c7 c6 98 da
69 82 48 c7 c7 f0
>>>>>>> 2420 [ 1317.470863] RSP: 0018:ffffa5fa4f7e7c68 EFLAGS:
00010246
>>>>>>> 2421 [ 1317.478294] RAX: 000000003d17d1b8 RBX:
ffff8ae6cb742690 RCX:
>>>>>>> 0000000000000000
>>>>>>> 2422 [ 1317.487878] RDX: 0000000000000000 RSI:
0000000000000000 RDI:
>>>>>>> ffff8ae6cb742690
>>>>>>> 2423 [ 1317.497469] RBP: ffff8ae6cb742580 R08:
0000000000023003 R09:
>>>>>>> ffff8ae6cb7426c8
>>>>>>> 2424 [ 1317.507057] R10: ffffa5fa4c7d7da0 R11:
fefefefefefefeff R12:
>>>>>>> 0000000000000801
>>>>>>> 2425 [ 1317.516650] R13: 0000000000000000 R14:
0000000000000000 R15:
>>>>>>> ffff8ac800bd1800
>>>>>>> 2426 [ 1317.526253] FS: 0000000000000000(0000)
GS:ffff8b077fac0000(0000)
>>>>>>> knlGS:0000000000000000
>>>>>>> 2427 [ 1317.536918] CS: 0010 DS: 0000 ES: 0000 CR0:
0000000080050033
>>>>>>> 2428 [ 1317.544952] CR2: 00007f39a5818004 CR3:
000000343440a006 CR4:
>>>>>>> 00000000001606e0
>>>>>>> 2429 [ 1317.554554] Call Trace:
>>>>>>> 2430 [ 1317.558921] sd_init_command+0x2ee/0xaf0 [sd_mod]
>>>>>>> 2431 [ 1317.565800] scsi_queue_rq+0x559/0xa80
>>>>>>> 2432 [ 1317.571607] blk_mq_dispatch_rq_list+0x3da/0x5b0
>>>>>>> 2433 [ 1317.578375] ? syscall_return_via_sysret+0x10/0x7f
>>>>>>> 2434 [ 1317.585339] ? __switch_to_asm+0x40/0x70
>>>>>>> 2435 [ 1317.591321] ? elv_rb_del+0x1f/0x30
>>>>>>> 2436 [ 1317.596807] ? deadline_remove_request+0x55/0xc0
>>>>>>> 2437 [ 1317.603560] blk_mq_do_dispatch_sched+0x76/0x120
>>>>>>> 2438 [ 1317.610302]
blk_mq_sched_dispatch_requests+0x100/0x170
>>>>>>> 2439 [ 1317.617733] __blk_mq_run_hw_queue+0x60/0x130
>>>>>>> 2440 [ 1317.624200] process_one_work+0x19c/0x3c0
>>>>>>> 2441 [ 1317.630258] worker_thread+0x3c/0x3b0
>>>>>>> 2442 [ 1317.635937] ? process_one_work+0x3c0/0x3c0
>>>>>>> 2443 [ 1317.642183] kthread+0x11e/0x140
>>>>>>> 2444 [ 1317.647346] ? kthread_park+0x90/0x90
>>>>>>> 2445 [ 1317.652986] ret_from_fork+0x35/0x40
>>>>>>> 2446 [ 1317.658523] ---[ end trace 61026ff97261da6a ]---
>>>>>>> 2447 [ 1317.665327] scsi_dump_rq: rq 6(f:801/23003, seg: 0)
>>>>>>> 2448 [ 1317.671601] bio 0: d602/801 0 1024971192 0
>>>>>>> 2449 [ 1317.676991] print_req_error: I/O error, dev sdb,
sector 1024971192
>>>>>>> flags 801
>>>>>>> 2450 [ 1317.685722] scsi_dump_rq: rq 8(f:801/23003, seg: 0)
>>>>>>> 2451 [ 1317.691994] bio 0: d602/801 0 1033359800 0
>>>>>> The above line shows the bio is zero-size, and this bio is
submitted
>>>>>> from bio, since the bio->bi_flags indicates that BIO_CLONED
isn't set.
>>>>>>
>>>>>> Please apply the following debug patch, and we may see where is
wrong.
>>>>>>
>>>>>> diff --git a/block/blk-core.c b/block/blk-core.c
>>>>>> index 8340f69670d8..86bb61b0df83 100644
>>>>>> --- a/block/blk-core.c
>>>>>> +++ b/block/blk-core.c
>>>>>> @@ -1125,6 +1125,7 @@ blk_qc_t submit_bio(struct bio *bio)
>>>>>> bio_devname(bio, b), count);
>>>>>> }
>>>>>> }
>>>>>> + WARN_ON(bio->bi_iter.bi_size == 0 && bio_op(bio) ==
REQ_OP_WRITE);
>>>>>> return generic_make_request(bio);
>>>>> Sorry for delay,
>>>>>
>>>>> ------
>>>>>
>>>>> [ 116.619136] 2019-06-29 16:12:41 mkdir -p /fs/sdb1
>>>>> [ 116.619138]
>>>>> [ 116.628743] 2019-06-29 16:12:41 mount -t xfs -o inode64 /dev/sdb1
>>>>> /fs/sdb1
>>>>> [ 116.628745]
>>>>> [ 116.634699] SGI XFS with ACLs, security attributes, realtime,
scrub,
>>>>> debug enabled
>>>>> [ 116.653110] XFS (sdb1): Mounting V5 Filesystem
>>>>> [ 116.670081] XFS (sdb1): Ending clean mount
>>>>> [ 690.106779] WARNING: CPU: 53 PID: 1477 at block/blk-core.c:1128
>>>>> submit_bio+0x93/0x150
>>>>> [ 690.117972] Modules linked in: xfs dm_mod btrfs xor
zstd_decompress
>>>>> zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac
>>>>> x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm
irqbypass
>>>>> mgag200 crct10dif_pclmul crc32_pclmul ttm crc32c_intel snd_pcm
>>>>> ghash_clmulni_intel drm_kms_helper snd_timer ahci syscopyarea
sysfillrect
>>>>> snd aesni_intel sysimgblt libahci ipmi_si soundcore fb_sys_fops
mei_me
>>>>> crypto_simd ipmi_devintf ioatdma cryptd pcspkr drm ipmi_msghandler
>>>>> glue_helper libata mei joydev lpc_ich i2c_i801 dca wmi
acpi_power_meter
>>>>> acpi_pad pcc_cpufreq ip_tables
>>>>> [ 690.176967] CPU: 53 PID: 1477 Comm: fio Tainted: G W I
>>>>> 5.2.0-rc5-00002-g9fa09bd #1
>>>>> [ 690.187198] Hardware name: Intel Corporation S2600WTT/S2600WTT,
BIOS
>>>>> SE5C610.86B.01.01.0008.021120151325 02/11/2015
>>>>> [ 690.199375] RIP: 0010:submit_bio+0x93/0x150
>>>>> [ 690.204583] Code: ee 74 45 83 e1 01 74 26 65 48 01 35 bf 39 3f 45
8b 0d
>>>>> ad 96 c4 01 85 c9 75 56 8b 43 28 85 c0 75 b3 0f b6 53 10 83 fa 01 75
aa <0f>
>>>>> 0b eb a6 65 48 8b 14 25 c0 6b 01 00 65 48 01 35 88 39 3f 45 48
>>>>> [ 690.226668] RSP: 0018:ffffb2d90efc3b40 EFLAGS: 00010246
>>>>> [ 690.233072] RAX: 0000000000000000 RBX: ffff9654c6ad9f40 RCX:
>>>>> 0000000000000801
>>>>> [ 690.241617] RDX: 0000000000000001 RSI: ffff9654c6ad9f00 RDI:
>>>>> ffff9654c6ad9f40
>>>>> [ 690.250146] RBP: 0000000000000000 R08: 0000000000000002 R09:
>>>>> 0000000000000000
>>>>> [ 690.258667] R10: 00000000000303a0 R11: ffff96757ffd4000 R12:
>>>>> ffffb2d90efc3e08
>>>>> [ 690.267183] R13: ffff9654c6ad9f00 R14: ffffb2d90efc3e08 R15:
>>>>> 0000000000000000
>>>>> [ 690.275696] FS: 00007fa8e005da80(0000) GS:ffff9654ffac0000(0000)
>>>>> knlGS:0000000000000000
>>>>> [ 690.285282] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>> [ 690.292252] CR2: 00007fa8cc05b3c0 CR3: 0000001fc8518001 CR4:
>>>>> 00000000001606e0
>>>>> [ 690.300770] Call Trace:
>>>>> [ 690.304123] xfs_submit_ioend+0x84/0x1e0 [xfs]
>>>>> [ 690.309668] xfs_do_writepage+0x246/0x5a0 [xfs]
>>>>> [ 690.315279] write_cache_pages+0x16b/0x440
>>>>> [ 690.320420] ? xfs_vm_releasepage+0x80/0x80 [xfs]
>>>>> [ 690.326238] xfs_vm_writepages+0x6b/0xa0 [xfs]
>>>>> [ 690.331742] do_writepages+0x4b/0xf0
>>>>> [ 690.336275] ? iomap_file_buffered_write+0x62/0x90
>>>>> [ 690.342162] ? iomap_write_end+0x240/0x240
>>>>> [ 690.347295] ? xfs_iunlock+0x165/0x220 [xfs]
>>>>> [ 690.352595] ? __filemap_fdatawrite_range+0xc1/0x100
>>>>> [ 690.358673] __filemap_fdatawrite_range+0xc1/0x100
>>>>> [ 690.364571] ? __switch_to_asm+0x34/0x70
>>>>> [ 690.369484] file_write_and_wait_range+0x4c/0xa0
>>>>> [ 690.375198] xfs_file_fsync+0x5d/0x230 [xfs]
>>>>> [ 690.380499] do_fsync+0x38/0x60
>>>>> [ 690.384534] __x64_sys_fsync+0x10/0x20
>>>>> [ 690.389243] do_syscall_64+0x5b/0x1e0
>>>>> [ 690.393857] entry_SYSCALL_64_after_hwframe+0x44/0xa9
>>>>> [ 690.400020] RIP: 0033:0x7fa8cbe4f70d
>>>>> [ 690.404533] Code: bc 20 00 00 75 10 b8 4a 00 00 00 0f 05 48 3d 01
f0 ff
>>>>> ff 73 31 c3 48 83 ec 08 e8 6e f7 ff ff 48 89 04 24 b8 4a 00 00 00 0f
05 <48>
>>>>> 8b 3c 24 48 89 c2 e8 b7 f7 ff ff 48 89 d0 48 83 c4 08 48 3d 01
>>>>> [ 690.426581] RSP: 002b:00007ffed6954160 EFLAGS: 00000293 ORIG_RAX:
>>>>> 000000000000004a
>>>>> [ 690.435583] RAX: ffffffffffffffda RBX: 00005573363790f0 RCX:
>>>>> 00007fa8cbe4f70d
>>>>> [ 690.444102] RDX: 0000000000200000 RSI: 00005573363790f0 RDI:
>>>>> 0000000000000003
>>>>> [ 690.452615] RBP: 00007fa8b337f418 R08: 0000000000000008 R09:
>>>>> 0000000000000001
>>>>> [ 690.461124] R10: 00000000897b0000 R11: 0000000000000293 R12:
>>>>> 0000000000000000
>>>>> [ 690.469631] R13: 0000001000000000 R14: 00007fa8c81dda50 R15:
>>>>> 0000000000000000
>>>>> [ 690.478130] ---[ end trace 90dd4ee6d5cdfcb8 ]---
>>>> I have just checked xfs code, and looks the only reason is that bio size
>>>> is overflowed, that means the whole bio may reach 4GB. That might be
>>>> possible in case of writeback. But I have to double check if that is
>>>> the case, could you test the following patch and provide the log?
>>>>
>>>> diff --git a/block/blk-core.c b/block/blk-core.c
>>>> index 8340f69670d8..9d61e408c608 100644
>>>> --- a/block/blk-core.c
>>>> +++ b/block/blk-core.c
>>>> @@ -1125,6 +1125,19 @@ blk_qc_t submit_bio(struct bio *bio)
>>>> bio_devname(bio, b), count);
>>>> }
>>>> }
>>>> + if (WARN_ON(bio->bi_iter.bi_size == 0 && bio_op(bio) ==
REQ_OP_WRITE)) {
>>>> + int i;
>>>> + printk("bio: %hx/%hx %u %llu %u\n", bio->bi_flags,
bio->bi_opf,
>>>> + bio->bi_vcnt,
>>>> + (unsigned long long)bio->bi_iter.bi_sector,
>>>> + bio->bi_iter.bi_size);
>>>> + for (i = 0; i < bio->bi_vcnt; i++) {
>>>> + struct bio_vec *bvec = &bio->bi_io_vec[i];
>>>> + printk("%d: pfn %lu offset %u len %u\n", i++,
>>>> + (unsigned
long)page_to_pfn(bvec->bv_page),
>>>> + bvec->bv_offset, bvec->bv_len);
>>>> + }
>>>> + }
>>>> return generic_make_request(bio);
>>>> }
>>> the newest log and i also see you fix patch, i will test your fix patch
>>> more times to check it.
>>>
>>> ------
>>>
>>> [ 54.656016] 2019-07-01 11:48:52 mkfs -t xfs /dev/sdb1
>>> [ 54.656017]
>>> [ 119.228467] WARNING: CPU: 5 PID: 1530 at block/blk-core.c:1128
>>> submit_bio+0xa5/0x1d0
>>> [ 119.240810] Modules linked in: dm_mod btrfs xor zstd_decompress
>>> zstd_compress raid6_pq libcrc32c sd_mod intel_rapl sg sb_edac
>>> x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
>>> mgag200 crct10dif_pclmul snd_pcm crc32_pclmul ttm crc32c_intel snd_timer
>>> drm_kms_helper ghash_clmulni_intel snd syscopyarea ipmi_si sysfillrect
>>> soundcore aesni_intel sysimgblt ipmi_devintf crypto_simd fb_sys_fops ahci
>>> mei_me libahci cryptd ipmi_msghandler pcspkr glue_helper drm mei libata
>>> joydev ioatdma lpc_ich i2c_i801 dca wmi acpi_power_meter acpi_pad
>>> pcc_cpufreq ip_tables
>>> [ 119.306252] CPU: 5 PID: 1530 Comm: mkfs.xfs Tainted: G W I
>>> 5.2.0-rc5-00001-g82f8097 #1
>>> [ 119.318274] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
>>> SE5C610.86B.01.01.0008.021120151325 02/11/2015
>>> [ 119.331852] RIP: 0010:submit_bio+0xa5/0x1d0
>>> [ 119.338422] Code: 00 00 65 48 01 35 b3 39 5f 76 8b 0d a1 96 c4 01 85 c9
>>> 0f 85 9b 00 00 00 8b 43 28 85 c0 75 a5 8b 53 10 0f b6 ca 83 f9 01 75 9a
<0f>
>>> 0b 0f b7 4b 68 0f b7 73 14 45 31 c9 4c 8b 43 20 48 c7 c7 f1 99
>>> [ 119.363296] RSP: 0018:ffff940c4fb17e58 EFLAGS: 00010246
>>> [ 119.371107] RAX: 0000000000000000 RBX: ffff88a53e3d6900 RCX:
>>> 0000000000000001
>>> [ 119.381068] RDX: 0000000000040801 RSI: ffff88868138c000 RDI:
>>> ffff88a53e3d6900
>>> [ 119.391041] RBP: ffff88a5448b0000 R08: ffff88a53e3d6900 R09:
>>> 0000000000000000
>>> [ 119.401003] R10: 0000000000000000 R11: 0000000000000000 R12:
>>> 0000000000000000
>>> [ 119.410967] R13: 0000000000000000 R14: 0000000000000000 R15:
>>> 0000000000000000
>>> [ 119.420881] FS: 00007f37eba81780(0000) GS:ffff88a57f340000(0000)
>>> knlGS:0000000000000000
>>> [ 119.431839] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> [ 119.440161] CR2: 00005630eb09d000 CR3: 0000001fc711c002 CR4:
>>> 00000000001606e0
>>> [ 119.450017] Call Trace:
>>> [ 119.454595] submit_bio_wait+0x53/0x80
>>> [ 119.460636] blkdev_issue_flush+0x81/0xb0
>>> [ 119.466930] blkdev_fsync+0x2f/0x40
>>> [ 119.472590] do_fsync+0x38/0x60
>>> [ 119.477810] __x64_sys_fsync+0x10/0x20
>>> [ 119.483722] do_syscall_64+0x5b/0x1e0
>>> [ 119.489516] entry_SYSCALL_64_after_hwframe+0x44/0xa9
>>> [ 119.496857] RIP: 0033:0x7f37eb2146f0
>>> [ 119.502502] Code: 48 3d 01 f0 ff ff 73 01 c3 48 8b 0d b2 78 20 00 f7 d8
>>> 64 89 01 48 83 c8 ff c3 83 3d d9 bc 20 00 00 75 10 b8 4a 00 00 00 0f 05
<48>
>>> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 6e f7 ff ff 48 89 04 24
>>> [ 119.526842] RSP: 002b:00007fffb27f4878 EFLAGS: 00000246 ORIG_RAX:
>>> 000000000000004a
>>> [ 119.536975] RAX: ffffffffffffffda RBX: 0000000000000811 RCX:
>>> 00007f37eb2146f0
>>> [ 119.546592] RDX: 00005630e91ba8e0 RSI: 0000000000000000 RDI:
>>> 0000000000000004
>>> [ 119.556171] RBP: 0000000000000004 R08: 00005630eb02b8a0 R09:
>>> 00007fffb27f4e00
>>> [ 119.565753] R10: 000000000000009b R11: 0000000000000246 R12:
>>> 00007fffb27f4c60
>>> [ 119.575339] R13: 00007fffb27f4a30 R14: 00005630eb048800 R15:
>>> 00005630eb05a800
>>> [ 119.584940] ---[ end trace 46a2e1214c930529 ]---
>>> [ 119.591765] bio: 0/801 0 0 0
>>>
>> Hi Yiding,
>>
>> The above warning is false positive, and it is flush request, and
>> it isn't same with your original report in which the following warning
>> is triggered.
>>
>> "drivers/scsi/scsi_lib.c:1034 scsi_init_io+0x11a/0x2e0"
>>
>> So please run the test kernel with the debug patch I sent to you today
>> and try to trigger the warning in scsi_init_io().
>
> ------
>
> [ 728.863669] 230: pfn 66784668 offset 0 len 17031168
> [ 728.869319] 232: pfn 66788826 offset 0 len 22708224
> [ 728.874946] 234: pfn 66794370 offset 0 len 29933568
> [ 728.880563] 236: pfn 66801678 offset 0 len 4136960
> [ 728.886086] 238: pfn 66820096 offset 0 len 2506752
Hi Yiding,
Could you include all logs dumped by the debug patch?
+ if (WARN_ON(bio->bi_iter.bi_size == 0 && bio_op(bio) == REQ_OP_WRITE))
{
+ int i;
+ printk("bio: %hx/%hx %u %llu %u\n", bio->bi_flags,
bio->bi_opf,
+ bio->bi_vcnt,
+ (unsigned long long)bio->bi_iter.bi_sector,
+ bio->bi_iter.bi_size);
+ for (i = 0; i < bio->bi_vcnt; i++) {
+ struct bio_vec *bvec = &bio->bi_io_vec[i];
+ printk("%d: pfn %lu offset %u len %u\n", i++,
+ (unsigned long)page_to_pfn(bvec->bv_page),
+ bvec->bv_offset, bvec->bv_len);
+ }
+ }
Hi, Ming. I have attatched all log in last email, can you receive that.
------
[ 124.334834] WARNING: CPU: 10 PID: 1421 at block/blk-core.c:1128
submit_bio+0xa5/0x1d0
[ 124.347274] Modules linked in: dm_mod btrfs xor zstd_decompress
zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac
x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
crct10dif_pclmul mgag200 crc32_pclmul ttm crc32c_intel
ghash_clmulni_intel drm_kms_helper snd_pcm syscopyarea sysfillrect
sysimgblt aesni_intel snd_timer ahci crypto_simd ipmi_si fb_sys_fops
mei_me libahci snd ipmi_devintf cryptd soundcore glue_helper
ipmi_msghandler pcspkr drm ioatdma mei libata joydev lpc_ich i2c_i801
dca wmi acpi_pad acpi_power_meter pcc_cpufreq ip_tables
[ 124.412706] CPU: 10 PID: 1421 Comm: mkfs.xfs Tainted: G W I
5.2.0-rc5-00001-g82f8097 #1
[ 124.424823] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
SE5C610.86B.01.01.0008.021120151325 02/11/2015
[ 124.438401] RIP: 0010:submit_bio+0xa5/0x1d0
[ 124.444974] Code: 00 00 65 48 01 35 b3 39 bf 6f 8b 0d a1 96 c4 01 85
c9 0f 85 9b 00 00 00 8b 43 28 85 c0 75 a5 8b 53 10 0f b6 ca 83 f9 01 75
9a <0f> 0b 0f b7 4b 68 0f b7 73 14 45 31 c9 4c 8b 43 20 48 c7 c7 f1 99
[ 124.469850] RSP: 0018:ffffac02ce72fe58 EFLAGS: 00010246
[ 124.477663] RAX: 0000000000000000 RBX: ffff95df3d416780 RCX:
0000000000000001
[ 124.487636] RDX: 0000000000040801 RSI: ffff95df3b4b6800 RDI:
ffff95df3d416780
[ 124.497605] RBP: ffff95df49ce8000 R08: ffff95df3d416780 R09:
0000000000000000
[ 124.507569] R10: 0000000000000000 R11: 0000000000000000 R12:
0000000000000000
[ 124.517535] R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
[ 124.527450] FS: 00007fee53609780(0000) GS:ffff95df7f480000(0000)
knlGS:0000000000000000
[ 124.538413] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 124.546738] CR2: 00005584d7290000 CR3: 0000001fbcd18003 CR4:
00000000001606e0
[ 124.556597] Call Trace:
[ 124.561176] submit_bio_wait+0x53/0x80
[ 124.567220] blkdev_issue_flush+0x81/0xb0
[ 124.573512] blkdev_fsync+0x2f/0x40
[ 124.579172] do_fsync+0x38/0x60
[ 124.584391] __x64_sys_fsync+0x10/0x20
[ 124.590301] do_syscall_64+0x5b/0x1e0
[ 124.596095] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 124.603437] RIP: 0033:0x7fee52d9c6f0
[ 124.609082] Code: 48 3d 01 f0 ff ff 73 01 c3 48 8b 0d b2 78 20 00 f7
d8 64 89 01 48 83 c8 ff c3 83 3d d9 bc 20 00 00 75 10 b8 4a 00 00 00 0f
05 <48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 6e f7 ff ff 48 89 04 24
[ 124.633417] RSP: 002b:00007ffc8c42b4d8 EFLAGS: 00000246 ORIG_RAX:
000000000000004a
[ 124.643548] RAX: ffffffffffffffda RBX: 0000000000000811 RCX:
00007fee52d9c6f0
[ 124.653166] RDX: 00005584d6db18e0 RSI: 0000000000000000 RDI:
0000000000000004
[ 124.662747] RBP: 0000000000000004 R08: 00005584d721e8a0 R09:
00007ffc8c42ba00
[ 124.672331] R10: 000000000000009b R11: 0000000000000246 R12:
00007ffc8c42b8c0
[ 124.681919] R13: 00007ffc8c42b690 R14: 00005584d723b800 R15:
00005584d724d800
[ 124.691520] ---[ end trace 929c66c4a933d3e2 ]---
[ 124.698346] bio: 0/801 0 0 0
[ 124.706748] meta-data=/dev/sdb1 isize=512 agcount=4,
agsize=48838198 blks
[ 124.706752]
[ 124.721970] = sectsz=4096 attr=2,
projid32bit=1
[ 124.721972]
[ 124.736132] = crc=1 finobt=1,
sparse=0, rmapbt=0, reflink=0
[ 124.736134]
[ 124.751645] data = bsize=4096
blocks=195352790, imaxpct=25
[ 124.751647]
[ 124.765522] = sunit=0 swidth=0 blks
[ 124.765524]
[ 124.778026] naming =version 2 bsize=4096 ascii-ci=0
ftype=1
[ 124.778028]
[ 124.791128] log =internal log bsize=4096 blocks=95387,
version=2
[ 124.791130]
[ 124.804674] = sectsz=4096 sunit=1
blks, lazy-count=1
[ 124.804675]
[ 124.818292] realtime =none extsz=4096 blocks=0,
rtextents=0
[ 124.818294]
[ 124.830580] 2019-07-01 12:36:49 mkdir -p /fs/sdb1
[ 124.830582]
[ 124.840509] 2019-07-01 12:36:49 mount -t xfs -o inode64 /dev/sdb1
/fs/sdb1
[ 124.840511]
[ 124.842506] SGI XFS with ACLs, security attributes, realtime, scrub,
debug enabled
[ 124.864478] XFS (sdb1): Mounting V5 Filesystem
[ 124.886520] XFS (sdb1): Ending clean mount
Starting watchdog daemon...
Starting Update UTMP about System Runlevel Changes...
[ 727.881932] WARNING: CPU: 0 PID: 1460 at block/blk-core.c:1128
submit_bio+0xa5/0x1d0
[ 727.892444] Modules linked in: xfs dm_mod btrfs xor zstd_decompress
zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac
x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
crct10dif_pclmul mgag200 crc32_pclmul ttm crc32c_intel
ghash_clmulni_intel drm_kms_helper snd_pcm syscopyarea sysfillrect
sysimgblt aesni_intel snd_timer ahci crypto_simd ipmi_si fb_sys_fops
mei_me libahci snd ipmi_devintf cryptd soundcore glue_helper
ipmi_msghandler pcspkr drm ioatdma mei libata joydev lpc_ich i2c_i801
dca wmi acpi_pad acpi_power_meter pcc_cpufreq ip_tables
[ 727.949105] CPU: 0 PID: 1460 Comm: fio Tainted: G W I
5.2.0-rc5-00001-g82f8097 #1
[ 727.958733] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
SE5C610.86B.01.01.0008.021120151325 02/11/2015
[ 727.970404] RIP: 0010:submit_bio+0xa5/0x1d0
[ 727.975087] Code: 00 00 65 48 01 35 b3 39 bf 6f 8b 0d a1 96 c4 01 85
c9 0f 85 9b 00 00 00 8b 43 28 85 c0 75 a5 8b 53 10 0f b6 ca 83 f9 01 75
9a <0f> 0b 0f b7 4b 68 0f b7 73 14 45 31 c9 4c 8b 43 20 48 c7 c7 f1 99
[ 727.996090] RSP: 0018:ffffac02cdaebb38 EFLAGS: 00010246
[ 728.001932] RAX: 0000000000000000 RBX: ffff95df4b754040 RCX:
0000000000000001
[ 728.009908] RDX: 0000000000000801 RSI: ffff95df4b754000 RDI:
ffff95df4b754040
[ 728.017885] RBP: 0000000000000000 R08: 0000000000000002 R09:
0000000000000000
[ 728.025861] R10: 0000000000030340 R11: ffff95dffffd5000 R12:
ffffac02cdaebe08
[ 728.033839] R13: ffff95df4b754000 R14: ffffac02cdaebe08 R15:
0000000000000000
[ 728.041817] FS: 00007f7492b0ca80(0000) GS:ffff95df7f200000(0000)
knlGS:0000000000000000
[ 728.050862] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 728.057286] CR2: 00007f746532e004 CR3: 0000001fdc20e005 CR4:
00000000001606f0
[ 728.065264] Call Trace:
[ 728.068068] xfs_submit_ioend+0x84/0x1e0 [xfs]
[ 728.073069] xfs_do_writepage+0x246/0x5a0 [xfs]
[ 728.078139] write_cache_pages+0x16b/0x440
[ 728.082741] ? xfs_vm_releasepage+0x80/0x80 [xfs]
[ 728.088025] xfs_vm_writepages+0x6b/0xa0 [xfs]
[ 728.092994] do_writepages+0x4b/0xf0
[ 728.096995] ? iomap_file_buffered_write+0x62/0x90
[ 728.102352] ? iomap_write_end+0x240/0x240
[ 728.106958] ? xfs_iunlock+0x165/0x220 [xfs]
[ 728.111734] ? __filemap_fdatawrite_range+0xc1/0x100
[ 728.117276] __filemap_fdatawrite_range+0xc1/0x100
[ 728.122625] file_write_and_wait_range+0x4c/0xa0
[ 728.127812] xfs_file_fsync+0x5d/0x230 [xfs]
[ 728.132591] do_fsync+0x38/0x60
[ 728.136105] __x64_sys_fsync+0x10/0x20
[ 728.140296] do_syscall_64+0x5b/0x1e0
[ 728.144390] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 728.150039] RIP: 0033:0x7f747e8fe70d
[ 728.154036] Code: bc 20 00 00 75 10 b8 4a 00 00 00 0f 05 48 3d 01 f0
ff ff 73 31 c3 48 83 ec 08 e8 6e f7 ff ff 48 89 04 24 b8 4a 00 00 00 0f
05 <48> 8b 3c 24 48 89 c2 e8 b7 f7 ff ff 48 89 d0 48 83 c4 08 48 3d 01
[ 728.175027] RSP: 002b:00007ffe67fd0f00 EFLAGS: 00000293 ORIG_RAX:
000000000000004a
[ 728.183491] RAX: ffffffffffffffda RBX: 000055d51d05b0f0 RCX:
00007f747e8fe70d
[ 728.191469] RDX: 0000000000200000 RSI: 000055d51d05b0f0 RDI:
0000000000000003
[ 728.199446] RBP: 00007f7465e2e418 R08: 0000000000000008 R09:
0000000000000001
[ 728.207961] R10: 00000000897b0000 R11: 0000000000000293 R12:
0000000000000000
[ 728.216467] R13: 0000001000000000 R14: 00007f747ac8ca50 R15:
0000000000000000
[ 728.224971] ---[ end trace 929c66c4a933d3e3 ]---
[ 728.230705] bio: c000/801 240 514831016 0
[ 728.235725] 0: pfn 66605336 offset 0 len 18096128
[ 728.241493] 2: pfn 66609754 offset 0 len 92037120
[ 728.247234] 4: pfn 66632224 offset 0 len 151552
[ 728.252776] 6: pfn 66632261 offset 0 len 94208
[ 728.258203] 8: pfn 66632284 offset 0 len 100556800
[ 728.264005] 10: pfn 66656834 offset 0 len 413696
[ 728.269591] 12: pfn 66656935 offset 0 len 643072
[ 728.275192] 14: pfn 66657092 offset 0 len 1982464
[ 728.280848] 16: pfn 66657576 offset 0 len 606208
[ 728.286416] 18: pfn 66657724 offset 0 len 126976
[ 728.291948] 20: pfn 66657755 offset 0 len 249856
[ 728.297470] 22: pfn 66657816 offset 0 len 626688
[ 728.302989] 24: pfn 66657969 offset 0 len 499712
[ 728.308505] 26: pfn 66658091 offset 0 len 610304
[ 728.314014] 28: pfn 66658240 offset 0 len 876544
[ 728.319516] 30: pfn 66658454 offset 0 len 495616
[ 728.325042] 32: pfn 66658575 offset 0 len 1372160
[ 728.330626] 34: pfn 66658910 offset 0 len 122880
[ 728.336134] 36: pfn 66658940 offset 0 len 118784
[ 728.341604] 38: pfn 66658969 offset 0 len 122880
[ 728.347078] 40: pfn 66658999 offset 0 len 118784
[ 728.352541] 42: pfn 66659028 offset 0 len 872448
[ 728.358003] 44: pfn 66659241 offset 0 len 610304
[ 728.363470] 46: pfn 66659390 offset 0 len 245760
[ 728.368935] 48: pfn 66659450 offset 0 len 385024
[ 728.374386] 50: pfn 66659544 offset 0 len 192512
[ 728.379846] 52: pfn 66659591 offset 0 len 339968
[ 728.385314] 54: pfn 66659674 offset 0 len 294912
[ 728.390790] 56: pfn 66659746 offset 0 len 98304
[ 728.396162] 58: pfn 66659770 offset 0 len 389120
[ 728.401592] 60: pfn 66659865 offset 0 len 872448
[ 728.407021] 62: pfn 66660078 offset 0 len 544768
[ 728.412458] 64: pfn 66660211 offset 0 len 704512
[ 728.417881] 66: pfn 66660383 offset 0 len 94208
[ 728.423232] 68: pfn 66660406 offset 0 len 94208
[ 728.428552] 70: pfn 66660429 offset 0 len 294912
[ 728.433964] 72: pfn 66660501 offset 0 len 180224
[ 728.439371] 74: pfn 66660545 offset 0 len 196608
[ 728.444777] 76: pfn 66660593 offset 0 len 290816
[ 728.450181] 78: pfn 66660664 offset 0 len 167936
[ 728.455547] 80: pfn 66660705 offset 0 len 81920
[ 728.460808] 82: pfn 66660725 offset 0 len 999424
[ 728.466189] 84: pfn 66660969 offset 0 len 442368
[ 728.471539] 86: pfn 66661077 offset 0 len 425984
[ 728.476887] 88: pfn 66661181 offset 0 len 348160
[ 728.482242] 90: pfn 66661266 offset 0 len 966656
[ 728.487577] 92: pfn 66661502 offset 0 len 176128
[ 728.492948] 94: pfn 66661545 offset 0 len 348160
[ 728.498287] 96: pfn 66661630 offset 0 len 1064960
[ 728.503770] 98: pfn 66661890 offset 0 len 180224
[ 728.509136] 100: pfn 66661934 offset 0 len 888832
[ 728.514574] 102: pfn 66662151 offset 0 len 176128
[ 728.520045] 104: pfn 66662194 offset 0 len 81920
[ 728.525381] 106: pfn 66662214 offset 0 len 356352
[ 728.530829] 108: pfn 66662301 offset 0 len 89661440
[ 728.536458] 110: pfn 66684191 offset 0 len 405504
[ 728.541883] 112: pfn 66684290 offset 0 len 860160
[ 728.547317] 114: pfn 66684500 offset 0 len 200704
[ 728.552742] 116: pfn 66684549 offset 0 len 299008
[ 728.558194] 118: pfn 66684622 offset 0 len 192512
[ 728.563609] 120: pfn 66684669 offset 0 len 102400
[ 728.569033] 122: pfn 66684694 offset 0 len 196608
[ 728.574473] 124: pfn 66684742 offset 0 len 98304
[ 728.579812] 126: pfn 66684766 offset 0 len 94208
[ 728.585165] 128: pfn 66684789 offset 0 len 749568
[ 728.590597] 130: pfn 66684972 offset 0 len 1490944
[ 728.596119] 132: pfn 66685336 offset 0 len 684032
[ 728.601542] 134: pfn 66685503 offset 0 len 495616
[ 728.606964] 136: pfn 66685624 offset 0 len 196608
[ 728.612391] 138: pfn 66685672 offset 0 len 585728
[ 728.617814] 140: pfn 66685815 offset 0 len 94208
[ 728.623168] 142: pfn 66685838 offset 0 len 94208
[ 728.628505] 144: pfn 66685861 offset 0 len 77824
[ 728.633830] 146: pfn 66685880 offset 0 len 778240
[ 728.639286] 148: pfn 66686070 offset 0 len 196608
[ 728.644710] 150: pfn 66686118 offset 0 len 503808
[ 728.650165] 152: pfn 66686241 offset 0 len 85626880
[ 728.655805] 154: pfn 66707146 offset 0 len 13893632
[ 728.661430] 156: pfn 66710538 offset 0 len 3411968
[ 728.666977] 158: pfn 66711371 offset 0 len 151552
[ 728.672412] 160: pfn 66711408 offset 0 len 450560
[ 728.677849] 162: pfn 66711518 offset 0 len 155648
[ 728.683299] 164: pfn 66711556 offset 0 len 323584
[ 728.688722] 166: pfn 66711635 offset 0 len 163840
[ 728.694174] 168: pfn 66711675 offset 0 len 327680
[ 728.699607] 170: pfn 66711755 offset 0 len 159744
[ 728.705032] 172: pfn 66711794 offset 0 len 1130496
[ 728.710544] 174: pfn 66712070 offset 0 len 974848
[ 728.715968] 176: pfn 66712308 offset 0 len 651264
[ 728.721384] 178: pfn 66712467 offset 0 len 2285568
[ 728.726906] 180: pfn 66713025 offset 0 len 57344
[ 728.732236] 182: pfn 66713039 offset 0 len 868352
[ 728.737663] 184: pfn 66713251 offset 0 len 491520
[ 728.743100] 186: pfn 66713371 offset 0 len 389120
[ 728.748535] 188: pfn 66713466 offset 0 len 753664
[ 728.753982] 190: pfn 66713650 offset 0 len 475136
[ 728.759404] 192: pfn 66713766 offset 0 len 192512
[ 728.764839] 194: pfn 66713813 offset 0 len 192512
[ 728.770285] 196: pfn 66713860 offset 0 len 196608
[ 728.775704] 198: pfn 66713908 offset 0 len 98304
[ 728.781035] 200: pfn 66713932 offset 0 len 2113536
[ 728.786562] 202: pfn 66714448 offset 0 len 94208
[ 728.791894] 204: pfn 66714471 offset 0 len 290816
[ 728.797323] 206: pfn 66714542 offset 0 len 65536
[ 728.802653] 208: pfn 66714558 offset 0 len 2453504
[ 728.808195] 210: pfn 66715157 offset 0 len 790528
[ 728.813612] 212: pfn 66715350 offset 0 len 1056768
[ 728.819129] 214: pfn 66715608 offset 0 len 76349440
[ 728.824778] 216: pfn 66734248 offset 0 len 63995904
[ 728.830407] 218: pfn 66749872 offset 0 len 52539392
[ 728.836033] 220: pfn 66762699 offset 0 len 33771520
[ 728.841649] 222: pfn 66770944 offset 0 len 5955584
[ 728.847178] 224: pfn 66772398 offset 0 len 41803776
[ 728.852797] 226: pfn 66782604 offset 0 len 16384
[ 728.858151] 228: pfn 66782608 offset 0 len 8437760
[ 728.863669] 230: pfn 66784668 offset 0 len 17031168
[ 728.869319] 232: pfn 66788826 offset 0 len 22708224
[ 728.874946] 234: pfn 66794370 offset 0 len 29933568
[ 728.880563] 236: pfn 66801678 offset 0 len 4136960
[ 728.886086] 238: pfn 66820096 offset 0 len 2506752
[ 729.336992] WARNING: CPU: 40 PID: 1312 at
drivers/scsi/scsi_lib.c:1010 scsi_init_io+0x103/0x170
[ 729.349150] Modules linked in: xfs dm_mod btrfs xor zstd_decompress
zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac
x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
crct10dif_pclmul mgag200 crc32_pclmul ttm crc32c_intel
ghash_clmulni_intel drm_kms_helper snd_pcm syscopyarea sysfillrect
sysimgblt aesni_intel snd_timer ahci crypto_simd ipmi_si fb_sys_fops
mei_me libahci snd ipmi_devintf cryptd soundcore glue_helper
ipmi_msghandler pcspkr drm ioatdma mei libata joydev lpc_ich i2c_i801
dca wmi acpi_pad acpi_power_meter pcc_cpufreq ip_tables
[ 729.409451] CPU: 40 PID: 1312 Comm: kworker/40:1H Tainted: G W
I 5.2.0-rc5-00001-g82f8097 #1
[ 729.421013] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
SE5C610.86B.01.01.0008.021120151325 02/11/2015
[ 729.433604] Workqueue: kblockd blk_mq_run_work_fn
[ 729.439805] RIP: 0010:scsi_init_io+0x103/0x170
[ 729.445726] Code: 5c 41 5d c3 be 01 00 00 00 e9 49 ff ff ff 8b 45 70
89 83 08 01 00 00 f6 45 1a 01 75 92 31 c0 48 83 c4 08 5b 5d 41 5c 41 5d
c3 <0f> 0b 48 83 c4 08 b8 0a 00 00 00 5b 5d 41 5c 41 5d c3 48 8b 75 38
[ 729.468787] RSP: 0018:ffffac02cea47c78 EFLAGS: 00010246
[ 729.475705] RAX: 000000001eafb2a8 RBX: ffff95df3e7a3950 RCX:
0000000000000000
[ 729.484790] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
ffff95df3e7a3950
[ 729.493889] RBP: ffff95df3e7a3840 R08: 0000000000000000 R09:
ffff95df3e7a3988
[ 729.502995] R10: ffffac02cc71fda0 R11: fefefefefefefeff R12:
0000000000000801
[ 729.512123] R13: 0000000000000000 R14: 0000000000000000 R15:
ffff95df3ba8e800
[ 729.521261] FS: 0000000000000000(0000) GS:ffff95df7f780000(0000)
knlGS:0000000000000000
[ 729.531505] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 729.539140] CR2: 00007f7465922814 CR3: 0000000a80c0a006 CR4:
00000000001606e0
[ 729.548359] Call Trace:
[ 729.552355] sd_init_command+0x2ee/0xaf0 [sd_mod]
[ 729.558882] scsi_queue_rq+0x559/0xa80
[ 729.564354] blk_mq_dispatch_rq_list+0x3da/0x5b0
[ 729.570820] ? syscall_return_via_sysret+0x10/0x7f
[ 729.577488] ? __switch_to_asm+0x40/0x70
[ 729.583192] ? elv_rb_del+0x1f/0x30
[ 729.588417] ? deadline_remove_request+0x55/0xc0
[ 729.594915] blk_mq_do_dispatch_sched+0x76/0x120
[ 729.601424] blk_mq_sched_dispatch_requests+0x100/0x170
[ 729.608626] __blk_mq_run_hw_queue+0x60/0x130
[ 729.614873] process_one_work+0x19c/0x3c0
[ 729.620734] worker_thread+0x3c/0x3b0
[ 729.626210] ? process_one_work+0x3c0/0x3c0
[ 729.632280] kthread+0x11e/0x140
[ 729.637281] ? kthread_park+0x90/0x90
[ 729.642773] ret_from_fork+0x35/0x40
[ 729.648172] ---[ end trace 929c66c4a933d3e4 ]---
[ 729.654870] print_req_error: I/O error, dev sdb, sector 514831016
flags 801
[ 729.849057] XFS (sdb1): writeback error on sector 514831016
[ 738.467814] WARNING: CPU: 1 PID: 1460 at block/blk-core.c:1128
submit_bio+0xa5/0x1d0
[ 738.478959] Modules linked in: xfs dm_mod btrfs xor zstd_decompress
zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac
x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
crct10dif_pclmul mgag200 crc32_pclmul ttm crc32c_intel
ghash_clmulni_intel drm_kms_helper snd_pcm syscopyarea sysfillrect
sysimgblt aesni_intel snd_timer ahci crypto_simd ipmi_si fb_sys_fops
mei_me libahci snd ipmi_devintf cryptd soundcore glue_helper
ipmi_msghandler pcspkr drm ioatdma mei libata joydev lpc_ich i2c_i801
dca wmi acpi_pad acpi_power_meter pcc_cpufreq ip_tables
[ 738.538361] CPU: 1 PID: 1460 Comm: fio Tainted: G W I
5.2.0-rc5-00001-g82f8097 #1
[ 738.548561] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
SE5C610.86B.01.01.0008.021120151325 02/11/2015
[ 738.560805] RIP: 0010:submit_bio+0xa5/0x1d0
[ 738.566050] Code: 00 00 65 48 01 35 b3 39 bf 6f 8b 0d a1 96 c4 01 85
c9 0f 85 9b 00 00 00 8b 43 28 85 c0 75 a5 8b 53 10 0f b6 ca 83 f9 01 75
9a <0f> 0b 0f b7 4b 68 0f b7 73 14 45 31 c9 4c 8b 43 20 48 c7 c7 f1 99
[ 738.588175] RSP: 0018:ffffac02cdaebb38 EFLAGS: 00010246
[ 738.594572] RAX: 0000000000000000 RBX: ffff95df3ce59e40 RCX:
0000000000000001
[ 738.603102] RDX: 0000000000000801 RSI: ffff95df3ce59e00 RDI:
ffff95df3ce59e40
[ 738.611611] RBP: 0000000000000000 R08: 0000000000000002 R09:
0000000000000000
[ 738.620118] R10: 0000000000030340 R11: ffff95dffffd5000 R12:
ffffac02cdaebe08
[ 738.628626] R13: ffff95df3ce59e00 R14: ffffac02cdaebe08 R15:
0000000000000000
[ 738.637125] FS: 00007f7492b0ca80(0000) GS:ffff95df7f240000(0000)
knlGS:0000000000000000
[ 738.646689] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 738.653626] CR2: 00007f7465355004 CR3: 0000001fdc20e004 CR4:
00000000001606e0
[ 738.662117] Call Trace:
[ 738.665420] xfs_submit_ioend+0x84/0x1e0 [xfs]
[ 738.670931] xfs_do_writepage+0x246/0x5a0 [xfs]
[ 738.676514] write_cache_pages+0x16b/0x440
[ 738.681638] ? xfs_vm_releasepage+0x80/0x80 [xfs]
[ 738.687425] xfs_vm_writepages+0x6b/0xa0 [xfs]
[ 738.692895] do_writepages+0x4b/0xf0
[ 738.697392] ? iomap_file_buffered_write+0x62/0x90
[ 738.703240] ? iomap_write_end+0x240/0x240
[ 738.708335] ? xfs_iunlock+0x165/0x220 [xfs]
[ 738.713600] ? __filemap_fdatawrite_range+0xc1/0x100
[ 738.719641] __filemap_fdatawrite_range+0xc1/0x100
[ 738.725489] file_write_and_wait_range+0x4c/0xa0
[ 738.731165] xfs_file_fsync+0x5d/0x230 [xfs]
[ 738.736435] do_fsync+0x38/0x60
[ 738.740439] __x64_sys_fsync+0x10/0x20
[ 738.745124] do_syscall_64+0x5b/0x1e0
[ 738.749711] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 738.755848] RIP: 0033:0x7f747e8fe70d
[ 738.760333] Code: bc 20 00 00 75 10 b8 4a 00 00 00 0f 05 48 3d 01 f0
ff ff 73 31 c3 48 83 ec 08 e8 6e f7 ff ff 48 89 04 24 b8 4a 00 00 00 0f
05 <48> 8b 3c 24 48 89 c2 e8 b7 f7 ff ff 48 89 d0 48 83 c4 08 48 3d 01
[ 738.782345] RSP: 002b:00007ffe67fd0f00 EFLAGS: 00000293 ORIG_RAX:
000000000000004a
[ 738.791335] RAX: ffffffffffffffda RBX: 000055d51d05b0f0 RCX:
00007f747e8fe70d
[ 738.799843] RDX: 0000000000200000 RSI: 000055d51d05b0f0 RDI:
0000000000000003
[ 738.808354] RBP: 00007f7465e2e418 R08: 0000000000000008 R09:
0000000000000001
[ 738.816865] R10: 00000000897b0000 R11: 0000000000000293 R12:
0000000000000000
[ 738.825368] R13: 0000001000000000 R14: 00007f747ac8ca50 R15:
0000000000000000
[ 738.833858] ---[ end trace 929c66c4a933d3e5 ]---
[ 738.839553] bio: c000/801 98 532946720 0
[ 738.844459] 0: pfn 32001772 offset 0 len 184487936
[ 738.850312] 2: pfn 61947904 offset 0 len 4194304
[ 738.855993] 4: pfn 61945856 offset 0 len 352256
[ 738.861521] 6: pfn 61945942 offset 0 len 3842048
[ 738.867130] 8: pfn 61943808 offset 0 len 4194304
[ 738.872741] 10: pfn 61941760 offset 0 len 4194304
[ 738.878426] 12: pfn 61939712 offset 0 len 3362816
[ 738.884095] 14: pfn 61940533 offset 0 len 831488
[ 738.889666] 16: pfn 61937664 offset 0 len 4194304
[ 738.895333] 18: pfn 32165430 offset 0 len 243589120
[ 738.901159] 20: pfn 61935616 offset 0 len 4194304
[ 738.906784] 22: pfn 61933568 offset 0 len 4194304
[ 738.912414] 24: pfn 61931520 offset 0 len 4194304
[ 738.918029] 26: pfn 61929472 offset 0 len 4194304
[ 738.923661] 28: pfn 32224900 offset 0 len 242958336
[ 738.929470] 30: pfn 61927424 offset 0 len 4194304
[ 738.935063] 32: pfn 32284216 offset 0 len 243081216
[ 738.940858] 34: pfn 61925376 offset 0 len 4194304
[ 738.946466] 36: pfn 32343562 offset 0 len 243232768
[ 738.952242] 38: pfn 32402945 offset 0 len 16384
[ 738.957612] 40: pfn 61923328 offset 0 len 4194304
[ 738.963182] 42: pfn 61921280 offset 0 len 4194304
[ 738.968774] 44: pfn 61919232 offset 0 len 3047424
[ 738.974328] 46: pfn 61919976 offset 0 len 1146880
[ 738.979888] 48: pfn 61917184 offset 0 len 4194304
[ 738.985456] 50: pfn 61915136 offset 0 len 4194304
[ 738.991017] 52: pfn 32462295 offset 0 len 243081216
[ 738.996775] 54: pfn 61913088 offset 0 len 4194304
[ 739.002323] 56: pfn 61911040 offset 0 len 3039232
[ 739.007892] 58: pfn 61911782 offset 0 len 1155072
[ 739.013445] 60: pfn 61908992 offset 0 len 1875968
[ 739.018982] 62: pfn 61909450 offset 0 len 20480
[ 739.024332] 64: pfn 61909455 offset 0 len 2297856
[ 739.029865] 66: pfn 61906944 offset 0 len 4194304
[ 739.035420] 68: pfn 61904896 offset 0 len 4194304
[ 739.040947] 70: pfn 32640459 offset 0 len 243032064
[ 739.046677] 72: pfn 61902848 offset 0 len 4194304
[ 739.052232] 74: pfn 32699793 offset 0 len 243228672
[ 739.057977] 76: pfn 61900800 offset 0 len 4194304
[ 739.063495] 78: pfn 61898752 offset 0 len 4194304
[ 739.069023] 80: pfn 61896704 offset 0 len 4194304
[ 739.074520] 82: pfn 32759175 offset 0 len 242909184
[ 739.080230] 84: pfn 61894656 offset 0 len 4194304
[ 739.085743] 86: pfn 32818479 offset 0 len 243081216
[ 739.091446] 88: pfn 61892608 offset 0 len 4194304
[ 739.096936] 90: pfn 61890560 offset 0 len 1286144
[ 739.102418] 92: pfn 61890874 offset 0 len 2908160
[ 739.107895] 94: pfn 61888512 offset 0 len 4194304
[ 739.113390] 96: pfn 61886464 offset 0 len 2572288
[ 739.118938] print_req_error: I/O error, dev sdb, sector 532946720
flags 801
[ 739.430464] XFS (sdb1): writeback error on sector 532946720
[ 742.762917] fio: pid=0, err=5/file:filesetup.c:190, func=fsync,
error=Input/output error
[ 742.762920]
------
Thanks,
Ming
12:29 a.m.
On Mon, Jul 01, 2019 at 03:18:01PM +0800, Liu Yiding wrote:
在 2019/7/1 下午3:14, Ming Lei 写道:
> On Mon, Jul 01, 2019 at 02:54:31PM +0800, Liu Yiding wrote:
> > 在 2019/7/1 下午2:44, Ming Lei 写道:
> > > On Mon, Jul 01, 2019 at 12:49:51PM +0800, Liu Yiding wrote:
> > > > 在 2019/7/1 上午10:47, Ming Lei 写道:
> > > > > [snip]
> > > > > > > > ------
> > > > > > > >
> > > > > > > > 2412 [ 112.558155] XFS (sdb1): Ending clean
mount
> > > > > > > > 2413 [ 1317.330820] WARNING: CPU: 63 PID: 1495
at
> > > > > > > > drivers/scsi/scsi_lib.c:1034
scsi_init_io+0x11a/0x2e0
> > > > > > > > 2414 [ 1317.344000] Modules linked in: xfs dm_mod
btrfs xor zstd_decompress
> > > > > > > > zstd_compress raid6_pq libcrc32c sd_mod sg
intel_rapl sb_edac
> > > > > > > > x86_pkg_temp_thermal intel_powerclamp coretemp
kvm_i ntel kvm irqbypass
> > > > > > > > crct10dif_pclmul crc32_pclmul crc32c_intel
mgag200 snd_pcm
> > > > > > > > ghash_clmulni_intel ttm snd_timer drm_kms_helper
ipmi_si ahci snd
> > > > > > > > aesni_intel syscopyarea libahci ipmi_devi ntf
sysfillrect sysimgblt
> > > > > > > > crypto_simd mei_me fb_sys_fops soundcore ioatdma
cryptd ipmi_msghandler
> > > > > > > > glue_helper pcspkr drm libata mei joydev lpc_ich
i2c_i801 dca wmi
> > > > > > > > acpi_power_meter acpi _pad pcc_cpufreq
ip_tables
> > > > > > > > 2415 [ 1317.408135] CPU: 63 PID: 1495 Comm:
kworker/63:1H Tainted:
> > > > > > > > G I 5.2.0-rc5-00001-gc2d55e2 #1
> > > > > > > > 2416 [ 1317.420350] Hardware name: Intel
Corporation S2600WTT/S2600WTT, BIOS
> > > > > > > > SE5C610.86B.01.01.0008.021120151325 02/11/2015
> > > > > > > > 2417 [ 1317.433545] Workqueue: kblockd
blk_mq_run_work_fn
> > > > > > > > 2418 [ 1317.440295] RIP:
0010:scsi_init_io+0x11a/0x2e0
> > > > > > > > 2419 [ 1317.446746] Code: be 01 00 00 00 e9 44 ff
ff ff 8b 45 70 89 83 08 01
> > > > > > > > 00 00 f6 45 1a 01 75 8a 31 c0 48 83 c4 08 5b 5d
41 5c 41 5d 41 5e 41 5f c3
> > > > > > > > <0f> 0b 8b 4d 18 8b 55 28 45 31 c9 48
c7 c6 98 da 69 82 48 c7 c7 f0
> > > > > > > > 2420 [ 1317.470863] RSP: 0018:ffffa5fa4f7e7c68
EFLAGS: 00010246
> > > > > > > > 2421 [ 1317.478294] RAX: 000000003d17d1b8 RBX:
ffff8ae6cb742690 RCX:
> > > > > > > > 0000000000000000
> > > > > > > > 2422 [ 1317.487878] RDX: 0000000000000000 RSI:
0000000000000000 RDI:
> > > > > > > > ffff8ae6cb742690
> > > > > > > > 2423 [ 1317.497469] RBP: ffff8ae6cb742580 R08:
0000000000023003 R09:
> > > > > > > > ffff8ae6cb7426c8
> > > > > > > > 2424 [ 1317.507057] R10: ffffa5fa4c7d7da0 R11:
fefefefefefefeff R12:
> > > > > > > > 0000000000000801
> > > > > > > > 2425 [ 1317.516650] R13: 0000000000000000 R14:
0000000000000000 R15:
> > > > > > > > ffff8ac800bd1800
> > > > > > > > 2426 [ 1317.526253] FS: 0000000000000000(0000)
GS:ffff8b077fac0000(0000)
> > > > > > > > knlGS:0000000000000000
> > > > > > > > 2427 [ 1317.536918] CS: 0010 DS: 0000 ES: 0000
CR0: 0000000080050033
> > > > > > > > 2428 [ 1317.544952] CR2: 00007f39a5818004 CR3:
000000343440a006 CR4:
> > > > > > > > 00000000001606e0
> > > > > > > > 2429 [ 1317.554554] Call Trace:
> > > > > > > > 2430 [ 1317.558921] sd_init_command+0x2ee/0xaf0
[sd_mod]
> > > > > > > > 2431 [ 1317.565800] scsi_queue_rq+0x559/0xa80
> > > > > > > > 2432 [ 1317.571607]
blk_mq_dispatch_rq_list+0x3da/0x5b0
> > > > > > > > 2433 [ 1317.578375] ?
syscall_return_via_sysret+0x10/0x7f
> > > > > > > > 2434 [ 1317.585339] ? __switch_to_asm+0x40/0x70
> > > > > > > > 2435 [ 1317.591321] ? elv_rb_del+0x1f/0x30
> > > > > > > > 2436 [ 1317.596807] ?
deadline_remove_request+0x55/0xc0
> > > > > > > > 2437 [ 1317.603560]
blk_mq_do_dispatch_sched+0x76/0x120
> > > > > > > > 2438 [ 1317.610302]
blk_mq_sched_dispatch_requests+0x100/0x170
> > > > > > > > 2439 [ 1317.617733]
__blk_mq_run_hw_queue+0x60/0x130
> > > > > > > > 2440 [ 1317.624200]
process_one_work+0x19c/0x3c0
> > > > > > > > 2441 [ 1317.630258] worker_thread+0x3c/0x3b0
> > > > > > > > 2442 [ 1317.635937] ?
process_one_work+0x3c0/0x3c0
> > > > > > > > 2443 [ 1317.642183] kthread+0x11e/0x140
> > > > > > > > 2444 [ 1317.647346] ? kthread_park+0x90/0x90
> > > > > > > > 2445 [ 1317.652986] ret_from_fork+0x35/0x40
> > > > > > > > 2446 [ 1317.658523] ---[ end trace
61026ff97261da6a ]---
> > > > > > > > 2447 [ 1317.665327] scsi_dump_rq: rq
6(f:801/23003, seg: 0)
> > > > > > > > 2448 [ 1317.671601] bio 0: d602/801 0 1024971192
0
> > > > > > > > 2449 [ 1317.676991] print_req_error: I/O error,
dev sdb, sector 1024971192
> > > > > > > > flags 801
> > > > > > > > 2450 [ 1317.685722] scsi_dump_rq: rq
8(f:801/23003, seg: 0)
> > > > > > > > 2451 [ 1317.691994] bio 0: d602/801 0 1033359800
0
> > > > > > > The above line shows the bio is zero-size, and this
bio is submitted
> > > > > > > from bio, since the bio->bi_flags indicates that
BIO_CLONED isn't set.
> > > > > > >
> > > > > > > Please apply the following debug patch, and we may see
where is wrong.
> > > > > > >
> > > > > > > diff --git a/block/blk-core.c b/block/blk-core.c
> > > > > > > index 8340f69670d8..86bb61b0df83 100644
> > > > > > > --- a/block/blk-core.c
> > > > > > > +++ b/block/blk-core.c
> > > > > > > @@ -1125,6 +1125,7 @@ blk_qc_t submit_bio(struct bio
*bio)
> > > > > > > bio_devname(bio, b), count);
> > > > > > > }
> > > > > > > }
> > > > > > > + WARN_ON(bio->bi_iter.bi_size == 0 &&
bio_op(bio) == REQ_OP_WRITE);
> > > > > > > return generic_make_request(bio);
> > > > > > Sorry for delay,
> > > > > >
> > > > > > ------
> > > > > >
> > > > > > [ 116.619136] 2019-06-29 16:12:41 mkdir -p /fs/sdb1
> > > > > > [ 116.619138]
> > > > > > [ 116.628743] 2019-06-29 16:12:41 mount -t xfs -o inode64
/dev/sdb1
> > > > > > /fs/sdb1
> > > > > > [ 116.628745]
> > > > > > [ 116.634699] SGI XFS with ACLs, security attributes,
realtime, scrub,
> > > > > > debug enabled
> > > > > > [ 116.653110] XFS (sdb1): Mounting V5 Filesystem
> > > > > > [ 116.670081] XFS (sdb1): Ending clean mount
> > > > > > [ 690.106779] WARNING: CPU: 53 PID: 1477 at
block/blk-core.c:1128
> > > > > > submit_bio+0x93/0x150
> > > > > > [ 690.117972] Modules linked in: xfs dm_mod btrfs xor
zstd_decompress
> > > > > > zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl
sb_edac
> > > > > > x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel
kvm irqbypass
> > > > > > mgag200 crct10dif_pclmul crc32_pclmul ttm crc32c_intel
snd_pcm
> > > > > > ghash_clmulni_intel drm_kms_helper snd_timer ahci
syscopyarea sysfillrect
> > > > > > snd aesni_intel sysimgblt libahci ipmi_si soundcore
fb_sys_fops mei_me
> > > > > > crypto_simd ipmi_devintf ioatdma cryptd pcspkr drm
ipmi_msghandler
> > > > > > glue_helper libata mei joydev lpc_ich i2c_i801 dca wmi
acpi_power_meter
> > > > > > acpi_pad pcc_cpufreq ip_tables
> > > > > > [ 690.176967] CPU: 53 PID: 1477 Comm: fio Tainted:
G W I
> > > > > > 5.2.0-rc5-00002-g9fa09bd #1
> > > > > > [ 690.187198] Hardware name: Intel Corporation
S2600WTT/S2600WTT, BIOS
> > > > > > SE5C610.86B.01.01.0008.021120151325 02/11/2015
> > > > > > [ 690.199375] RIP: 0010:submit_bio+0x93/0x150
> > > > > > [ 690.204583] Code: ee 74 45 83 e1 01 74 26 65 48 01 35 bf
39 3f 45 8b 0d
> > > > > > ad 96 c4 01 85 c9 75 56 8b 43 28 85 c0 75 b3 0f b6 53 10 83
fa 01 75 aa <0f>
> > > > > > 0b eb a6 65 48 8b 14 25 c0 6b 01 00 65 48 01 35 88 39 3f 45
48
> > > > > > [ 690.226668] RSP: 0018:ffffb2d90efc3b40 EFLAGS: 00010246
> > > > > > [ 690.233072] RAX: 0000000000000000 RBX: ffff9654c6ad9f40
RCX:
> > > > > > 0000000000000801
> > > > > > [ 690.241617] RDX: 0000000000000001 RSI: ffff9654c6ad9f00
RDI:
> > > > > > ffff9654c6ad9f40
> > > > > > [ 690.250146] RBP: 0000000000000000 R08: 0000000000000002
R09:
> > > > > > 0000000000000000
> > > > > > [ 690.258667] R10: 00000000000303a0 R11: ffff96757ffd4000
R12:
> > > > > > ffffb2d90efc3e08
> > > > > > [ 690.267183] R13: ffff9654c6ad9f00 R14: ffffb2d90efc3e08
R15:
> > > > > > 0000000000000000
> > > > > > [ 690.275696] FS: 00007fa8e005da80(0000)
GS:ffff9654ffac0000(0000)
> > > > > > knlGS:0000000000000000
> > > > > > [ 690.285282] CS: 0010 DS: 0000 ES: 0000 CR0:
0000000080050033
> > > > > > [ 690.292252] CR2: 00007fa8cc05b3c0 CR3: 0000001fc8518001
CR4:
> > > > > > 00000000001606e0
> > > > > > [ 690.300770] Call Trace:
> > > > > > [ 690.304123] xfs_submit_ioend+0x84/0x1e0 [xfs]
> > > > > > [ 690.309668] xfs_do_writepage+0x246/0x5a0 [xfs]
> > > > > > [ 690.315279] write_cache_pages+0x16b/0x440
> > > > > > [ 690.320420] ? xfs_vm_releasepage+0x80/0x80 [xfs]
> > > > > > [ 690.326238] xfs_vm_writepages+0x6b/0xa0 [xfs]
> > > > > > [ 690.331742] do_writepages+0x4b/0xf0
> > > > > > [ 690.336275] ? iomap_file_buffered_write+0x62/0x90
> > > > > > [ 690.342162] ? iomap_write_end+0x240/0x240
> > > > > > [ 690.347295] ? xfs_iunlock+0x165/0x220 [xfs]
> > > > > > [ 690.352595] ? __filemap_fdatawrite_range+0xc1/0x100
> > > > > > [ 690.358673] __filemap_fdatawrite_range+0xc1/0x100
> > > > > > [ 690.364571] ? __switch_to_asm+0x34/0x70
> > > > > > [ 690.369484] file_write_and_wait_range+0x4c/0xa0
> > > > > > [ 690.375198] xfs_file_fsync+0x5d/0x230 [xfs]
> > > > > > [ 690.380499] do_fsync+0x38/0x60
> > > > > > [ 690.384534] __x64_sys_fsync+0x10/0x20
> > > > > > [ 690.389243] do_syscall_64+0x5b/0x1e0
> > > > > > [ 690.393857] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > > > > > [ 690.400020] RIP: 0033:0x7fa8cbe4f70d
> > > > > > [ 690.404533] Code: bc 20 00 00 75 10 b8 4a 00 00 00 0f 05
48 3d 01 f0 ff
> > > > > > ff 73 31 c3 48 83 ec 08 e8 6e f7 ff ff 48 89 04 24 b8 4a 00
00 00 0f 05 <48>
> > > > > > 8b 3c 24 48 89 c2 e8 b7 f7 ff ff 48 89 d0 48 83 c4 08 48 3d
01
> > > > > > [ 690.426581] RSP: 002b:00007ffed6954160 EFLAGS: 00000293
ORIG_RAX:
> > > > > > 000000000000004a
> > > > > > [ 690.435583] RAX: ffffffffffffffda RBX: 00005573363790f0
RCX:
> > > > > > 00007fa8cbe4f70d
> > > > > > [ 690.444102] RDX: 0000000000200000 RSI: 00005573363790f0
RDI:
> > > > > > 0000000000000003
> > > > > > [ 690.452615] RBP: 00007fa8b337f418 R08: 0000000000000008
R09:
> > > > > > 0000000000000001
> > > > > > [ 690.461124] R10: 00000000897b0000 R11: 0000000000000293
R12:
> > > > > > 0000000000000000
> > > > > > [ 690.469631] R13: 0000001000000000 R14: 00007fa8c81dda50
R15:
> > > > > > 0000000000000000
> > > > > > [ 690.478130] ---[ end trace 90dd4ee6d5cdfcb8 ]---
> > > > > I have just checked xfs code, and looks the only reason is that
bio size
> > > > > is overflowed, that means the whole bio may reach 4GB. That
might be
> > > > > possible in case of writeback. But I have to double check if
that is
> > > > > the case, could you test the following patch and provide the
log?
> > > > >
> > > > > diff --git a/block/blk-core.c b/block/blk-core.c
> > > > > index 8340f69670d8..9d61e408c608 100644
> > > > > --- a/block/blk-core.c
> > > > > +++ b/block/blk-core.c
> > > > > @@ -1125,6 +1125,19 @@ blk_qc_t submit_bio(struct bio *bio)
> > > > > bio_devname(bio, b), count);
> > > > > }
> > > > > }
> > > > > + if (WARN_ON(bio->bi_iter.bi_size == 0 &&
bio_op(bio) == REQ_OP_WRITE)) {
> > > > > + int i;
> > > > > + printk("bio: %hx/%hx %u %llu %u\n",
bio->bi_flags, bio->bi_opf,
> > > > > + bio->bi_vcnt,
> > > > > + (unsigned long
long)bio->bi_iter.bi_sector,
> > > > > + bio->bi_iter.bi_size);
> > > > > + for (i = 0; i < bio->bi_vcnt; i++) {
> > > > > + struct bio_vec *bvec = &bio->bi_io_vec[i];
> > > > > + printk("%d: pfn %lu offset %u len %u\n", i++,
> > > > > + (unsigned
long)page_to_pfn(bvec->bv_page),
> > > > > + bvec->bv_offset,
bvec->bv_len);
> > > > > + }
> > > > > + }
> > > > > return generic_make_request(bio);
> > > > > }
> > > > the newest log and i also see you fix patch, i will test your fix
patch
> > > > more times to check it.
> > > >
> > > > ------
> > > >
> > > > [ 54.656016] 2019-07-01 11:48:52 mkfs -t xfs /dev/sdb1
> > > > [ 54.656017]
> > > > [ 119.228467] WARNING: CPU: 5 PID: 1530 at block/blk-core.c:1128
> > > > submit_bio+0xa5/0x1d0
> > > > [ 119.240810] Modules linked in: dm_mod btrfs xor zstd_decompress
> > > > zstd_compress raid6_pq libcrc32c sd_mod intel_rapl sg sb_edac
> > > > x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm
irqbypass
> > > > mgag200 crct10dif_pclmul snd_pcm crc32_pclmul ttm crc32c_intel
snd_timer
> > > > drm_kms_helper ghash_clmulni_intel snd syscopyarea ipmi_si
sysfillrect
> > > > soundcore aesni_intel sysimgblt ipmi_devintf crypto_simd fb_sys_fops
ahci
> > > > mei_me libahci cryptd ipmi_msghandler pcspkr glue_helper drm mei
libata
> > > > joydev ioatdma lpc_ich i2c_i801 dca wmi acpi_power_meter acpi_pad
> > > > pcc_cpufreq ip_tables
> > > > [ 119.306252] CPU: 5 PID: 1530 Comm: mkfs.xfs Tainted: G W I
> > > > 5.2.0-rc5-00001-g82f8097 #1
> > > > [ 119.318274] Hardware name: Intel Corporation S2600WTT/S2600WTT,
BIOS
> > > > SE5C610.86B.01.01.0008.021120151325 02/11/2015
> > > > [ 119.331852] RIP: 0010:submit_bio+0xa5/0x1d0
> > > > [ 119.338422] Code: 00 00 65 48 01 35 b3 39 5f 76 8b 0d a1 96 c4 01
85 c9
> > > > 0f 85 9b 00 00 00 8b 43 28 85 c0 75 a5 8b 53 10 0f b6 ca 83 f9 01 75
9a <0f>
> > > > 0b 0f b7 4b 68 0f b7 73 14 45 31 c9 4c 8b 43 20 48 c7 c7 f1 99
> > > > [ 119.363296] RSP: 0018:ffff940c4fb17e58 EFLAGS: 00010246
> > > > [ 119.371107] RAX: 0000000000000000 RBX: ffff88a53e3d6900 RCX:
> > > > 0000000000000001
> > > > [ 119.381068] RDX: 0000000000040801 RSI: ffff88868138c000 RDI:
> > > > ffff88a53e3d6900
> > > > [ 119.391041] RBP: ffff88a5448b0000 R08: ffff88a53e3d6900 R09:
> > > > 0000000000000000
> > > > [ 119.401003] R10: 0000000000000000 R11: 0000000000000000 R12:
> > > > 0000000000000000
> > > > [ 119.410967] R13: 0000000000000000 R14: 0000000000000000 R15:
> > > > 0000000000000000
> > > > [ 119.420881] FS: 00007f37eba81780(0000) GS:ffff88a57f340000(0000)
> > > > knlGS:0000000000000000
> > > > [ 119.431839] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > [ 119.440161] CR2: 00005630eb09d000 CR3: 0000001fc711c002 CR4:
> > > > 00000000001606e0
> > > > [ 119.450017] Call Trace:
> > > > [ 119.454595] submit_bio_wait+0x53/0x80
> > > > [ 119.460636] blkdev_issue_flush+0x81/0xb0
> > > > [ 119.466930] blkdev_fsync+0x2f/0x40
> > > > [ 119.472590] do_fsync+0x38/0x60
> > > > [ 119.477810] __x64_sys_fsync+0x10/0x20
> > > > [ 119.483722] do_syscall_64+0x5b/0x1e0
> > > > [ 119.489516] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > > > [ 119.496857] RIP: 0033:0x7f37eb2146f0
> > > > [ 119.502502] Code: 48 3d 01 f0 ff ff 73 01 c3 48 8b 0d b2 78 20 00
f7 d8
> > > > 64 89 01 48 83 c8 ff c3 83 3d d9 bc 20 00 00 75 10 b8 4a 00 00 00 0f
05 <48>
> > > > 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 6e f7 ff ff 48 89 04 24
> > > > [ 119.526842] RSP: 002b:00007fffb27f4878 EFLAGS: 00000246 ORIG_RAX:
> > > > 000000000000004a
> > > > [ 119.536975] RAX: ffffffffffffffda RBX: 0000000000000811 RCX:
> > > > 00007f37eb2146f0
> > > > [ 119.546592] RDX: 00005630e91ba8e0 RSI: 0000000000000000 RDI:
> > > > 0000000000000004
> > > > [ 119.556171] RBP: 0000000000000004 R08: 00005630eb02b8a0 R09:
> > > > 00007fffb27f4e00
> > > > [ 119.565753] R10: 000000000000009b R11: 0000000000000246 R12:
> > > > 00007fffb27f4c60
> > > > [ 119.575339] R13: 00007fffb27f4a30 R14: 00005630eb048800 R15:
> > > > 00005630eb05a800
> > > > [ 119.584940] ---[ end trace 46a2e1214c930529 ]---
> > > > [ 119.591765] bio: 0/801 0 0 0
> > > >
> > > Hi Yiding,
> > >
> > > The above warning is false positive, and it is flush request, and
> > > it isn't same with your original report in which the following
warning
> > > is triggered.
> > >
> > > "drivers/scsi/scsi_lib.c:1034 scsi_init_io+0x11a/0x2e0"
> > >
> > > So please run the test kernel with the debug patch I sent to you today
> > > and try to trigger the warning in scsi_init_io().
> >
> > ------
> >
> > [ 728.863669] 230: pfn 66784668 offset 0 len 17031168
> > [ 728.869319] 232: pfn 66788826 offset 0 len 22708224
> > [ 728.874946] 234: pfn 66794370 offset 0 len 29933568
> > [ 728.880563] 236: pfn 66801678 offset 0 len 4136960
> > [ 728.886086] 238: pfn 66820096 offset 0 len 2506752
> Hi Yiding,
>
> Could you include all logs dumped by the debug patch?
>
> + if (WARN_ON(bio->bi_iter.bi_size == 0 && bio_op(bio) ==
REQ_OP_WRITE)) {
> + int i;
> + printk("bio: %hx/%hx %u %llu %u\n", bio->bi_flags,
bio->bi_opf,
> + bio->bi_vcnt,
> + (unsigned long long)bio->bi_iter.bi_sector,
> + bio->bi_iter.bi_size);
> + for (i = 0; i < bio->bi_vcnt; i++) {
> + struct bio_vec *bvec = &bio->bi_io_vec[i];
> + printk("%d: pfn %lu offset %u len %u\n", i++,
> + (unsigned long)page_to_pfn(bvec->bv_page),
> + bvec->bv_offset, bvec->bv_len);
> + }
> + }
Hi, Ming. I have attatched all log in last email, can you receive that.
------
[ 124.334834] WARNING: CPU: 10 PID: 1421 at block/blk-core.c:1128
submit_bio+0xa5/0x1d0
[ 124.347274] Modules linked in: dm_mod btrfs xor zstd_decompress
zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac
x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
crct10dif_pclmul mgag200 crc32_pclmul ttm crc32c_intel ghash_clmulni_intel
drm_kms_helper snd_pcm syscopyarea sysfillrect sysimgblt aesni_intel
snd_timer ahci crypto_simd ipmi_si fb_sys_fops mei_me libahci snd
ipmi_devintf cryptd soundcore glue_helper ipmi_msghandler pcspkr drm ioatdma
mei libata joydev lpc_ich i2c_i801 dca wmi acpi_pad acpi_power_meter
pcc_cpufreq ip_tables
[ 124.412706] CPU: 10 PID: 1421 Comm: mkfs.xfs Tainted: G W I
5.2.0-rc5-00001-g82f8097 #1
[ 124.424823] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
SE5C610.86B.01.01.0008.021120151325 02/11/2015
[ 124.438401] RIP: 0010:submit_bio+0xa5/0x1d0
[ 124.444974] Code: 00 00 65 48 01 35 b3 39 bf 6f 8b 0d a1 96 c4 01 85 c9
0f 85 9b 00 00 00 8b 43 28 85 c0 75 a5 8b 53 10 0f b6 ca 83 f9 01 75 9a <0f>
0b 0f b7 4b 68 0f b7 73 14 45 31 c9 4c 8b 43 20 48 c7 c7 f1 99
[ 124.469850] RSP: 0018:ffffac02ce72fe58 EFLAGS: 00010246
[ 124.477663] RAX: 0000000000000000 RBX: ffff95df3d416780 RCX:
0000000000000001
[ 124.487636] RDX: 0000000000040801 RSI: ffff95df3b4b6800 RDI:
ffff95df3d416780
[ 124.497605] RBP: ffff95df49ce8000 R08: ffff95df3d416780 R09:
0000000000000000
[ 124.507569] R10: 0000000000000000 R11: 0000000000000000 R12:
0000000000000000
[ 124.517535] R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
[ 124.527450] FS: 00007fee53609780(0000) GS:ffff95df7f480000(0000)
knlGS:0000000000000000
[ 124.538413] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 124.546738] CR2: 00005584d7290000 CR3: 0000001fbcd18003 CR4:
00000000001606e0
[ 124.556597] Call Trace:
[ 124.561176] submit_bio_wait+0x53/0x80
[ 124.567220] blkdev_issue_flush+0x81/0xb0
[ 124.573512] blkdev_fsync+0x2f/0x40
[ 124.579172] do_fsync+0x38/0x60
[ 124.584391] __x64_sys_fsync+0x10/0x20
[ 124.590301] do_syscall_64+0x5b/0x1e0
[ 124.596095] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 124.603437] RIP: 0033:0x7fee52d9c6f0
[ 124.609082] Code: 48 3d 01 f0 ff ff 73 01 c3 48 8b 0d b2 78 20 00 f7 d8
64 89 01 48 83 c8 ff c3 83 3d d9 bc 20 00 00 75 10 b8 4a 00 00 00 0f 05 <48>
3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 6e f7 ff ff 48 89 04 24
[ 124.633417] RSP: 002b:00007ffc8c42b4d8 EFLAGS: 00000246 ORIG_RAX:
000000000000004a
[ 124.643548] RAX: ffffffffffffffda RBX: 0000000000000811 RCX:
00007fee52d9c6f0
[ 124.653166] RDX: 00005584d6db18e0 RSI: 0000000000000000 RDI:
0000000000000004
[ 124.662747] RBP: 0000000000000004 R08: 00005584d721e8a0 R09:
00007ffc8c42ba00
[ 124.672331] R10: 000000000000009b R11: 0000000000000246 R12:
00007ffc8c42b8c0
[ 124.681919] R13: 00007ffc8c42b690 R14: 00005584d723b800 R15:
00005584d724d800
[ 124.691520] ---[ end trace 929c66c4a933d3e2 ]---
[ 124.698346] bio: 0/801 0 0 0
[ 124.706748] meta-data=/dev/sdb1 isize=512 agcount=4,
agsize=48838198 blks
[ 124.706752]
[ 124.721970] = sectsz=4096 attr=2,
projid32bit=1
[ 124.721972]
[ 124.736132] = crc=1 finobt=1, sparse=0,
rmapbt=0, reflink=0
[ 124.736134]
[ 124.751645] data = bsize=4096 blocks=195352790,
imaxpct=25
[ 124.751647]
[ 124.765522] = sunit=0 swidth=0 blks
[ 124.765524]
[ 124.778026] naming =version 2 bsize=4096 ascii-ci=0
ftype=1
[ 124.778028]
[ 124.791128] log =internal log bsize=4096 blocks=95387,
version=2
[ 124.791130]
[ 124.804674] = sectsz=4096 sunit=1 blks,
lazy-count=1
[ 124.804675]
[ 124.818292] realtime =none extsz=4096 blocks=0,
rtextents=0
[ 124.818294]
[ 124.830580] 2019-07-01 12:36:49 mkdir -p /fs/sdb1
[ 124.830582]
[ 124.840509] 2019-07-01 12:36:49 mount -t xfs -o inode64 /dev/sdb1
/fs/sdb1
[ 124.840511]
[ 124.842506] SGI XFS with ACLs, security attributes, realtime, scrub,
debug enabled
[ 124.864478] XFS (sdb1): Mounting V5 Filesystem
[ 124.886520] XFS (sdb1): Ending clean mount
Starting watchdog daemon...
Starting Update UTMP about System Runlevel Changes...
[ 727.881932] WARNING: CPU: 0 PID: 1460 at block/blk-core.c:1128
submit_bio+0xa5/0x1d0
[ 727.892444] Modules linked in: xfs dm_mod btrfs xor zstd_decompress
zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac
x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
crct10dif_pclmul mgag200 crc32_pclmul ttm crc32c_intel ghash_clmulni_intel
drm_kms_helper snd_pcm syscopyarea sysfillrect sysimgblt aesni_intel
snd_timer ahci crypto_simd ipmi_si fb_sys_fops mei_me libahci snd
ipmi_devintf cryptd soundcore glue_helper ipmi_msghandler pcspkr drm ioatdma
mei libata joydev lpc_ich i2c_i801 dca wmi acpi_pad acpi_power_meter
pcc_cpufreq ip_tables
[ 727.949105] CPU: 0 PID: 1460 Comm: fio Tainted: G W I
5.2.0-rc5-00001-g82f8097 #1
[ 727.958733] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
SE5C610.86B.01.01.0008.021120151325 02/11/2015
[ 727.970404] RIP: 0010:submit_bio+0xa5/0x1d0
[ 727.975087] Code: 00 00 65 48 01 35 b3 39 bf 6f 8b 0d a1 96 c4 01 85 c9
0f 85 9b 00 00 00 8b 43 28 85 c0 75 a5 8b 53 10 0f b6 ca 83 f9 01 75 9a <0f>
0b 0f b7 4b 68 0f b7 73 14 45 31 c9 4c 8b 43 20 48 c7 c7 f1 99
[ 727.996090] RSP: 0018:ffffac02cdaebb38 EFLAGS: 00010246
[ 728.001932] RAX: 0000000000000000 RBX: ffff95df4b754040 RCX:
0000000000000001
[ 728.009908] RDX: 0000000000000801 RSI: ffff95df4b754000 RDI:
ffff95df4b754040
[ 728.017885] RBP: 0000000000000000 R08: 0000000000000002 R09:
0000000000000000
[ 728.025861] R10: 0000000000030340 R11: ffff95dffffd5000 R12:
ffffac02cdaebe08
[ 728.033839] R13: ffff95df4b754000 R14: ffffac02cdaebe08 R15:
0000000000000000
[ 728.041817] FS: 00007f7492b0ca80(0000) GS:ffff95df7f200000(0000)
knlGS:0000000000000000
[ 728.050862] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 728.057286] CR2: 00007f746532e004 CR3: 0000001fdc20e005 CR4:
00000000001606f0
[ 728.065264] Call Trace:
[ 728.068068] xfs_submit_ioend+0x84/0x1e0 [xfs]
[ 728.073069] xfs_do_writepage+0x246/0x5a0 [xfs]
[ 728.078139] write_cache_pages+0x16b/0x440
[ 728.082741] ? xfs_vm_releasepage+0x80/0x80 [xfs]
[ 728.088025] xfs_vm_writepages+0x6b/0xa0 [xfs]
[ 728.092994] do_writepages+0x4b/0xf0
[ 728.096995] ? iomap_file_buffered_write+0x62/0x90
[ 728.102352] ? iomap_write_end+0x240/0x240
[ 728.106958] ? xfs_iunlock+0x165/0x220 [xfs]
[ 728.111734] ? __filemap_fdatawrite_range+0xc1/0x100
[ 728.117276] __filemap_fdatawrite_range+0xc1/0x100
[ 728.122625] file_write_and_wait_range+0x4c/0xa0
[ 728.127812] xfs_file_fsync+0x5d/0x230 [xfs]
[ 728.132591] do_fsync+0x38/0x60
[ 728.136105] __x64_sys_fsync+0x10/0x20
[ 728.140296] do_syscall_64+0x5b/0x1e0
[ 728.144390] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 728.150039] RIP: 0033:0x7f747e8fe70d
[ 728.154036] Code: bc 20 00 00 75 10 b8 4a 00 00 00 0f 05 48 3d 01 f0 ff
ff 73 31 c3 48 83 ec 08 e8 6e f7 ff ff 48 89 04 24 b8 4a 00 00 00 0f 05 <48>
8b 3c 24 48 89 c2 e8 b7 f7 ff ff 48 89 d0 48 83 c4 08 48 3d 01
[ 728.175027] RSP: 002b:00007ffe67fd0f00 EFLAGS: 00000293 ORIG_RAX:
000000000000004a
[ 728.183491] RAX: ffffffffffffffda RBX: 000055d51d05b0f0 RCX:
00007f747e8fe70d
[ 728.191469] RDX: 0000000000200000 RSI: 000055d51d05b0f0 RDI:
0000000000000003
[ 728.199446] RBP: 00007f7465e2e418 R08: 0000000000000008 R09:
0000000000000001
[ 728.207961] R10: 00000000897b0000 R11: 0000000000000293 R12:
0000000000000000
[ 728.216467] R13: 0000001000000000 R14: 00007f747ac8ca50 R15:
0000000000000000
[ 728.224971] ---[ end trace 929c66c4a933d3e3 ]---
[ 728.230705] bio: c000/801 240 514831016 0
[ 728.235725] 0: pfn 66605336 offset 0 len 18096128
[ 728.241493] 2: pfn 66609754 offset 0 len 92037120
[ 728.247234] 4: pfn 66632224 offset 0 len 151552
[ 728.252776] 6: pfn 66632261 offset 0 len 94208
[ 728.258203] 8: pfn 66632284 offset 0 len 100556800
[ 728.264005] 10: pfn 66656834 offset 0 len 413696
[ 728.269591] 12: pfn 66656935 offset 0 len 643072
[ 728.275192] 14: pfn 66657092 offset 0 len 1982464
[ 728.280848] 16: pfn 66657576 offset 0 len 606208
[ 728.286416] 18: pfn 66657724 offset 0 len 126976
[ 728.291948] 20: pfn 66657755 offset 0 len 249856
[ 728.297470] 22: pfn 66657816 offset 0 len 626688
[ 728.302989] 24: pfn 66657969 offset 0 len 499712
[ 728.308505] 26: pfn 66658091 offset 0 len 610304
[ 728.314014] 28: pfn 66658240 offset 0 len 876544
[ 728.319516] 30: pfn 66658454 offset 0 len 495616
[ 728.325042] 32: pfn 66658575 offset 0 len 1372160
[ 728.330626] 34: pfn 66658910 offset 0 len 122880
[ 728.336134] 36: pfn 66658940 offset 0 len 118784
[ 728.341604] 38: pfn 66658969 offset 0 len 122880
[ 728.347078] 40: pfn 66658999 offset 0 len 118784
[ 728.352541] 42: pfn 66659028 offset 0 len 872448
[ 728.358003] 44: pfn 66659241 offset 0 len 610304
[ 728.363470] 46: pfn 66659390 offset 0 len 245760
[ 728.368935] 48: pfn 66659450 offset 0 len 385024
[ 728.374386] 50: pfn 66659544 offset 0 len 192512
[ 728.379846] 52: pfn 66659591 offset 0 len 339968
[ 728.385314] 54: pfn 66659674 offset 0 len 294912
[ 728.390790] 56: pfn 66659746 offset 0 len 98304
[ 728.396162] 58: pfn 66659770 offset 0 len 389120
[ 728.401592] 60: pfn 66659865 offset 0 len 872448
[ 728.407021] 62: pfn 66660078 offset 0 len 544768
[ 728.412458] 64: pfn 66660211 offset 0 len 704512
[ 728.417881] 66: pfn 66660383 offset 0 len 94208
[ 728.423232] 68: pfn 66660406 offset 0 len 94208
[ 728.428552] 70: pfn 66660429 offset 0 len 294912
[ 728.433964] 72: pfn 66660501 offset 0 len 180224
[ 728.439371] 74: pfn 66660545 offset 0 len 196608
[ 728.444777] 76: pfn 66660593 offset 0 len 290816
[ 728.450181] 78: pfn 66660664 offset 0 len 167936
[ 728.455547] 80: pfn 66660705 offset 0 len 81920
[ 728.460808] 82: pfn 66660725 offset 0 len 999424
[ 728.466189] 84: pfn 66660969 offset 0 len 442368
[ 728.471539] 86: pfn 66661077 offset 0 len 425984
[ 728.476887] 88: pfn 66661181 offset 0 len 348160
[ 728.482242] 90: pfn 66661266 offset 0 len 966656
[ 728.487577] 92: pfn 66661502 offset 0 len 176128
[ 728.492948] 94: pfn 66661545 offset 0 len 348160
[ 728.498287] 96: pfn 66661630 offset 0 len 1064960
[ 728.503770] 98: pfn 66661890 offset 0 len 180224
[ 728.509136] 100: pfn 66661934 offset 0 len 888832
[ 728.514574] 102: pfn 66662151 offset 0 len 176128
[ 728.520045] 104: pfn 66662194 offset 0 len 81920
[ 728.525381] 106: pfn 66662214 offset 0 len 356352
[ 728.530829] 108: pfn 66662301 offset 0 len 89661440
[ 728.536458] 110: pfn 66684191 offset 0 len 405504
[ 728.541883] 112: pfn 66684290 offset 0 len 860160
[ 728.547317] 114: pfn 66684500 offset 0 len 200704
[ 728.552742] 116: pfn 66684549 offset 0 len 299008
[ 728.558194] 118: pfn 66684622 offset 0 len 192512
[ 728.563609] 120: pfn 66684669 offset 0 len 102400
[ 728.569033] 122: pfn 66684694 offset 0 len 196608
[ 728.574473] 124: pfn 66684742 offset 0 len 98304
[ 728.579812] 126: pfn 66684766 offset 0 len 94208
[ 728.585165] 128: pfn 66684789 offset 0 len 749568
[ 728.590597] 130: pfn 66684972 offset 0 len 1490944
[ 728.596119] 132: pfn 66685336 offset 0 len 684032
[ 728.601542] 134: pfn 66685503 offset 0 len 495616
[ 728.606964] 136: pfn 66685624 offset 0 len 196608
[ 728.612391] 138: pfn 66685672 offset 0 len 585728
[ 728.617814] 140: pfn 66685815 offset 0 len 94208
[ 728.623168] 142: pfn 66685838 offset 0 len 94208
[ 728.628505] 144: pfn 66685861 offset 0 len 77824
[ 728.633830] 146: pfn 66685880 offset 0 len 778240
[ 728.639286] 148: pfn 66686070 offset 0 len 196608
[ 728.644710] 150: pfn 66686118 offset 0 len 503808
[ 728.650165] 152: pfn 66686241 offset 0 len 85626880
[ 728.655805] 154: pfn 66707146 offset 0 len 13893632
[ 728.661430] 156: pfn 66710538 offset 0 len 3411968
[ 728.666977] 158: pfn 66711371 offset 0 len 151552
[ 728.672412] 160: pfn 66711408 offset 0 len 450560
[ 728.677849] 162: pfn 66711518 offset 0 len 155648
[ 728.683299] 164: pfn 66711556 offset 0 len 323584
[ 728.688722] 166: pfn 66711635 offset 0 len 163840
[ 728.694174] 168: pfn 66711675 offset 0 len 327680
[ 728.699607] 170: pfn 66711755 offset 0 len 159744
[ 728.705032] 172: pfn 66711794 offset 0 len 1130496
[ 728.710544] 174: pfn 66712070 offset 0 len 974848
[ 728.715968] 176: pfn 66712308 offset 0 len 651264
[ 728.721384] 178: pfn 66712467 offset 0 len 2285568
[ 728.726906] 180: pfn 66713025 offset 0 len 57344
[ 728.732236] 182: pfn 66713039 offset 0 len 868352
[ 728.737663] 184: pfn 66713251 offset 0 len 491520
[ 728.743100] 186: pfn 66713371 offset 0 len 389120
[ 728.748535] 188: pfn 66713466 offset 0 len 753664
[ 728.753982] 190: pfn 66713650 offset 0 len 475136
[ 728.759404] 192: pfn 66713766 offset 0 len 192512
[ 728.764839] 194: pfn 66713813 offset 0 len 192512
[ 728.770285] 196: pfn 66713860 offset 0 len 196608
[ 728.775704] 198: pfn 66713908 offset 0 len 98304
[ 728.781035] 200: pfn 66713932 offset 0 len 2113536
[ 728.786562] 202: pfn 66714448 offset 0 len 94208
[ 728.791894] 204: pfn 66714471 offset 0 len 290816
[ 728.797323] 206: pfn 66714542 offset 0 len 65536
[ 728.802653] 208: pfn 66714558 offset 0 len 2453504
[ 728.808195] 210: pfn 66715157 offset 0 len 790528
[ 728.813612] 212: pfn 66715350 offset 0 len 1056768
[ 728.819129] 214: pfn 66715608 offset 0 len 76349440
[ 728.824778] 216: pfn 66734248 offset 0 len 63995904
[ 728.830407] 218: pfn 66749872 offset 0 len 52539392
[ 728.836033] 220: pfn 66762699 offset 0 len 33771520
[ 728.841649] 222: pfn 66770944 offset 0 len 5955584
[ 728.847178] 224: pfn 66772398 offset 0 len 41803776
[ 728.852797] 226: pfn 66782604 offset 0 len 16384
[ 728.858151] 228: pfn 66782608 offset 0 len 8437760
[ 728.863669] 230: pfn 66784668 offset 0 len 17031168
[ 728.869319] 232: pfn 66788826 offset 0 len 22708224
[ 728.874946] 234: pfn 66794370 offset 0 len 29933568
[ 728.880563] 236: pfn 66801678 offset 0 len 4136960
[ 728.886086] 238: pfn 66820096 offset 0 len 2506752
[ 729.336992] WARNING: CPU: 40 PID: 1312 at drivers/scsi/scsi_lib.c:1010
scsi_init_io+0x103/0x170
[ 729.349150] Modules linked in: xfs dm_mod btrfs xor zstd_decompress
zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac
x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
crct10dif_pclmul mgag200 crc32_pclmul ttm crc32c_intel ghash_clmulni_intel
drm_kms_helper snd_pcm syscopyarea sysfillrect sysimgblt aesni_intel
snd_timer ahci crypto_simd ipmi_si fb_sys_fops mei_me libahci snd
ipmi_devintf cryptd soundcore glue_helper ipmi_msghandler pcspkr drm ioatdma
mei libata joydev lpc_ich i2c_i801 dca wmi acpi_pad acpi_power_meter
pcc_cpufreq ip_tables
[ 729.409451] CPU: 40 PID: 1312 Comm: kworker/40:1H Tainted: G W
I 5.2.0-rc5-00001-g82f8097 #1
[ 729.421013] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
SE5C610.86B.01.01.0008.021120151325 02/11/2015
[ 729.433604] Workqueue: kblockd blk_mq_run_work_fn
[ 729.439805] RIP: 0010:scsi_init_io+0x103/0x170
[ 729.445726] Code: 5c 41 5d c3 be 01 00 00 00 e9 49 ff ff ff 8b 45 70 89
83 08 01 00 00 f6 45 1a 01 75 92 31 c0 48 83 c4 08 5b 5d 41 5c 41 5d c3 <0f>
0b 48 83 c4 08 b8 0a 00 00 00 5b 5d 41 5c 41 5d c3 48 8b 75 38
[ 729.468787] RSP: 0018:ffffac02cea47c78 EFLAGS: 00010246
[ 729.475705] RAX: 000000001eafb2a8 RBX: ffff95df3e7a3950 RCX:
0000000000000000
[ 729.484790] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
ffff95df3e7a3950
[ 729.493889] RBP: ffff95df3e7a3840 R08: 0000000000000000 R09:
ffff95df3e7a3988
[ 729.502995] R10: ffffac02cc71fda0 R11: fefefefefefefeff R12:
0000000000000801
[ 729.512123] R13: 0000000000000000 R14: 0000000000000000 R15:
ffff95df3ba8e800
[ 729.521261] FS: 0000000000000000(0000) GS:ffff95df7f780000(0000)
knlGS:0000000000000000
[ 729.531505] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 729.539140] CR2: 00007f7465922814 CR3: 0000000a80c0a006 CR4:
00000000001606e0
[ 729.548359] Call Trace:
[ 729.552355] sd_init_command+0x2ee/0xaf0 [sd_mod]
[ 729.558882] scsi_queue_rq+0x559/0xa80
[ 729.564354] blk_mq_dispatch_rq_list+0x3da/0x5b0
[ 729.570820] ? syscall_return_via_sysret+0x10/0x7f
[ 729.577488] ? __switch_to_asm+0x40/0x70
[ 729.583192] ? elv_rb_del+0x1f/0x30
[ 729.588417] ? deadline_remove_request+0x55/0xc0
[ 729.594915] blk_mq_do_dispatch_sched+0x76/0x120
[ 729.601424] blk_mq_sched_dispatch_requests+0x100/0x170
[ 729.608626] __blk_mq_run_hw_queue+0x60/0x130
[ 729.614873] process_one_work+0x19c/0x3c0
[ 729.620734] worker_thread+0x3c/0x3b0
[ 729.626210] ? process_one_work+0x3c0/0x3c0
[ 729.632280] kthread+0x11e/0x140
[ 729.637281] ? kthread_park+0x90/0x90
[ 729.642773] ret_from_fork+0x35/0x40
[ 729.648172] ---[ end trace 929c66c4a933d3e4 ]---
[ 729.654870] print_req_error: I/O error, dev sdb, sector 514831016 flags
801
[ 729.849057] XFS (sdb1): writeback error on sector 514831016
[ 738.467814] WARNING: CPU: 1 PID: 1460 at block/blk-core.c:1128
submit_bio+0xa5/0x1d0
[ 738.478959] Modules linked in: xfs dm_mod btrfs xor zstd_decompress
zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac
x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
crct10dif_pclmul mgag200 crc32_pclmul ttm crc32c_intel ghash_clmulni_intel
drm_kms_helper snd_pcm syscopyarea sysfillrect sysimgblt aesni_intel
snd_timer ahci crypto_simd ipmi_si fb_sys_fops mei_me libahci snd
ipmi_devintf cryptd soundcore glue_helper ipmi_msghandler pcspkr drm ioatdma
mei libata joydev lpc_ich i2c_i801 dca wmi acpi_pad acpi_power_meter
pcc_cpufreq ip_tables
[ 738.538361] CPU: 1 PID: 1460 Comm: fio Tainted: G W I
5.2.0-rc5-00001-g82f8097 #1
[ 738.548561] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
SE5C610.86B.01.01.0008.021120151325 02/11/2015
[ 738.560805] RIP: 0010:submit_bio+0xa5/0x1d0
[ 738.566050] Code: 00 00 65 48 01 35 b3 39 bf 6f 8b 0d a1 96 c4 01 85 c9
0f 85 9b 00 00 00 8b 43 28 85 c0 75 a5 8b 53 10 0f b6 ca 83 f9 01 75 9a <0f>
0b 0f b7 4b 68 0f b7 73 14 45 31 c9 4c 8b 43 20 48 c7 c7 f1 99
[ 738.588175] RSP: 0018:ffffac02cdaebb38 EFLAGS: 00010246
[ 738.594572] RAX: 0000000000000000 RBX: ffff95df3ce59e40 RCX:
0000000000000001
[ 738.603102] RDX: 0000000000000801 RSI: ffff95df3ce59e00 RDI:
ffff95df3ce59e40
[ 738.611611] RBP: 0000000000000000 R08: 0000000000000002 R09:
0000000000000000
[ 738.620118] R10: 0000000000030340 R11: ffff95dffffd5000 R12:
ffffac02cdaebe08
[ 738.628626] R13: ffff95df3ce59e00 R14: ffffac02cdaebe08 R15:
0000000000000000
[ 738.637125] FS: 00007f7492b0ca80(0000) GS:ffff95df7f240000(0000)
knlGS:0000000000000000
[ 738.646689] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 738.653626] CR2: 00007f7465355004 CR3: 0000001fdc20e004 CR4:
00000000001606e0
[ 738.662117] Call Trace:
[ 738.665420] xfs_submit_ioend+0x84/0x1e0 [xfs]
[ 738.670931] xfs_do_writepage+0x246/0x5a0 [xfs]
[ 738.676514] write_cache_pages+0x16b/0x440
[ 738.681638] ? xfs_vm_releasepage+0x80/0x80 [xfs]
[ 738.687425] xfs_vm_writepages+0x6b/0xa0 [xfs]
[ 738.692895] do_writepages+0x4b/0xf0
[ 738.697392] ? iomap_file_buffered_write+0x62/0x90
[ 738.703240] ? iomap_write_end+0x240/0x240
[ 738.708335] ? xfs_iunlock+0x165/0x220 [xfs]
[ 738.713600] ? __filemap_fdatawrite_range+0xc1/0x100
[ 738.719641] __filemap_fdatawrite_range+0xc1/0x100
[ 738.725489] file_write_and_wait_range+0x4c/0xa0
[ 738.731165] xfs_file_fsync+0x5d/0x230 [xfs]
[ 738.736435] do_fsync+0x38/0x60
[ 738.740439] __x64_sys_fsync+0x10/0x20
[ 738.745124] do_syscall_64+0x5b/0x1e0
[ 738.749711] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 738.755848] RIP: 0033:0x7f747e8fe70d
[ 738.760333] Code: bc 20 00 00 75 10 b8 4a 00 00 00 0f 05 48 3d 01 f0 ff
ff 73 31 c3 48 83 ec 08 e8 6e f7 ff ff 48 89 04 24 b8 4a 00 00 00 0f 05 <48>
8b 3c 24 48 89 c2 e8 b7 f7 ff ff 48 89 d0 48 83 c4 08 48 3d 01
[ 738.782345] RSP: 002b:00007ffe67fd0f00 EFLAGS: 00000293 ORIG_RAX:
000000000000004a
[ 738.791335] RAX: ffffffffffffffda RBX: 000055d51d05b0f0 RCX:
00007f747e8fe70d
[ 738.799843] RDX: 0000000000200000 RSI: 000055d51d05b0f0 RDI:
0000000000000003
[ 738.808354] RBP: 00007f7465e2e418 R08: 0000000000000008 R09:
0000000000000001
[ 738.816865] R10: 00000000897b0000 R11: 0000000000000293 R12:
0000000000000000
[ 738.825368] R13: 0000001000000000 R14: 00007f747ac8ca50 R15:
0000000000000000
[ 738.833858] ---[ end trace 929c66c4a933d3e5 ]---
[ 738.839553] bio: c000/801 98 532946720 0
[ 738.844459] 0: pfn 32001772 offset 0 len 184487936
[ 738.850312] 2: pfn 61947904 offset 0 len 4194304
[ 738.855993] 4: pfn 61945856 offset 0 len 352256
[ 738.861521] 6: pfn 61945942 offset 0 len 3842048
[ 738.867130] 8: pfn 61943808 offset 0 len 4194304
[ 738.872741] 10: pfn 61941760 offset 0 len 4194304
[ 738.878426] 12: pfn 61939712 offset 0 len 3362816
[ 738.884095] 14: pfn 61940533 offset 0 len 831488
[ 738.889666] 16: pfn 61937664 offset 0 len 4194304
[ 738.895333] 18: pfn 32165430 offset 0 len 243589120
[ 738.901159] 20: pfn 61935616 offset 0 len 4194304
[ 738.906784] 22: pfn 61933568 offset 0 len 4194304
[ 738.912414] 24: pfn 61931520 offset 0 len 4194304
[ 738.918029] 26: pfn 61929472 offset 0 len 4194304
[ 738.923661] 28: pfn 32224900 offset 0 len 242958336
[ 738.929470] 30: pfn 61927424 offset 0 len 4194304
[ 738.935063] 32: pfn 32284216 offset 0 len 243081216
[ 738.940858] 34: pfn 61925376 offset 0 len 4194304
[ 738.946466] 36: pfn 32343562 offset 0 len 243232768
[ 738.952242] 38: pfn 32402945 offset 0 len 16384
[ 738.957612] 40: pfn 61923328 offset 0 len 4194304
[ 738.963182] 42: pfn 61921280 offset 0 len 4194304
[ 738.968774] 44: pfn 61919232 offset 0 len 3047424
[ 738.974328] 46: pfn 61919976 offset 0 len 1146880
[ 738.979888] 48: pfn 61917184 offset 0 len 4194304
[ 738.985456] 50: pfn 61915136 offset 0 len 4194304
[ 738.991017] 52: pfn 32462295 offset 0 len 243081216
[ 738.996775] 54: pfn 61913088 offset 0 len 4194304
[ 739.002323] 56: pfn 61911040 offset 0 len 3039232
[ 739.007892] 58: pfn 61911782 offset 0 len 1155072
[ 739.013445] 60: pfn 61908992 offset 0 len 1875968
[ 739.018982] 62: pfn 61909450 offset 0 len 20480
[ 739.024332] 64: pfn 61909455 offset 0 len 2297856
[ 739.029865] 66: pfn 61906944 offset 0 len 4194304
[ 739.035420] 68: pfn 61904896 offset 0 len 4194304
[ 739.040947] 70: pfn 32640459 offset 0 len 243032064
[ 739.046677] 72: pfn 61902848 offset 0 len 4194304
[ 739.052232] 74: pfn 32699793 offset 0 len 243228672
[ 739.057977] 76: pfn 61900800 offset 0 len 4194304
[ 739.063495] 78: pfn 61898752 offset 0 len 4194304
[ 739.069023] 80: pfn 61896704 offset 0 len 4194304
[ 739.074520] 82: pfn 32759175 offset 0 len 242909184
[ 739.080230] 84: pfn 61894656 offset 0 len 4194304
[ 739.085743] 86: pfn 32818479 offset 0 len 243081216
[ 739.091446] 88: pfn 61892608 offset 0 len 4194304
[ 739.096936] 90: pfn 61890560 offset 0 len 1286144
[ 739.102418] 92: pfn 61890874 offset 0 len 2908160
[ 739.107895] 94: pfn 61888512 offset 0 len 4194304
[ 739.113390] 96: pfn 61886464 offset 0 len 2572288
[ 739.118938] print_req_error: I/O error, dev sdb, sector 532946720 flags
801
[ 739.430464] XFS (sdb1): writeback error on sector 532946720
[ 742.762917] fio: pid=0, err=5/file:filesetup.c:190, func=fsync,
error=Input/output error
[ 742.762920]
Sorry for missing your dmesg log.
Now, the issue is quite clear, and it is one int overflow issue,
which should be addressed by the patch I posted.
Please go ahead and give a test on V2!
thanks,
Ming
9:46 p.m.
Hi, Ming. I have tested your v2 fix patch on fio test 5 times, error has
gone. Thanks.
在 2019/7/1 下午3:29, Ming Lei 写道:
On Mon, Jul 01, 2019 at 03:18:01PM +0800, Liu Yiding wrote:
> 在 2019/7/1 下午3:14, Ming Lei 写道:
>> On Mon, Jul 01, 2019 at 02:54:31PM +0800, Liu Yiding wrote:
>>> 在 2019/7/1 下午2:44, Ming Lei 写道:
>>>> On Mon, Jul 01, 2019 at 12:49:51PM +0800, Liu Yiding wrote:
>>>>> 在 2019/7/1 上午10:47, Ming Lei 写道:
>>>>>> [snip]
>>>>>>>>> ------
>>>>>>>>>
>>>>>>>>> 2412 [ 112.558155] XFS (sdb1): Ending clean mount
>>>>>>>>> 2413 [ 1317.330820] WARNING: CPU: 63 PID: 1495 at
>>>>>>>>> drivers/scsi/scsi_lib.c:1034
scsi_init_io+0x11a/0x2e0
>>>>>>>>> 2414 [ 1317.344000] Modules linked in: xfs dm_mod
btrfs xor zstd_decompress
>>>>>>>>> zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl
sb_edac
>>>>>>>>> x86_pkg_temp_thermal intel_powerclamp coretemp
kvm_i ntel kvm irqbypass
>>>>>>>>> crct10dif_pclmul crc32_pclmul crc32c_intel mgag200
snd_pcm
>>>>>>>>> ghash_clmulni_intel ttm snd_timer drm_kms_helper
ipmi_si ahci snd
>>>>>>>>> aesni_intel syscopyarea libahci ipmi_devi ntf
sysfillrect sysimgblt
>>>>>>>>> crypto_simd mei_me fb_sys_fops soundcore ioatdma
cryptd ipmi_msghandler
>>>>>>>>> glue_helper pcspkr drm libata mei joydev lpc_ich
i2c_i801 dca wmi
>>>>>>>>> acpi_power_meter acpi _pad pcc_cpufreq ip_tables
>>>>>>>>> 2415 [ 1317.408135] CPU: 63 PID: 1495 Comm:
kworker/63:1H Tainted:
>>>>>>>>> G I 5.2.0-rc5-00001-gc2d55e2 #1
>>>>>>>>> 2416 [ 1317.420350] Hardware name: Intel Corporation
S2600WTT/S2600WTT, BIOS
>>>>>>>>> SE5C610.86B.01.01.0008.021120151325 02/11/2015
>>>>>>>>> 2417 [ 1317.433545] Workqueue: kblockd
blk_mq_run_work_fn
>>>>>>>>> 2418 [ 1317.440295] RIP:
0010:scsi_init_io+0x11a/0x2e0
>>>>>>>>> 2419 [ 1317.446746] Code: be 01 00 00 00 e9 44 ff ff
ff 8b 45 70 89 83 08 01
>>>>>>>>> 00 00 f6 45 1a 01 75 8a 31 c0 48 83 c4 08 5b 5d 41 5c
41 5d 41 5e 41 5f c3
>>>>>>>>> <0f> 0b 8b 4d 18 8b 55 28 45 31 c9 48 c7
c6 98 da 69 82 48 c7 c7 f0
>>>>>>>>> 2420 [ 1317.470863] RSP: 0018:ffffa5fa4f7e7c68
EFLAGS: 00010246
>>>>>>>>> 2421 [ 1317.478294] RAX: 000000003d17d1b8 RBX:
ffff8ae6cb742690 RCX:
>>>>>>>>> 0000000000000000
>>>>>>>>> 2422 [ 1317.487878] RDX: 0000000000000000 RSI:
0000000000000000 RDI:
>>>>>>>>> ffff8ae6cb742690
>>>>>>>>> 2423 [ 1317.497469] RBP: ffff8ae6cb742580 R08:
0000000000023003 R09:
>>>>>>>>> ffff8ae6cb7426c8
>>>>>>>>> 2424 [ 1317.507057] R10: ffffa5fa4c7d7da0 R11:
fefefefefefefeff R12:
>>>>>>>>> 0000000000000801
>>>>>>>>> 2425 [ 1317.516650] R13: 0000000000000000 R14:
0000000000000000 R15:
>>>>>>>>> ffff8ac800bd1800
>>>>>>>>> 2426 [ 1317.526253] FS: 0000000000000000(0000)
GS:ffff8b077fac0000(0000)
>>>>>>>>> knlGS:0000000000000000
>>>>>>>>> 2427 [ 1317.536918] CS: 0010 DS: 0000 ES: 0000 CR0:
0000000080050033
>>>>>>>>> 2428 [ 1317.544952] CR2: 00007f39a5818004 CR3:
000000343440a006 CR4:
>>>>>>>>> 00000000001606e0
>>>>>>>>> 2429 [ 1317.554554] Call Trace:
>>>>>>>>> 2430 [ 1317.558921] sd_init_command+0x2ee/0xaf0
[sd_mod]
>>>>>>>>> 2431 [ 1317.565800] scsi_queue_rq+0x559/0xa80
>>>>>>>>> 2432 [ 1317.571607]
blk_mq_dispatch_rq_list+0x3da/0x5b0
>>>>>>>>> 2433 [ 1317.578375] ?
syscall_return_via_sysret+0x10/0x7f
>>>>>>>>> 2434 [ 1317.585339] ? __switch_to_asm+0x40/0x70
>>>>>>>>> 2435 [ 1317.591321] ? elv_rb_del+0x1f/0x30
>>>>>>>>> 2436 [ 1317.596807] ?
deadline_remove_request+0x55/0xc0
>>>>>>>>> 2437 [ 1317.603560]
blk_mq_do_dispatch_sched+0x76/0x120
>>>>>>>>> 2438 [ 1317.610302]
blk_mq_sched_dispatch_requests+0x100/0x170
>>>>>>>>> 2439 [ 1317.617733]
__blk_mq_run_hw_queue+0x60/0x130
>>>>>>>>> 2440 [ 1317.624200] process_one_work+0x19c/0x3c0
>>>>>>>>> 2441 [ 1317.630258] worker_thread+0x3c/0x3b0
>>>>>>>>> 2442 [ 1317.635937] ? process_one_work+0x3c0/0x3c0
>>>>>>>>> 2443 [ 1317.642183] kthread+0x11e/0x140
>>>>>>>>> 2444 [ 1317.647346] ? kthread_park+0x90/0x90
>>>>>>>>> 2445 [ 1317.652986] ret_from_fork+0x35/0x40
>>>>>>>>> 2446 [ 1317.658523] ---[ end trace 61026ff97261da6a
]---
>>>>>>>>> 2447 [ 1317.665327] scsi_dump_rq: rq 6(f:801/23003,
seg: 0)
>>>>>>>>> 2448 [ 1317.671601] bio 0: d602/801 0 1024971192 0
>>>>>>>>> 2449 [ 1317.676991] print_req_error: I/O error, dev
sdb, sector 1024971192
>>>>>>>>> flags 801
>>>>>>>>> 2450 [ 1317.685722] scsi_dump_rq: rq 8(f:801/23003,
seg: 0)
>>>>>>>>> 2451 [ 1317.691994] bio 0: d602/801 0 1033359800 0
>>>>>>>> The above line shows the bio is zero-size, and this bio
is submitted
>>>>>>>> from bio, since the bio->bi_flags indicates that
BIO_CLONED isn't set.
>>>>>>>>
>>>>>>>> Please apply the following debug patch, and we may see
where is wrong.
>>>>>>>>
>>>>>>>> diff --git a/block/blk-core.c b/block/blk-core.c
>>>>>>>> index 8340f69670d8..86bb61b0df83 100644
>>>>>>>> --- a/block/blk-core.c
>>>>>>>> +++ b/block/blk-core.c
>>>>>>>> @@ -1125,6 +1125,7 @@ blk_qc_t submit_bio(struct bio
*bio)
>>>>>>>> bio_devname(bio, b), count);
>>>>>>>> }
>>>>>>>> }
>>>>>>>> + WARN_ON(bio->bi_iter.bi_size == 0 &&
bio_op(bio) == REQ_OP_WRITE);
>>>>>>>> return generic_make_request(bio);
>>>>>>> Sorry for delay,
>>>>>>>
>>>>>>> ------
>>>>>>>
>>>>>>> [ 116.619136] 2019-06-29 16:12:41 mkdir -p /fs/sdb1
>>>>>>> [ 116.619138]
>>>>>>> [ 116.628743] 2019-06-29 16:12:41 mount -t xfs -o inode64
/dev/sdb1
>>>>>>> /fs/sdb1
>>>>>>> [ 116.628745]
>>>>>>> [ 116.634699] SGI XFS with ACLs, security attributes,
realtime, scrub,
>>>>>>> debug enabled
>>>>>>> [ 116.653110] XFS (sdb1): Mounting V5 Filesystem
>>>>>>> [ 116.670081] XFS (sdb1): Ending clean mount
>>>>>>> [ 690.106779] WARNING: CPU: 53 PID: 1477 at
block/blk-core.c:1128
>>>>>>> submit_bio+0x93/0x150
>>>>>>> [ 690.117972] Modules linked in: xfs dm_mod btrfs xor
zstd_decompress
>>>>>>> zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl
sb_edac
>>>>>>> x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm
irqbypass
>>>>>>> mgag200 crct10dif_pclmul crc32_pclmul ttm crc32c_intel
snd_pcm
>>>>>>> ghash_clmulni_intel drm_kms_helper snd_timer ahci syscopyarea
sysfillrect
>>>>>>> snd aesni_intel sysimgblt libahci ipmi_si soundcore
fb_sys_fops mei_me
>>>>>>> crypto_simd ipmi_devintf ioatdma cryptd pcspkr drm
ipmi_msghandler
>>>>>>> glue_helper libata mei joydev lpc_ich i2c_i801 dca wmi
acpi_power_meter
>>>>>>> acpi_pad pcc_cpufreq ip_tables
>>>>>>> [ 690.176967] CPU: 53 PID: 1477 Comm: fio Tainted: G
W I
>>>>>>> 5.2.0-rc5-00002-g9fa09bd #1
>>>>>>> [ 690.187198] Hardware name: Intel Corporation
S2600WTT/S2600WTT, BIOS
>>>>>>> SE5C610.86B.01.01.0008.021120151325 02/11/2015
>>>>>>> [ 690.199375] RIP: 0010:submit_bio+0x93/0x150
>>>>>>> [ 690.204583] Code: ee 74 45 83 e1 01 74 26 65 48 01 35 bf
39 3f 45 8b 0d
>>>>>>> ad 96 c4 01 85 c9 75 56 8b 43 28 85 c0 75 b3 0f b6 53 10 83
fa 01 75 aa <0f>
>>>>>>> 0b eb a6 65 48 8b 14 25 c0 6b 01 00 65 48 01 35 88 39 3f 45
48
>>>>>>> [ 690.226668] RSP: 0018:ffffb2d90efc3b40 EFLAGS: 00010246
>>>>>>> [ 690.233072] RAX: 0000000000000000 RBX: ffff9654c6ad9f40
RCX:
>>>>>>> 0000000000000801
>>>>>>> [ 690.241617] RDX: 0000000000000001 RSI: ffff9654c6ad9f00
RDI:
>>>>>>> ffff9654c6ad9f40
>>>>>>> [ 690.250146] RBP: 0000000000000000 R08: 0000000000000002
R09:
>>>>>>> 0000000000000000
>>>>>>> [ 690.258667] R10: 00000000000303a0 R11: ffff96757ffd4000
R12:
>>>>>>> ffffb2d90efc3e08
>>>>>>> [ 690.267183] R13: ffff9654c6ad9f00 R14: ffffb2d90efc3e08
R15:
>>>>>>> 0000000000000000
>>>>>>> [ 690.275696] FS: 00007fa8e005da80(0000)
GS:ffff9654ffac0000(0000)
>>>>>>> knlGS:0000000000000000
>>>>>>> [ 690.285282] CS: 0010 DS: 0000 ES: 0000 CR0:
0000000080050033
>>>>>>> [ 690.292252] CR2: 00007fa8cc05b3c0 CR3: 0000001fc8518001
CR4:
>>>>>>> 00000000001606e0
>>>>>>> [ 690.300770] Call Trace:
>>>>>>> [ 690.304123] xfs_submit_ioend+0x84/0x1e0 [xfs]
>>>>>>> [ 690.309668] xfs_do_writepage+0x246/0x5a0 [xfs]
>>>>>>> [ 690.315279] write_cache_pages+0x16b/0x440
>>>>>>> [ 690.320420] ? xfs_vm_releasepage+0x80/0x80 [xfs]
>>>>>>> [ 690.326238] xfs_vm_writepages+0x6b/0xa0 [xfs]
>>>>>>> [ 690.331742] do_writepages+0x4b/0xf0
>>>>>>> [ 690.336275] ? iomap_file_buffered_write+0x62/0x90
>>>>>>> [ 690.342162] ? iomap_write_end+0x240/0x240
>>>>>>> [ 690.347295] ? xfs_iunlock+0x165/0x220 [xfs]
>>>>>>> [ 690.352595] ? __filemap_fdatawrite_range+0xc1/0x100
>>>>>>> [ 690.358673] __filemap_fdatawrite_range+0xc1/0x100
>>>>>>> [ 690.364571] ? __switch_to_asm+0x34/0x70
>>>>>>> [ 690.369484] file_write_and_wait_range+0x4c/0xa0
>>>>>>> [ 690.375198] xfs_file_fsync+0x5d/0x230 [xfs]
>>>>>>> [ 690.380499] do_fsync+0x38/0x60
>>>>>>> [ 690.384534] __x64_sys_fsync+0x10/0x20
>>>>>>> [ 690.389243] do_syscall_64+0x5b/0x1e0
>>>>>>> [ 690.393857] entry_SYSCALL_64_after_hwframe+0x44/0xa9
>>>>>>> [ 690.400020] RIP: 0033:0x7fa8cbe4f70d
>>>>>>> [ 690.404533] Code: bc 20 00 00 75 10 b8 4a 00 00 00 0f 05
48 3d 01 f0 ff
>>>>>>> ff 73 31 c3 48 83 ec 08 e8 6e f7 ff ff 48 89 04 24 b8 4a 00
00 00 0f 05 <48>
>>>>>>> 8b 3c 24 48 89 c2 e8 b7 f7 ff ff 48 89 d0 48 83 c4 08 48 3d
01
>>>>>>> [ 690.426581] RSP: 002b:00007ffed6954160 EFLAGS: 00000293
ORIG_RAX:
>>>>>>> 000000000000004a
>>>>>>> [ 690.435583] RAX: ffffffffffffffda RBX: 00005573363790f0
RCX:
>>>>>>> 00007fa8cbe4f70d
>>>>>>> [ 690.444102] RDX: 0000000000200000 RSI: 00005573363790f0
RDI:
>>>>>>> 0000000000000003
>>>>>>> [ 690.452615] RBP: 00007fa8b337f418 R08: 0000000000000008
R09:
>>>>>>> 0000000000000001
>>>>>>> [ 690.461124] R10: 00000000897b0000 R11: 0000000000000293
R12:
>>>>>>> 0000000000000000
>>>>>>> [ 690.469631] R13: 0000001000000000 R14: 00007fa8c81dda50
R15:
>>>>>>> 0000000000000000
>>>>>>> [ 690.478130] ---[ end trace 90dd4ee6d5cdfcb8 ]---
>>>>>> I have just checked xfs code, and looks the only reason is that
bio size
>>>>>> is overflowed, that means the whole bio may reach 4GB. That might
be
>>>>>> possible in case of writeback. But I have to double check if that
is
>>>>>> the case, could you test the following patch and provide the
log?
>>>>>>
>>>>>> diff --git a/block/blk-core.c b/block/blk-core.c
>>>>>> index 8340f69670d8..9d61e408c608 100644
>>>>>> --- a/block/blk-core.c
>>>>>> +++ b/block/blk-core.c
>>>>>> @@ -1125,6 +1125,19 @@ blk_qc_t submit_bio(struct bio *bio)
>>>>>> bio_devname(bio, b), count);
>>>>>> }
>>>>>> }
>>>>>> + if (WARN_ON(bio->bi_iter.bi_size == 0 && bio_op(bio)
== REQ_OP_WRITE)) {
>>>>>> + int i;
>>>>>> + printk("bio: %hx/%hx %u %llu %u\n",
bio->bi_flags, bio->bi_opf,
>>>>>> + bio->bi_vcnt,
>>>>>> + (unsigned long
long)bio->bi_iter.bi_sector,
>>>>>> + bio->bi_iter.bi_size);
>>>>>> + for (i = 0; i < bio->bi_vcnt; i++) {
>>>>>> + struct bio_vec *bvec = &bio->bi_io_vec[i];
>>>>>> + printk("%d: pfn %lu offset %u len %u\n", i++,
>>>>>> + (unsigned
long)page_to_pfn(bvec->bv_page),
>>>>>> + bvec->bv_offset,
bvec->bv_len);
>>>>>> + }
>>>>>> + }
>>>>>> return generic_make_request(bio);
>>>>>> }
>>>>> the newest log and i also see you fix patch, i will test your fix
patch
>>>>> more times to check it.
>>>>>
>>>>> ------
>>>>>
>>>>> [ 54.656016] 2019-07-01 11:48:52 mkfs -t xfs /dev/sdb1
>>>>> [ 54.656017]
>>>>> [ 119.228467] WARNING: CPU: 5 PID: 1530 at block/blk-core.c:1128
>>>>> submit_bio+0xa5/0x1d0
>>>>> [ 119.240810] Modules linked in: dm_mod btrfs xor zstd_decompress
>>>>> zstd_compress raid6_pq libcrc32c sd_mod intel_rapl sg sb_edac
>>>>> x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm
irqbypass
>>>>> mgag200 crct10dif_pclmul snd_pcm crc32_pclmul ttm crc32c_intel
snd_timer
>>>>> drm_kms_helper ghash_clmulni_intel snd syscopyarea ipmi_si
sysfillrect
>>>>> soundcore aesni_intel sysimgblt ipmi_devintf crypto_simd fb_sys_fops
ahci
>>>>> mei_me libahci cryptd ipmi_msghandler pcspkr glue_helper drm mei
libata
>>>>> joydev ioatdma lpc_ich i2c_i801 dca wmi acpi_power_meter acpi_pad
>>>>> pcc_cpufreq ip_tables
>>>>> [ 119.306252] CPU: 5 PID: 1530 Comm: mkfs.xfs Tainted: G W I
>>>>> 5.2.0-rc5-00001-g82f8097 #1
>>>>> [ 119.318274] Hardware name: Intel Corporation S2600WTT/S2600WTT,
BIOS
>>>>> SE5C610.86B.01.01.0008.021120151325 02/11/2015
>>>>> [ 119.331852] RIP: 0010:submit_bio+0xa5/0x1d0
>>>>> [ 119.338422] Code: 00 00 65 48 01 35 b3 39 5f 76 8b 0d a1 96 c4 01
85 c9
>>>>> 0f 85 9b 00 00 00 8b 43 28 85 c0 75 a5 8b 53 10 0f b6 ca 83 f9 01 75
9a <0f>
>>>>> 0b 0f b7 4b 68 0f b7 73 14 45 31 c9 4c 8b 43 20 48 c7 c7 f1 99
>>>>> [ 119.363296] RSP: 0018:ffff940c4fb17e58 EFLAGS: 00010246
>>>>> [ 119.371107] RAX: 0000000000000000 RBX: ffff88a53e3d6900 RCX:
>>>>> 0000000000000001
>>>>> [ 119.381068] RDX: 0000000000040801 RSI: ffff88868138c000 RDI:
>>>>> ffff88a53e3d6900
>>>>> [ 119.391041] RBP: ffff88a5448b0000 R08: ffff88a53e3d6900 R09:
>>>>> 0000000000000000
>>>>> [ 119.401003] R10: 0000000000000000 R11: 0000000000000000 R12:
>>>>> 0000000000000000
>>>>> [ 119.410967] R13: 0000000000000000 R14: 0000000000000000 R15:
>>>>> 0000000000000000
>>>>> [ 119.420881] FS: 00007f37eba81780(0000) GS:ffff88a57f340000(0000)
>>>>> knlGS:0000000000000000
>>>>> [ 119.431839] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>> [ 119.440161] CR2: 00005630eb09d000 CR3: 0000001fc711c002 CR4:
>>>>> 00000000001606e0
>>>>> [ 119.450017] Call Trace:
>>>>> [ 119.454595] submit_bio_wait+0x53/0x80
>>>>> [ 119.460636] blkdev_issue_flush+0x81/0xb0
>>>>> [ 119.466930] blkdev_fsync+0x2f/0x40
>>>>> [ 119.472590] do_fsync+0x38/0x60
>>>>> [ 119.477810] __x64_sys_fsync+0x10/0x20
>>>>> [ 119.483722] do_syscall_64+0x5b/0x1e0
>>>>> [ 119.489516] entry_SYSCALL_64_after_hwframe+0x44/0xa9
>>>>> [ 119.496857] RIP: 0033:0x7f37eb2146f0
>>>>> [ 119.502502] Code: 48 3d 01 f0 ff ff 73 01 c3 48 8b 0d b2 78 20 00
f7 d8
>>>>> 64 89 01 48 83 c8 ff c3 83 3d d9 bc 20 00 00 75 10 b8 4a 00 00 00 0f
05 <48>
>>>>> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 6e f7 ff ff 48 89 04 24
>>>>> [ 119.526842] RSP: 002b:00007fffb27f4878 EFLAGS: 00000246 ORIG_RAX:
>>>>> 000000000000004a
>>>>> [ 119.536975] RAX: ffffffffffffffda RBX: 0000000000000811 RCX:
>>>>> 00007f37eb2146f0
>>>>> [ 119.546592] RDX: 00005630e91ba8e0 RSI: 0000000000000000 RDI:
>>>>> 0000000000000004
>>>>> [ 119.556171] RBP: 0000000000000004 R08: 00005630eb02b8a0 R09:
>>>>> 00007fffb27f4e00
>>>>> [ 119.565753] R10: 000000000000009b R11: 0000000000000246 R12:
>>>>> 00007fffb27f4c60
>>>>> [ 119.575339] R13: 00007fffb27f4a30 R14: 00005630eb048800 R15:
>>>>> 00005630eb05a800
>>>>> [ 119.584940] ---[ end trace 46a2e1214c930529 ]---
>>>>> [ 119.591765] bio: 0/801 0 0 0
>>>>>
>>>> Hi Yiding,
>>>>
>>>> The above warning is false positive, and it is flush request, and
>>>> it isn't same with your original report in which the following
warning
>>>> is triggered.
>>>>
>>>> "drivers/scsi/scsi_lib.c:1034 scsi_init_io+0x11a/0x2e0"
>>>>
>>>> So please run the test kernel with the debug patch I sent to you today
>>>> and try to trigger the warning in scsi_init_io().
>>> ------
>>>
>>> [ 728.863669] 230: pfn 66784668 offset 0 len 17031168
>>> [ 728.869319] 232: pfn 66788826 offset 0 len 22708224
>>> [ 728.874946] 234: pfn 66794370 offset 0 len 29933568
>>> [ 728.880563] 236: pfn 66801678 offset 0 len 4136960
>>> [ 728.886086] 238: pfn 66820096 offset 0 len 2506752
>> Hi Yiding,
>>
>> Could you include all logs dumped by the debug patch?
>>
>> + if (WARN_ON(bio->bi_iter.bi_size == 0 && bio_op(bio) ==
REQ_OP_WRITE)) {
>> + int i;
>> + printk("bio: %hx/%hx %u %llu %u\n", bio->bi_flags,
bio->bi_opf,
>> + bio->bi_vcnt,
>> + (unsigned long long)bio->bi_iter.bi_sector,
>> + bio->bi_iter.bi_size);
>> + for (i = 0; i < bio->bi_vcnt; i++) {
>> + struct bio_vec *bvec = &bio->bi_io_vec[i];
>> + printk("%d: pfn %lu offset %u len %u\n", i++,
>> + (unsigned long)page_to_pfn(bvec->bv_page),
>> + bvec->bv_offset, bvec->bv_len);
>> + }
>> + }
>
> Hi, Ming. I have attatched all log in last email, can you receive that.
>
> ------
>
> [ 124.334834] WARNING: CPU: 10 PID: 1421 at block/blk-core.c:1128
> submit_bio+0xa5/0x1d0
> [ 124.347274] Modules linked in: dm_mod btrfs xor zstd_decompress
> zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac
> x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
> crct10dif_pclmul mgag200 crc32_pclmul ttm crc32c_intel ghash_clmulni_intel
> drm_kms_helper snd_pcm syscopyarea sysfillrect sysimgblt aesni_intel
> snd_timer ahci crypto_simd ipmi_si fb_sys_fops mei_me libahci snd
> ipmi_devintf cryptd soundcore glue_helper ipmi_msghandler pcspkr drm ioatdma
> mei libata joydev lpc_ich i2c_i801 dca wmi acpi_pad acpi_power_meter
> pcc_cpufreq ip_tables
> [ 124.412706] CPU: 10 PID: 1421 Comm: mkfs.xfs Tainted: G W I
> 5.2.0-rc5-00001-g82f8097 #1
> [ 124.424823] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
> SE5C610.86B.01.01.0008.021120151325 02/11/2015
> [ 124.438401] RIP: 0010:submit_bio+0xa5/0x1d0
> [ 124.444974] Code: 00 00 65 48 01 35 b3 39 bf 6f 8b 0d a1 96 c4 01 85 c9
> 0f 85 9b 00 00 00 8b 43 28 85 c0 75 a5 8b 53 10 0f b6 ca 83 f9 01 75 9a <0f>
> 0b 0f b7 4b 68 0f b7 73 14 45 31 c9 4c 8b 43 20 48 c7 c7 f1 99
> [ 124.469850] RSP: 0018:ffffac02ce72fe58 EFLAGS: 00010246
> [ 124.477663] RAX: 0000000000000000 RBX: ffff95df3d416780 RCX:
> 0000000000000001
> [ 124.487636] RDX: 0000000000040801 RSI: ffff95df3b4b6800 RDI:
> ffff95df3d416780
> [ 124.497605] RBP: ffff95df49ce8000 R08: ffff95df3d416780 R09:
> 0000000000000000
> [ 124.507569] R10: 0000000000000000 R11: 0000000000000000 R12:
> 0000000000000000
> [ 124.517535] R13: 0000000000000000 R14: 0000000000000000 R15:
> 0000000000000000
> [ 124.527450] FS: 00007fee53609780(0000) GS:ffff95df7f480000(0000)
> knlGS:0000000000000000
> [ 124.538413] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 124.546738] CR2: 00005584d7290000 CR3: 0000001fbcd18003 CR4:
> 00000000001606e0
> [ 124.556597] Call Trace:
> [ 124.561176] submit_bio_wait+0x53/0x80
> [ 124.567220] blkdev_issue_flush+0x81/0xb0
> [ 124.573512] blkdev_fsync+0x2f/0x40
> [ 124.579172] do_fsync+0x38/0x60
> [ 124.584391] __x64_sys_fsync+0x10/0x20
> [ 124.590301] do_syscall_64+0x5b/0x1e0
> [ 124.596095] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [ 124.603437] RIP: 0033:0x7fee52d9c6f0
> [ 124.609082] Code: 48 3d 01 f0 ff ff 73 01 c3 48 8b 0d b2 78 20 00 f7 d8
> 64 89 01 48 83 c8 ff c3 83 3d d9 bc 20 00 00 75 10 b8 4a 00 00 00 0f 05 <48>
> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 6e f7 ff ff 48 89 04 24
> [ 124.633417] RSP: 002b:00007ffc8c42b4d8 EFLAGS: 00000246 ORIG_RAX:
> 000000000000004a
> [ 124.643548] RAX: ffffffffffffffda RBX: 0000000000000811 RCX:
> 00007fee52d9c6f0
> [ 124.653166] RDX: 00005584d6db18e0 RSI: 0000000000000000 RDI:
> 0000000000000004
> [ 124.662747] RBP: 0000000000000004 R08: 00005584d721e8a0 R09:
> 00007ffc8c42ba00
> [ 124.672331] R10: 000000000000009b R11: 0000000000000246 R12:
> 00007ffc8c42b8c0
> [ 124.681919] R13: 00007ffc8c42b690 R14: 00005584d723b800 R15:
> 00005584d724d800
> [ 124.691520] ---[ end trace 929c66c4a933d3e2 ]---
> [ 124.698346] bio: 0/801 0 0 0
> [ 124.706748] meta-data=/dev/sdb1 isize=512 agcount=4,
> agsize=48838198 blks
> [ 124.706752]
> [ 124.721970] = sectsz=4096 attr=2,
> projid32bit=1
> [ 124.721972]
> [ 124.736132] = crc=1 finobt=1, sparse=0,
> rmapbt=0, reflink=0
> [ 124.736134]
> [ 124.751645] data = bsize=4096 blocks=195352790,
> imaxpct=25
> [ 124.751647]
> [ 124.765522] = sunit=0 swidth=0 blks
> [ 124.765524]
> [ 124.778026] naming =version 2 bsize=4096 ascii-ci=0
> ftype=1
> [ 124.778028]
> [ 124.791128] log =internal log bsize=4096 blocks=95387,
> version=2
> [ 124.791130]
> [ 124.804674] = sectsz=4096 sunit=1 blks,
> lazy-count=1
> [ 124.804675]
> [ 124.818292] realtime =none extsz=4096 blocks=0,
> rtextents=0
> [ 124.818294]
> [ 124.830580] 2019-07-01 12:36:49 mkdir -p /fs/sdb1
> [ 124.830582]
> [ 124.840509] 2019-07-01 12:36:49 mount -t xfs -o inode64 /dev/sdb1
> /fs/sdb1
> [ 124.840511]
> [ 124.842506] SGI XFS with ACLs, security attributes, realtime, scrub,
> debug enabled
> [ 124.864478] XFS (sdb1): Mounting V5 Filesystem
> [ 124.886520] XFS (sdb1): Ending clean mount
> Starting watchdog daemon...
> Starting Update UTMP about System Runlevel Changes...
> [ 727.881932] WARNING: CPU: 0 PID: 1460 at block/blk-core.c:1128
> submit_bio+0xa5/0x1d0
> [ 727.892444] Modules linked in: xfs dm_mod btrfs xor zstd_decompress
> zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac
> x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
> crct10dif_pclmul mgag200 crc32_pclmul ttm crc32c_intel ghash_clmulni_intel
> drm_kms_helper snd_pcm syscopyarea sysfillrect sysimgblt aesni_intel
> snd_timer ahci crypto_simd ipmi_si fb_sys_fops mei_me libahci snd
> ipmi_devintf cryptd soundcore glue_helper ipmi_msghandler pcspkr drm ioatdma
> mei libata joydev lpc_ich i2c_i801 dca wmi acpi_pad acpi_power_meter
> pcc_cpufreq ip_tables
> [ 727.949105] CPU: 0 PID: 1460 Comm: fio Tainted: G W I
> 5.2.0-rc5-00001-g82f8097 #1
> [ 727.958733] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
> SE5C610.86B.01.01.0008.021120151325 02/11/2015
> [ 727.970404] RIP: 0010:submit_bio+0xa5/0x1d0
> [ 727.975087] Code: 00 00 65 48 01 35 b3 39 bf 6f 8b 0d a1 96 c4 01 85 c9
> 0f 85 9b 00 00 00 8b 43 28 85 c0 75 a5 8b 53 10 0f b6 ca 83 f9 01 75 9a <0f>
> 0b 0f b7 4b 68 0f b7 73 14 45 31 c9 4c 8b 43 20 48 c7 c7 f1 99
> [ 727.996090] RSP: 0018:ffffac02cdaebb38 EFLAGS: 00010246
> [ 728.001932] RAX: 0000000000000000 RBX: ffff95df4b754040 RCX:
> 0000000000000001
> [ 728.009908] RDX: 0000000000000801 RSI: ffff95df4b754000 RDI:
> ffff95df4b754040
> [ 728.017885] RBP: 0000000000000000 R08: 0000000000000002 R09:
> 0000000000000000
> [ 728.025861] R10: 0000000000030340 R11: ffff95dffffd5000 R12:
> ffffac02cdaebe08
> [ 728.033839] R13: ffff95df4b754000 R14: ffffac02cdaebe08 R15:
> 0000000000000000
> [ 728.041817] FS: 00007f7492b0ca80(0000) GS:ffff95df7f200000(0000)
> knlGS:0000000000000000
> [ 728.050862] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 728.057286] CR2: 00007f746532e004 CR3: 0000001fdc20e005 CR4:
> 00000000001606f0
> [ 728.065264] Call Trace:
> [ 728.068068] xfs_submit_ioend+0x84/0x1e0 [xfs]
> [ 728.073069] xfs_do_writepage+0x246/0x5a0 [xfs]
> [ 728.078139] write_cache_pages+0x16b/0x440
> [ 728.082741] ? xfs_vm_releasepage+0x80/0x80 [xfs]
> [ 728.088025] xfs_vm_writepages+0x6b/0xa0 [xfs]
> [ 728.092994] do_writepages+0x4b/0xf0
> [ 728.096995] ? iomap_file_buffered_write+0x62/0x90
> [ 728.102352] ? iomap_write_end+0x240/0x240
> [ 728.106958] ? xfs_iunlock+0x165/0x220 [xfs]
> [ 728.111734] ? __filemap_fdatawrite_range+0xc1/0x100
> [ 728.117276] __filemap_fdatawrite_range+0xc1/0x100
> [ 728.122625] file_write_and_wait_range+0x4c/0xa0
> [ 728.127812] xfs_file_fsync+0x5d/0x230 [xfs]
> [ 728.132591] do_fsync+0x38/0x60
> [ 728.136105] __x64_sys_fsync+0x10/0x20
> [ 728.140296] do_syscall_64+0x5b/0x1e0
> [ 728.144390] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [ 728.150039] RIP: 0033:0x7f747e8fe70d
> [ 728.154036] Code: bc 20 00 00 75 10 b8 4a 00 00 00 0f 05 48 3d 01 f0 ff
> ff 73 31 c3 48 83 ec 08 e8 6e f7 ff ff 48 89 04 24 b8 4a 00 00 00 0f 05 <48>
> 8b 3c 24 48 89 c2 e8 b7 f7 ff ff 48 89 d0 48 83 c4 08 48 3d 01
> [ 728.175027] RSP: 002b:00007ffe67fd0f00 EFLAGS: 00000293 ORIG_RAX:
> 000000000000004a
> [ 728.183491] RAX: ffffffffffffffda RBX: 000055d51d05b0f0 RCX:
> 00007f747e8fe70d
> [ 728.191469] RDX: 0000000000200000 RSI: 000055d51d05b0f0 RDI:
> 0000000000000003
> [ 728.199446] RBP: 00007f7465e2e418 R08: 0000000000000008 R09:
> 0000000000000001
> [ 728.207961] R10: 00000000897b0000 R11: 0000000000000293 R12:
> 0000000000000000
> [ 728.216467] R13: 0000001000000000 R14: 00007f747ac8ca50 R15:
> 0000000000000000
> [ 728.224971] ---[ end trace 929c66c4a933d3e3 ]---
> [ 728.230705] bio: c000/801 240 514831016 0
> [ 728.235725] 0: pfn 66605336 offset 0 len 18096128
> [ 728.241493] 2: pfn 66609754 offset 0 len 92037120
> [ 728.247234] 4: pfn 66632224 offset 0 len 151552
> [ 728.252776] 6: pfn 66632261 offset 0 len 94208
> [ 728.258203] 8: pfn 66632284 offset 0 len 100556800
> [ 728.264005] 10: pfn 66656834 offset 0 len 413696
> [ 728.269591] 12: pfn 66656935 offset 0 len 643072
> [ 728.275192] 14: pfn 66657092 offset 0 len 1982464
> [ 728.280848] 16: pfn 66657576 offset 0 len 606208
> [ 728.286416] 18: pfn 66657724 offset 0 len 126976
> [ 728.291948] 20: pfn 66657755 offset 0 len 249856
> [ 728.297470] 22: pfn 66657816 offset 0 len 626688
> [ 728.302989] 24: pfn 66657969 offset 0 len 499712
> [ 728.308505] 26: pfn 66658091 offset 0 len 610304
> [ 728.314014] 28: pfn 66658240 offset 0 len 876544
> [ 728.319516] 30: pfn 66658454 offset 0 len 495616
> [ 728.325042] 32: pfn 66658575 offset 0 len 1372160
> [ 728.330626] 34: pfn 66658910 offset 0 len 122880
> [ 728.336134] 36: pfn 66658940 offset 0 len 118784
> [ 728.341604] 38: pfn 66658969 offset 0 len 122880
> [ 728.347078] 40: pfn 66658999 offset 0 len 118784
> [ 728.352541] 42: pfn 66659028 offset 0 len 872448
> [ 728.358003] 44: pfn 66659241 offset 0 len 610304
> [ 728.363470] 46: pfn 66659390 offset 0 len 245760
> [ 728.368935] 48: pfn 66659450 offset 0 len 385024
> [ 728.374386] 50: pfn 66659544 offset 0 len 192512
> [ 728.379846] 52: pfn 66659591 offset 0 len 339968
> [ 728.385314] 54: pfn 66659674 offset 0 len 294912
> [ 728.390790] 56: pfn 66659746 offset 0 len 98304
> [ 728.396162] 58: pfn 66659770 offset 0 len 389120
> [ 728.401592] 60: pfn 66659865 offset 0 len 872448
> [ 728.407021] 62: pfn 66660078 offset 0 len 544768
> [ 728.412458] 64: pfn 66660211 offset 0 len 704512
> [ 728.417881] 66: pfn 66660383 offset 0 len 94208
> [ 728.423232] 68: pfn 66660406 offset 0 len 94208
> [ 728.428552] 70: pfn 66660429 offset 0 len 294912
> [ 728.433964] 72: pfn 66660501 offset 0 len 180224
> [ 728.439371] 74: pfn 66660545 offset 0 len 196608
> [ 728.444777] 76: pfn 66660593 offset 0 len 290816
> [ 728.450181] 78: pfn 66660664 offset 0 len 167936
> [ 728.455547] 80: pfn 66660705 offset 0 len 81920
> [ 728.460808] 82: pfn 66660725 offset 0 len 999424
> [ 728.466189] 84: pfn 66660969 offset 0 len 442368
> [ 728.471539] 86: pfn 66661077 offset 0 len 425984
> [ 728.476887] 88: pfn 66661181 offset 0 len 348160
> [ 728.482242] 90: pfn 66661266 offset 0 len 966656
> [ 728.487577] 92: pfn 66661502 offset 0 len 176128
> [ 728.492948] 94: pfn 66661545 offset 0 len 348160
> [ 728.498287] 96: pfn 66661630 offset 0 len 1064960
> [ 728.503770] 98: pfn 66661890 offset 0 len 180224
> [ 728.509136] 100: pfn 66661934 offset 0 len 888832
> [ 728.514574] 102: pfn 66662151 offset 0 len 176128
> [ 728.520045] 104: pfn 66662194 offset 0 len 81920
> [ 728.525381] 106: pfn 66662214 offset 0 len 356352
> [ 728.530829] 108: pfn 66662301 offset 0 len 89661440
> [ 728.536458] 110: pfn 66684191 offset 0 len 405504
> [ 728.541883] 112: pfn 66684290 offset 0 len 860160
> [ 728.547317] 114: pfn 66684500 offset 0 len 200704
> [ 728.552742] 116: pfn 66684549 offset 0 len 299008
> [ 728.558194] 118: pfn 66684622 offset 0 len 192512
> [ 728.563609] 120: pfn 66684669 offset 0 len 102400
> [ 728.569033] 122: pfn 66684694 offset 0 len 196608
> [ 728.574473] 124: pfn 66684742 offset 0 len 98304
> [ 728.579812] 126: pfn 66684766 offset 0 len 94208
> [ 728.585165] 128: pfn 66684789 offset 0 len 749568
> [ 728.590597] 130: pfn 66684972 offset 0 len 1490944
> [ 728.596119] 132: pfn 66685336 offset 0 len 684032
> [ 728.601542] 134: pfn 66685503 offset 0 len 495616
> [ 728.606964] 136: pfn 66685624 offset 0 len 196608
> [ 728.612391] 138: pfn 66685672 offset 0 len 585728
> [ 728.617814] 140: pfn 66685815 offset 0 len 94208
> [ 728.623168] 142: pfn 66685838 offset 0 len 94208
> [ 728.628505] 144: pfn 66685861 offset 0 len 77824
> [ 728.633830] 146: pfn 66685880 offset 0 len 778240
> [ 728.639286] 148: pfn 66686070 offset 0 len 196608
> [ 728.644710] 150: pfn 66686118 offset 0 len 503808
> [ 728.650165] 152: pfn 66686241 offset 0 len 85626880
> [ 728.655805] 154: pfn 66707146 offset 0 len 13893632
> [ 728.661430] 156: pfn 66710538 offset 0 len 3411968
> [ 728.666977] 158: pfn 66711371 offset 0 len 151552
> [ 728.672412] 160: pfn 66711408 offset 0 len 450560
> [ 728.677849] 162: pfn 66711518 offset 0 len 155648
> [ 728.683299] 164: pfn 66711556 offset 0 len 323584
> [ 728.688722] 166: pfn 66711635 offset 0 len 163840
> [ 728.694174] 168: pfn 66711675 offset 0 len 327680
> [ 728.699607] 170: pfn 66711755 offset 0 len 159744
> [ 728.705032] 172: pfn 66711794 offset 0 len 1130496
> [ 728.710544] 174: pfn 66712070 offset 0 len 974848
> [ 728.715968] 176: pfn 66712308 offset 0 len 651264
> [ 728.721384] 178: pfn 66712467 offset 0 len 2285568
> [ 728.726906] 180: pfn 66713025 offset 0 len 57344
> [ 728.732236] 182: pfn 66713039 offset 0 len 868352
> [ 728.737663] 184: pfn 66713251 offset 0 len 491520
> [ 728.743100] 186: pfn 66713371 offset 0 len 389120
> [ 728.748535] 188: pfn 66713466 offset 0 len 753664
> [ 728.753982] 190: pfn 66713650 offset 0 len 475136
> [ 728.759404] 192: pfn 66713766 offset 0 len 192512
> [ 728.764839] 194: pfn 66713813 offset 0 len 192512
> [ 728.770285] 196: pfn 66713860 offset 0 len 196608
> [ 728.775704] 198: pfn 66713908 offset 0 len 98304
> [ 728.781035] 200: pfn 66713932 offset 0 len 2113536
> [ 728.786562] 202: pfn 66714448 offset 0 len 94208
> [ 728.791894] 204: pfn 66714471 offset 0 len 290816
> [ 728.797323] 206: pfn 66714542 offset 0 len 65536
> [ 728.802653] 208: pfn 66714558 offset 0 len 2453504
> [ 728.808195] 210: pfn 66715157 offset 0 len 790528
> [ 728.813612] 212: pfn 66715350 offset 0 len 1056768
> [ 728.819129] 214: pfn 66715608 offset 0 len 76349440
> [ 728.824778] 216: pfn 66734248 offset 0 len 63995904
> [ 728.830407] 218: pfn 66749872 offset 0 len 52539392
> [ 728.836033] 220: pfn 66762699 offset 0 len 33771520
> [ 728.841649] 222: pfn 66770944 offset 0 len 5955584
> [ 728.847178] 224: pfn 66772398 offset 0 len 41803776
> [ 728.852797] 226: pfn 66782604 offset 0 len 16384
> [ 728.858151] 228: pfn 66782608 offset 0 len 8437760
> [ 728.863669] 230: pfn 66784668 offset 0 len 17031168
> [ 728.869319] 232: pfn 66788826 offset 0 len 22708224
> [ 728.874946] 234: pfn 66794370 offset 0 len 29933568
> [ 728.880563] 236: pfn 66801678 offset 0 len 4136960
> [ 728.886086] 238: pfn 66820096 offset 0 len 2506752
> [ 729.336992] WARNING: CPU: 40 PID: 1312 at drivers/scsi/scsi_lib.c:1010
> scsi_init_io+0x103/0x170
> [ 729.349150] Modules linked in: xfs dm_mod btrfs xor zstd_decompress
> zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac
> x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
> crct10dif_pclmul mgag200 crc32_pclmul ttm crc32c_intel ghash_clmulni_intel
> drm_kms_helper snd_pcm syscopyarea sysfillrect sysimgblt aesni_intel
> snd_timer ahci crypto_simd ipmi_si fb_sys_fops mei_me libahci snd
> ipmi_devintf cryptd soundcore glue_helper ipmi_msghandler pcspkr drm ioatdma
> mei libata joydev lpc_ich i2c_i801 dca wmi acpi_pad acpi_power_meter
> pcc_cpufreq ip_tables
> [ 729.409451] CPU: 40 PID: 1312 Comm: kworker/40:1H Tainted: G W
> I 5.2.0-rc5-00001-g82f8097 #1
> [ 729.421013] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
> SE5C610.86B.01.01.0008.021120151325 02/11/2015
> [ 729.433604] Workqueue: kblockd blk_mq_run_work_fn
> [ 729.439805] RIP: 0010:scsi_init_io+0x103/0x170
> [ 729.445726] Code: 5c 41 5d c3 be 01 00 00 00 e9 49 ff ff ff 8b 45 70 89
> 83 08 01 00 00 f6 45 1a 01 75 92 31 c0 48 83 c4 08 5b 5d 41 5c 41 5d c3 <0f>
> 0b 48 83 c4 08 b8 0a 00 00 00 5b 5d 41 5c 41 5d c3 48 8b 75 38
> [ 729.468787] RSP: 0018:ffffac02cea47c78 EFLAGS: 00010246
> [ 729.475705] RAX: 000000001eafb2a8 RBX: ffff95df3e7a3950 RCX:
> 0000000000000000
> [ 729.484790] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
> ffff95df3e7a3950
> [ 729.493889] RBP: ffff95df3e7a3840 R08: 0000000000000000 R09:
> ffff95df3e7a3988
> [ 729.502995] R10: ffffac02cc71fda0 R11: fefefefefefefeff R12:
> 0000000000000801
> [ 729.512123] R13: 0000000000000000 R14: 0000000000000000 R15:
> ffff95df3ba8e800
> [ 729.521261] FS: 0000000000000000(0000) GS:ffff95df7f780000(0000)
> knlGS:0000000000000000
> [ 729.531505] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 729.539140] CR2: 00007f7465922814 CR3: 0000000a80c0a006 CR4:
> 00000000001606e0
> [ 729.548359] Call Trace:
> [ 729.552355] sd_init_command+0x2ee/0xaf0 [sd_mod]
> [ 729.558882] scsi_queue_rq+0x559/0xa80
> [ 729.564354] blk_mq_dispatch_rq_list+0x3da/0x5b0
> [ 729.570820] ? syscall_return_via_sysret+0x10/0x7f
> [ 729.577488] ? __switch_to_asm+0x40/0x70
> [ 729.583192] ? elv_rb_del+0x1f/0x30
> [ 729.588417] ? deadline_remove_request+0x55/0xc0
> [ 729.594915] blk_mq_do_dispatch_sched+0x76/0x120
> [ 729.601424] blk_mq_sched_dispatch_requests+0x100/0x170
> [ 729.608626] __blk_mq_run_hw_queue+0x60/0x130
> [ 729.614873] process_one_work+0x19c/0x3c0
> [ 729.620734] worker_thread+0x3c/0x3b0
> [ 729.626210] ? process_one_work+0x3c0/0x3c0
> [ 729.632280] kthread+0x11e/0x140
> [ 729.637281] ? kthread_park+0x90/0x90
> [ 729.642773] ret_from_fork+0x35/0x40
> [ 729.648172] ---[ end trace 929c66c4a933d3e4 ]---
> [ 729.654870] print_req_error: I/O error, dev sdb, sector 514831016 flags
> 801
> [ 729.849057] XFS (sdb1): writeback error on sector 514831016
> [ 738.467814] WARNING: CPU: 1 PID: 1460 at block/blk-core.c:1128
> submit_bio+0xa5/0x1d0
> [ 738.478959] Modules linked in: xfs dm_mod btrfs xor zstd_decompress
> zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac
> x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
> crct10dif_pclmul mgag200 crc32_pclmul ttm crc32c_intel ghash_clmulni_intel
> drm_kms_helper snd_pcm syscopyarea sysfillrect sysimgblt aesni_intel
> snd_timer ahci crypto_simd ipmi_si fb_sys_fops mei_me libahci snd
> ipmi_devintf cryptd soundcore glue_helper ipmi_msghandler pcspkr drm ioatdma
> mei libata joydev lpc_ich i2c_i801 dca wmi acpi_pad acpi_power_meter
> pcc_cpufreq ip_tables
> [ 738.538361] CPU: 1 PID: 1460 Comm: fio Tainted: G W I
> 5.2.0-rc5-00001-g82f8097 #1
> [ 738.548561] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS
> SE5C610.86B.01.01.0008.021120151325 02/11/2015
> [ 738.560805] RIP: 0010:submit_bio+0xa5/0x1d0
> [ 738.566050] Code: 00 00 65 48 01 35 b3 39 bf 6f 8b 0d a1 96 c4 01 85 c9
> 0f 85 9b 00 00 00 8b 43 28 85 c0 75 a5 8b 53 10 0f b6 ca 83 f9 01 75 9a <0f>
> 0b 0f b7 4b 68 0f b7 73 14 45 31 c9 4c 8b 43 20 48 c7 c7 f1 99
> [ 738.588175] RSP: 0018:ffffac02cdaebb38 EFLAGS: 00010246
> [ 738.594572] RAX: 0000000000000000 RBX: ffff95df3ce59e40 RCX:
> 0000000000000001
> [ 738.603102] RDX: 0000000000000801 RSI: ffff95df3ce59e00 RDI:
> ffff95df3ce59e40
> [ 738.611611] RBP: 0000000000000000 R08: 0000000000000002 R09:
> 0000000000000000
> [ 738.620118] R10: 0000000000030340 R11: ffff95dffffd5000 R12:
> ffffac02cdaebe08
> [ 738.628626] R13: ffff95df3ce59e00 R14: ffffac02cdaebe08 R15:
> 0000000000000000
> [ 738.637125] FS: 00007f7492b0ca80(0000) GS:ffff95df7f240000(0000)
> knlGS:0000000000000000
> [ 738.646689] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 738.653626] CR2: 00007f7465355004 CR3: 0000001fdc20e004 CR4:
> 00000000001606e0
> [ 738.662117] Call Trace:
> [ 738.665420] xfs_submit_ioend+0x84/0x1e0 [xfs]
> [ 738.670931] xfs_do_writepage+0x246/0x5a0 [xfs]
> [ 738.676514] write_cache_pages+0x16b/0x440
> [ 738.681638] ? xfs_vm_releasepage+0x80/0x80 [xfs]
> [ 738.687425] xfs_vm_writepages+0x6b/0xa0 [xfs]
> [ 738.692895] do_writepages+0x4b/0xf0
> [ 738.697392] ? iomap_file_buffered_write+0x62/0x90
> [ 738.703240] ? iomap_write_end+0x240/0x240
> [ 738.708335] ? xfs_iunlock+0x165/0x220 [xfs]
> [ 738.713600] ? __filemap_fdatawrite_range+0xc1/0x100
> [ 738.719641] __filemap_fdatawrite_range+0xc1/0x100
> [ 738.725489] file_write_and_wait_range+0x4c/0xa0
> [ 738.731165] xfs_file_fsync+0x5d/0x230 [xfs]
> [ 738.736435] do_fsync+0x38/0x60
> [ 738.740439] __x64_sys_fsync+0x10/0x20
> [ 738.745124] do_syscall_64+0x5b/0x1e0
> [ 738.749711] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [ 738.755848] RIP: 0033:0x7f747e8fe70d
> [ 738.760333] Code: bc 20 00 00 75 10 b8 4a 00 00 00 0f 05 48 3d 01 f0 ff
> ff 73 31 c3 48 83 ec 08 e8 6e f7 ff ff 48 89 04 24 b8 4a 00 00 00 0f 05 <48>
> 8b 3c 24 48 89 c2 e8 b7 f7 ff ff 48 89 d0 48 83 c4 08 48 3d 01
> [ 738.782345] RSP: 002b:00007ffe67fd0f00 EFLAGS: 00000293 ORIG_RAX:
> 000000000000004a
> [ 738.791335] RAX: ffffffffffffffda RBX: 000055d51d05b0f0 RCX:
> 00007f747e8fe70d
> [ 738.799843] RDX: 0000000000200000 RSI: 000055d51d05b0f0 RDI:
> 0000000000000003
> [ 738.808354] RBP: 00007f7465e2e418 R08: 0000000000000008 R09:
> 0000000000000001
> [ 738.816865] R10: 00000000897b0000 R11: 0000000000000293 R12:
> 0000000000000000
> [ 738.825368] R13: 0000001000000000 R14: 00007f747ac8ca50 R15:
> 0000000000000000
> [ 738.833858] ---[ end trace 929c66c4a933d3e5 ]---
> [ 738.839553] bio: c000/801 98 532946720 0
> [ 738.844459] 0: pfn 32001772 offset 0 len 184487936
> [ 738.850312] 2: pfn 61947904 offset 0 len 4194304
> [ 738.855993] 4: pfn 61945856 offset 0 len 352256
> [ 738.861521] 6: pfn 61945942 offset 0 len 3842048
> [ 738.867130] 8: pfn 61943808 offset 0 len 4194304
> [ 738.872741] 10: pfn 61941760 offset 0 len 4194304
> [ 738.878426] 12: pfn 61939712 offset 0 len 3362816
> [ 738.884095] 14: pfn 61940533 offset 0 len 831488
> [ 738.889666] 16: pfn 61937664 offset 0 len 4194304
> [ 738.895333] 18: pfn 32165430 offset 0 len 243589120
> [ 738.901159] 20: pfn 61935616 offset 0 len 4194304
> [ 738.906784] 22: pfn 61933568 offset 0 len 4194304
> [ 738.912414] 24: pfn 61931520 offset 0 len 4194304
> [ 738.918029] 26: pfn 61929472 offset 0 len 4194304
> [ 738.923661] 28: pfn 32224900 offset 0 len 242958336
> [ 738.929470] 30: pfn 61927424 offset 0 len 4194304
> [ 738.935063] 32: pfn 32284216 offset 0 len 243081216
> [ 738.940858] 34: pfn 61925376 offset 0 len 4194304
> [ 738.946466] 36: pfn 32343562 offset 0 len 243232768
> [ 738.952242] 38: pfn 32402945 offset 0 len 16384
> [ 738.957612] 40: pfn 61923328 offset 0 len 4194304
> [ 738.963182] 42: pfn 61921280 offset 0 len 4194304
> [ 738.968774] 44: pfn 61919232 offset 0 len 3047424
> [ 738.974328] 46: pfn 61919976 offset 0 len 1146880
> [ 738.979888] 48: pfn 61917184 offset 0 len 4194304
> [ 738.985456] 50: pfn 61915136 offset 0 len 4194304
> [ 738.991017] 52: pfn 32462295 offset 0 len 243081216
> [ 738.996775] 54: pfn 61913088 offset 0 len 4194304
> [ 739.002323] 56: pfn 61911040 offset 0 len 3039232
> [ 739.007892] 58: pfn 61911782 offset 0 len 1155072
> [ 739.013445] 60: pfn 61908992 offset 0 len 1875968
> [ 739.018982] 62: pfn 61909450 offset 0 len 20480
> [ 739.024332] 64: pfn 61909455 offset 0 len 2297856
> [ 739.029865] 66: pfn 61906944 offset 0 len 4194304
> [ 739.035420] 68: pfn 61904896 offset 0 len 4194304
> [ 739.040947] 70: pfn 32640459 offset 0 len 243032064
> [ 739.046677] 72: pfn 61902848 offset 0 len 4194304
> [ 739.052232] 74: pfn 32699793 offset 0 len 243228672
> [ 739.057977] 76: pfn 61900800 offset 0 len 4194304
> [ 739.063495] 78: pfn 61898752 offset 0 len 4194304
> [ 739.069023] 80: pfn 61896704 offset 0 len 4194304
> [ 739.074520] 82: pfn 32759175 offset 0 len 242909184
> [ 739.080230] 84: pfn 61894656 offset 0 len 4194304
> [ 739.085743] 86: pfn 32818479 offset 0 len 243081216
> [ 739.091446] 88: pfn 61892608 offset 0 len 4194304
> [ 739.096936] 90: pfn 61890560 offset 0 len 1286144
> [ 739.102418] 92: pfn 61890874 offset 0 len 2908160
> [ 739.107895] 94: pfn 61888512 offset 0 len 4194304
> [ 739.113390] 96: pfn 61886464 offset 0 len 2572288
> [ 739.118938] print_req_error: I/O error, dev sdb, sector 532946720 flags
> 801
> [ 739.430464] XFS (sdb1): writeback error on sector 532946720
> [ 742.762917] fio: pid=0, err=5/file:filesetup.c:190, func=fsync,
> error=Input/output error
> [ 742.762920]
Sorry for missing your dmesg log.
Now, the issue is quite clear, and it is one int overflow issue,
which should be addressed by the patch I posted.
Please go ahead and give a test on V2!
thanks,
Ming
563
days inactive
570
days old
19 comments
3 participants
participants (3)
-
kernel test robot -
Liu Yiding -
Ming Lei