5:12 p.m.
Greeting,
FYI, we noticed the following commit (built with gcc-9):
commit: 2cb09c90a4cde464dbfe78a57a9c128625c31122 ("irqdomain: Introduce
irq_resolve_mapping()")
https://git.kernel.org/cgit/linux/kernel/git/maz/arm-platforms.git
irq/generic_handle_irq_domain
in testcase: suspend-stress
version:
with following parameters:
mode: mem
iterations: 10
on test machine: 4 threads Ivy Bridge with 4G memory
caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang(a)intel.com>
test started
<--- but cannot finish. in attached dmesg.log
[ 56.989808] SUSPEND RESUME TEST STARTED
[ 56.994723] Suspend to mem 1/10:
[ 57.393914] /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8
http://internal-lkp-server:80/~lkp/cgi-bin/lkp-jobfile-append-var?job_fil...
-O /dev/null
[ 57.402918] Done
[ 57.408029] Sleep for 10 seconds
[ 57.413073] Suspend to mem 2/10:
[ 57.417905] network not ready
[ 57.422647] network not ready
[ 57.427328] network not ready
[ 57.431923] network not ready
[ 57.436467] network not ready
[ 57.440953] network not ready
[ 57.445380] network not ready
[ 57.449680] network not ready
[ 57.453960] network not ready
[ 58.066532] network not ready
[ 59.069490] network not ready
[ 60.072399] network not ready
[ 61.075285] network not ready
[ 62.078110] network not ready
[ 62.899660] e1000e 0000:00:19.0 eth0: NIC Link is Up 100 Mbps Full Duplex, Flow
Control: None
To reproduce:
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
bin/lkp run generated-yaml-file
---
0DAY/LKP+ Test Infrastructure Open Source Technology Center
https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation
Thanks,
Oliver Sang
5:55 p.m.
Hi Oliver,
On Wed, 02 Jun 2021 08:12:31 +0100,
kernel test robot <oliver.sang(a)intel.com> wrote:
Greeting,
FYI, we noticed the following commit (built with gcc-9):
commit: 2cb09c90a4cde464dbfe78a57a9c128625c31122 ("irqdomain: Introduce
irq_resolve_mapping()")
https://git.kernel.org/cgit/linux/kernel/git/maz/arm-platforms.git
irq/generic_handle_irq_domain
in testcase: suspend-stress
version:
with following parameters:
mode: mem
iterations: 10
on test machine: 4 threads Ivy Bridge with 4G memory
caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang(a)intel.com>
test started
<--- but cannot finish. in attached dmesg.log
[ 56.989808] SUSPEND RESUME TEST STARTED
[ 56.994723] Suspend to mem 1/10:
[ 57.393914] /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8
http://internal-lkp-server:80/~lkp/cgi-bin/lkp-jobfile-append-var?job_fil...
-O /dev/null
[ 57.402918] Done
[ 57.408029] Sleep for 10 seconds
[ 57.413073] Suspend to mem 2/10:
[ 57.417905] network not ready
[ 57.422647] network not ready
[ 57.427328] network not ready
[ 57.431923] network not ready
[ 57.436467] network not ready
[ 57.440953] network not ready
[ 57.445380] network not ready
[ 57.449680] network not ready
[ 57.453960] network not ready
[ 58.066532] network not ready
[ 59.069490] network not ready
[ 60.072399] network not ready
[ 61.075285] network not ready
[ 62.078110] network not ready
[ 62.899660] e1000e 0000:00:19.0 eth0: NIC Link is Up 100 Mbps Full Duplex, Flow
Control: None
Can you give a few more details about the failure mode? Did you expect
the network interface to show up earlier? How repeatable is this
issue? Does it show up on any other platform?
It may take me some time to find similar HW, so any help pinpointing
the issue is most welcome.
Thanks,
M.
--
Without deviation from the norm, progress is not possible.
6:25 p.m.
On Wed, 2021-06-02 at 08:55 +0100, Marc Zyngier wrote:
Hi Oliver,
On Wed, 02 Jun 2021 08:12:31 +0100,
kernel test robot <oliver.sang(a)intel.com> wrote:
>
> Greeting,
>
> FYI, we noticed the following commit (built with gcc-9):
>
> commit: 2cb09c90a4cde464dbfe78a57a9c128625c31122 ("irqdomain:
> Introduce irq_resolve_mapping()")
> https://git.kernel.org/cgit/linux/kernel/git/maz/arm-platforms.git
> irq/generic_handle_irq_domain
>
>
> in testcase: suspend-stress
> version:
> with following parameters:
>
> mode: mem
> iterations: 10
>
>
>
> on test machine: 4 threads Ivy Bridge with 4G memory
>
> caused below changes (please refer to attached dmesg/kmsg for
> entire log/backtrace):
>
>
>
>
> If you fix the issue, kindly add following tag
> Reported-by: kernel test robot <oliver.sang(a)intel.com>
>
> test started
> <--- but cannot finish. in attached dmesg.log
> [ 56.989808] SUSPEND RESUME TEST STARTED
>
> [ 56.994723] Suspend to mem 1/10:
>
> [ 57.393914] /usr/bin/wget -q --timeout=1800 --tries=1 --local-
> encoding=UTF-8
>
http://internal-lkp-server:80/~lkp/cgi-bin/lkp-jobfile-append-var?job_fil...
> -O /dev/null
>
> [ 57.402918] Done
>
> [ 57.408029] Sleep for 10 seconds
>
> [ 57.413073] Suspend to mem 2/10:
>
> [ 57.417905] network not ready
>
> [ 57.422647] network not ready
>
> [ 57.427328] network not ready
>
> [ 57.431923] network not ready
>
> [ 57.436467] network not ready
>
> [ 57.440953] network not ready
>
> [ 57.445380] network not ready
>
> [ 57.449680] network not ready
>
> [ 57.453960] network not ready
>
> [ 58.066532] network not ready
>
> [ 59.069490] network not ready
>
> [ 60.072399] network not ready
>
> [ 61.075285] network not ready
>
> [ 62.078110] network not ready
>
> [ 62.899660] e1000e 0000:00:19.0 eth0: NIC Link is Up 100 Mbps
> Full Duplex, Flow Control: None
Can you give a few more details about the failure mode? Did you
expect
the network interface to show up earlier?
TBH, we don't know if this is caused by the network failure or not.
It is a remote machine that I need to take some time to verify this.
How repeatable is this
issue?
It seems to be intermittent but relatively easy to reproduce.
Here is what I did.
1. run 10 suspend iterations after boot
2. reboot and repeat step 1 by 10 times.
The test is done on three different machines, an old Ivybridge laptop,
a skylake laptop and a kabylake laptop.
without the commit, all three machines are still running well after 10
10 suspend iterations, in 10 different fresh boot.
with the commit, when doing 10 back-to-back suspend, the three machines
are alive only 1 or even 0 time, out of 10 reboot cycles.
So I think this is kind of a generic problem that may affect many
machines. You can pick some laptop and do 10 suspend iteration and
check if the system is still alive, if yes, do a reboot and check if it
still can pass the 10 suspend test.
thanks,
rui
Does it show up on any other platform?
It may take me some time to find similar HW, so any help pinpointing
the issue is most welcome.
Thanks,
M.
2:24 a.m.
Hi Rui,
On Wed, 02 Jun 2021 09:25:31 +0100,
Zhang Rui <rui.zhang(a)intel.com> wrote:
On Wed, 2021-06-02 at 08:55 +0100, Marc Zyngier wrote:
> Hi Oliver,
>
> On Wed, 02 Jun 2021 08:12:31 +0100,
> kernel test robot <oliver.sang(a)intel.com> wrote:
> >
> > Greeting,
> >
> > FYI, we noticed the following commit (built with gcc-9):
> >
> > commit: 2cb09c90a4cde464dbfe78a57a9c128625c31122 ("irqdomain:
> > Introduce irq_resolve_mapping()")
> > https://git.kernel.org/cgit/linux/kernel/git/maz/arm-platforms.git
> > irq/generic_handle_irq_domain
[...]
>
> Can you give a few more details about the failure mode? Did you
> expect
> the network interface to show up earlier?
TBH, we don't know if this is caused by the network failure or not.
It is a remote machine that I need to take some time to verify this.
> How repeatable is this
> issue?
It seems to be intermittent but relatively easy to reproduce.
Here is what I did.
1. run 10 suspend iterations after boot
2. reboot and repeat step 1 by 10 times.
The test is done on three different machines, an old Ivybridge laptop,
a skylake laptop and a kabylake laptop.
Let's just say that I very little idea of what these names represent,
but I do have an *old* laptop around (i7-2760QM), on which I was able
to reproduce something.
I have the strong feeling that something in the resume path is tearing
down a live mapping while the interrupt is being looked up, which
would be an interesting bug on its own. But it should be pretty cheap
to work around that. Could you try the hack below on top of the
offending commit? It did fix the resume issue for me:
<quote>
maz@hina:~$ sudo dmesg| grep "ACPI: Low-level resume complete"| wc -l
20
</quote>
Thanks,
M.
diff --git a/include/linux/irqdomain.h b/include/linux/irqdomain.h
index 189f559fb26c..5b7de0647ea7 100644
--- a/include/linux/irqdomain.h
+++ b/include/linux/irqdomain.h
@@ -402,8 +402,15 @@ static inline unsigned int irq_create_mapping(struct irq_domain
*host,
return irq_create_mapping_affinity(host, hwirq, NULL);
}
-extern struct irq_desc *irq_resolve_mapping(struct irq_domain *domain,
- irq_hw_number_t hwirq);
+extern struct irq_desc *__irq_resolve_mapping(struct irq_domain *domain,
+ irq_hw_number_t hwirq,
+ unsigned int *irq);
+
+static inline struct irq_desc *irq_resolve_mapping(struct irq_domain *domain,
+ irq_hw_number_t hwirq)
+{
+ return __irq_resolve_mapping(domain, hwirq, NULL);
+}
/**
* irq_find_mapping() - Find a linux irq from a hw irq number.
@@ -413,12 +420,12 @@ extern struct irq_desc *irq_resolve_mapping(struct irq_domain
*domain,
static inline unsigned int irq_find_mapping(struct irq_domain *domain,
irq_hw_number_t hwirq)
{
- struct irq_desc *desc = irq_resolve_mapping(domain, hwirq);
+ unsigned int irq;
- if (unlikely(!desc))
- return 0;
+ if (__irq_resolve_mapping(domain, hwirq, &irq))
+ return irq;
- return irq_desc_get_irq(desc);
+ return 0;
}
static inline unsigned int irq_linear_revmap(struct irq_domain *domain,
diff --git a/kernel/irq/irqdomain.c b/kernel/irq/irqdomain.c
index 13a3021e9370..012d6bf456aa 100644
--- a/kernel/irq/irqdomain.c
+++ b/kernel/irq/irqdomain.c
@@ -879,11 +879,13 @@ EXPORT_SYMBOL_GPL(irq_dispose_mapping);
* irq_resolve_mapping() - Find a linux irq from a hw irq number.
* @domain: domain owning this hardware interrupt
* @hwirq: hardware irq number in that domain space
+ * @irq: optional pointer to return the Linux irq if required
*
* Returns the interrupt descriptor.
*/
-struct irq_desc *irq_resolve_mapping(struct irq_domain *domain,
- irq_hw_number_t hwirq)
+struct irq_desc *__irq_resolve_mapping(struct irq_domain *domain,
+ irq_hw_number_t hwirq,
+ unsigned int *irq)
{
struct irq_desc *desc = NULL;
struct irq_data *data;
@@ -911,12 +913,16 @@ struct irq_desc *irq_resolve_mapping(struct irq_domain *domain,
else
data = radix_tree_lookup(&domain->revmap_tree, hwirq);
- if (likely(data))
+ if (likely(data)) {
desc = irq_data_to_desc(data);
+ if (irq)
+ *irq = data->irq;
+ }
+
rcu_read_unlock();
return desc;
}
-EXPORT_SYMBOL_GPL(irq_resolve_mapping);
+EXPORT_SYMBOL_GPL(__irq_resolve_mapping);
/**
* irq_domain_xlate_onecell() - Generic xlate for direct one cell bindings
--
Without deviation from the norm, progress is not possible.
6:37 p.m.
hi Marc,
On Wed, Jun 02, 2021 at 05:24:04PM +0100, Marc Zyngier wrote:
Hi Rui,
On Wed, 02 Jun 2021 09:25:31 +0100,
Zhang Rui <rui.zhang(a)intel.com> wrote:
>
> On Wed, 2021-06-02 at 08:55 +0100, Marc Zyngier wrote:
> > Hi Oliver,
> >
> > On Wed, 02 Jun 2021 08:12:31 +0100,
> > kernel test robot <oliver.sang(a)intel.com> wrote:
> > >
> > > Greeting,
> > >
> > > FYI, we noticed the following commit (built with gcc-9):
> > >
> > > commit: 2cb09c90a4cde464dbfe78a57a9c128625c31122 ("irqdomain:
> > > Introduce irq_resolve_mapping()")
> > > https://git.kernel.org/cgit/linux/kernel/git/maz/arm-platforms.git
> > > irq/generic_handle_irq_domain
[...]
> >
> > Can you give a few more details about the failure mode? Did you
> > expect
> > the network interface to show up earlier?
>
> TBH, we don't know if this is caused by the network failure or not.
> It is a remote machine that I need to take some time to verify this.
>
> > How repeatable is this
> > issue?
>
> It seems to be intermittent but relatively easy to reproduce.
> Here is what I did.
> 1. run 10 suspend iterations after boot
> 2. reboot and repeat step 1 by 10 times.
>
> The test is done on three different machines, an old Ivybridge laptop,
> a skylake laptop and a kabylake laptop.
Let's just say that I very little idea of what these names represent,
but I do have an *old* laptop around (i7-2760QM), on which I was able
to reproduce something.
I have the strong feeling that something in the resume path is tearing
down a live mapping while the interrupt is being looked up, which
would be an interesting bug on its own. But it should be pretty cheap
to work around that. Could you try the hack below on top of the
offending commit? It did fix the resume issue for me:
we tested below patch on platform where we made out original report,
confirmed the failure was fixed.
we will try to test on other platforms and if there are any problem,
we will let you know. Thanks
> <quote>
> maz@hina:~$ sudo dmesg| grep "ACPI: Low-level resume complete"| wc -l
> 20
> </quote>
>
> Thanks,
>
> M.
>
> diff --git a/include/linux/irqdomain.h b/include/linux/irqdomain.h
> index 189f559fb26c..5b7de0647ea7 100644
> --- a/include/linux/irqdomain.h
> +++ b/include/linux/irqdomain.h
> @@ -402,8 +402,15 @@ static inline unsigned int irq_create_mapping(struct irq_domain
*host,
> return irq_create_mapping_affinity(host, hwirq, NULL);
> }
>
> -extern struct irq_desc *irq_resolve_mapping(struct irq_domain *domain,
> - irq_hw_number_t hwirq);
> +extern struct irq_desc *__irq_resolve_mapping(struct irq_domain *domain,
> + irq_hw_number_t hwirq,
> + unsigned int *irq);
> +
> +static inline struct irq_desc *irq_resolve_mapping(struct irq_domain *domain,
> + irq_hw_number_t hwirq)
> +{
> + return __irq_resolve_mapping(domain, hwirq, NULL);
> +}
>
> /**
> * irq_find_mapping() - Find a linux irq from a hw irq number.
> @@ -413,12 +420,12 @@ extern struct irq_desc *irq_resolve_mapping(struct irq_domain
*domain,
> static inline unsigned int irq_find_mapping(struct irq_domain *domain,
> irq_hw_number_t hwirq)
> {
> - struct irq_desc *desc = irq_resolve_mapping(domain, hwirq);
> + unsigned int irq;
>
> - if (unlikely(!desc))
> - return 0;
> + if (__irq_resolve_mapping(domain, hwirq, &irq))
> + return irq;
>
> - return irq_desc_get_irq(desc);
> + return 0;
> }
>
> static inline unsigned int irq_linear_revmap(struct irq_domain *domain,
> diff --git a/kernel/irq/irqdomain.c b/kernel/irq/irqdomain.c
> index 13a3021e9370..012d6bf456aa 100644
> --- a/kernel/irq/irqdomain.c
> +++ b/kernel/irq/irqdomain.c
> @@ -879,11 +879,13 @@ EXPORT_SYMBOL_GPL(irq_dispose_mapping);
> * irq_resolve_mapping() - Find a linux irq from a hw irq number.
> * @domain: domain owning this hardware interrupt
> * @hwirq: hardware irq number in that domain space
> + * @irq: optional pointer to return the Linux irq if required
> *
> * Returns the interrupt descriptor.
> */
> -struct irq_desc *irq_resolve_mapping(struct irq_domain *domain,
> - irq_hw_number_t hwirq)
> +struct irq_desc *__irq_resolve_mapping(struct irq_domain *domain,
> + irq_hw_number_t hwirq,
> + unsigned int *irq)
> {
> struct irq_desc *desc = NULL;
> struct irq_data *data;
> @@ -911,12 +913,16 @@ struct irq_desc *irq_resolve_mapping(struct irq_domain
*domain,
> else
> data = radix_tree_lookup(&domain->revmap_tree, hwirq);
>
> - if (likely(data))
> + if (likely(data)) {
> desc = irq_data_to_desc(data);
> + if (irq)
> + *irq = data->irq;
> + }
> +
> rcu_read_unlock();
> return desc;
> }
> -EXPORT_SYMBOL_GPL(irq_resolve_mapping);
> +EXPORT_SYMBOL_GPL(__irq_resolve_mapping);
>
> /**
> * irq_domain_xlate_onecell() - Generic xlate for direct one cell bindings
>
> --
> Without deviation from the norm, progress is not possible.
6:23 p.m.
Hi Oliver,
On Wed, 09 Jun 2021 09:37:08 +0100,
Oliver Sang <oliver.sang(a)intel.com> wrote:
hi Marc,
On Wed, Jun 02, 2021 at 05:24:04PM +0100, Marc Zyngier wrote:
> Hi Rui,
>
> On Wed, 02 Jun 2021 09:25:31 +0100,
> Zhang Rui <rui.zhang(a)intel.com> wrote:
> >
> > On Wed, 2021-06-02 at 08:55 +0100, Marc Zyngier wrote:
> > > Hi Oliver,
> > >
> > > On Wed, 02 Jun 2021 08:12:31 +0100,
> > > kernel test robot <oliver.sang(a)intel.com> wrote:
> > > >
> > > > Greeting,
> > > >
> > > > FYI, we noticed the following commit (built with gcc-9):
> > > >
> > > > commit: 2cb09c90a4cde464dbfe78a57a9c128625c31122 ("irqdomain:
> > > > Introduce irq_resolve_mapping()")
> > > > https://git.kernel.org/cgit/linux/kernel/git/maz/arm-platforms.git
> > > > irq/generic_handle_irq_domain
>
> [...]
>
> > >
> > > Can you give a few more details about the failure mode? Did you
> > > expect
> > > the network interface to show up earlier?
> >
> > TBH, we don't know if this is caused by the network failure or not.
> > It is a remote machine that I need to take some time to verify this.
> >
> > > How repeatable is this
> > > issue?
> >
> > It seems to be intermittent but relatively easy to reproduce.
> > Here is what I did.
> > 1. run 10 suspend iterations after boot
> > 2. reboot and repeat step 1 by 10 times.
> >
> > The test is done on three different machines, an old Ivybridge laptop,
> > a skylake laptop and a kabylake laptop.
>
> Let's just say that I very little idea of what these names represent,
> but I do have an *old* laptop around (i7-2760QM), on which I was able
> to reproduce something.
>
> I have the strong feeling that something in the resume path is tearing
> down a live mapping while the interrupt is being looked up, which
> would be an interesting bug on its own. But it should be pretty cheap
> to work around that. Could you try the hack below on top of the
> offending commit? It did fix the resume issue for me:
>
we tested below patch on platform where we made out original report,
confirmed the failure was fixed.
we will try to test on other platforms and if there are any problem,
we will let you know. Thanks
Great, thanks for letting me know. The fixed version is now in -next.
Should if you spot anything else.
M.
--
Without deviation from the norm, progress is not possible.
noon
Hi Marc,
-----Original Message-----
From: Marc Zyngier <maz(a)kernel.org>
Sent: Wednesday, June 9, 2021 4:23 PM
To: Sang, Oliver <oliver.sang(a)intel.com>
Cc: Zhang, Rui <rui.zhang(a)intel.com>; lkp(a)lists.01.org; lkp <lkp(a)intel.com>;
Chen, Yu C <yu.c.chen(a)intel.com>
Subject: [LKP] Re: [irqdomain] 2cb09c90a4: suspend-stress.fail
Hi Oliver,
On Wed, 09 Jun 2021 09:37:08 +0100,
Oliver Sang <oliver.sang(a)intel.com> wrote:
>
> hi Marc,
>
> On Wed, Jun 02, 2021 at 05:24:04PM +0100, Marc Zyngier wrote:
> > Hi Rui,
> >
> > On Wed, 02 Jun 2021 09:25:31 +0100,
> > Zhang Rui <rui.zhang(a)intel.com> wrote:
> > >
> > > On Wed, 2021-06-02 at 08:55 +0100, Marc Zyngier wrote:
> > > > Hi Oliver,
> > > >
> > > > On Wed, 02 Jun 2021 08:12:31 +0100, kernel test robot
> > > > <oliver.sang(a)intel.com> wrote:
> > > > >
> > > > > Greeting,
> > > > >
> > > > > FYI, we noticed the following commit (built with gcc-9):
> > > > >
> > > > > commit: 2cb09c90a4cde464dbfe78a57a9c128625c31122
("irqdomain:
> > > > > Introduce irq_resolve_mapping()")
> > > > > https://git.kernel.org/cgit/linux/kernel/git/maz/arm-platforms
> > > > > .git
> > > > > irq/generic_handle_irq_domain
> >
> > [...]
> >
> > > >
> > > > Can you give a few more details about the failure mode? Did you
> > > > expect the network interface to show up earlier?
> > >
> > > TBH, we don't know if this is caused by the network failure or not.
> > > It is a remote machine that I need to take some time to verify this.
> > >
> > > > How repeatable is this
> > > > issue?
> > >
> > > It seems to be intermittent but relatively easy to reproduce.
> > > Here is what I did.
> > > 1. run 10 suspend iterations after boot 2. reboot and repeat step
> > > 1 by 10 times.
> > >
> > > The test is done on three different machines, an old Ivybridge
> > > laptop, a skylake laptop and a kabylake laptop.
> >
> > Let's just say that I very little idea of what these names
> > represent, but I do have an *old* laptop around (i7-2760QM), on
> > which I was able to reproduce something.
> >
> > I have the strong feeling that something in the resume path is
> > tearing down a live mapping while the interrupt is being looked up,
> > which would be an interesting bug on its own. But it should be
> > pretty cheap to work around that. Could you try the hack below on
> > top of the offending commit? It did fix the resume issue for me:
> >
>
> we tested below patch on platform where we made out original report,
> confirmed the failure was fixed.
This test was on a Ivybridge laptop.
> we will try to test on other platforms and if there are any
problem,
> we will let you know. Thanks
Great, thanks for letting me know. The fixed version is now in -next.
Should if you spot anything else.
M.
We have also tested the patch on a Skylake laptop, and confirmed the failure was fixed on
this platform, too.
Thanks.
> --
> Without deviation from the norm, progress is not possible.
> _______________________________________________
> LKP mailing list -- lkp(a)lists.01.org
> To unsubscribe send an email to lkp-leave(a)lists.01.org
465
days inactive
473
days old
6 comments
5 participants
participants (5)
-
kernel test robot -
Liu, Yujie -
Marc Zyngier -
Oliver Sang
-
Zhang Rui