Ping... (I know that the recent news at Intel might be quite distracting for some you...)

On Fri, Apr 15, 2016 at 1:05 PM, Hart, Darren <darren.hart(a)intel.com&gt; wrote: > Hi Fengguang, > > I met with Kostya at Intel Tech Days and he had some compelling arguments > for including some of these tests. > > Dave H: I took a quick look at Kasan, which appears to require an existing > config option (CONFIG_KASAN) for dynamic memory access checking using > shadow memory. Is this something you would like to see added to 0-day? Do > we have anything today which provides comparable coverage? > > Combining Kasan and other existing kernel integrity checking, the > syzkaller fuzz tester is showing promising results and the reports come in > the form of kernel oops and similar things which we already check for in > 0-day. > > For the others on Cc, do you have additional context for or against > including syzkaller and kasan in 0day? > > Fengguang, what are your thoughts on including these in 0-day? > > Thanks, > > -- > Darren Hart > Intel Open Source Technology Center > > On 4/12/16, 10:24 PM, "Kostya Serebryany" <kcc(a)google.com&lt;mailto: > kcc(a)google.com&gt;&gt; wrote: > > CC-ing more people after today's conversation at the Intel Tech Days. > > We'd like to add kasan and syzkaller [1,2,3,4] to the 0-day kbuild system. > We believe this has a large potential to find old bugs and prevent > regressions in the Kernel. > How do we achieve this? > > Thanks, > > --kcc > > [1] https://github.com/google/syzkaller > [2] https://github.com/google/syzkaller/wiki/Found-Bugs > [3] https://lwn.net/Articles/677764/ > [4] https://www.kernel.org/doc/Documentation/kasan.txt > > > On Tue, Dec 15, 2015 at 3:49 AM, David Drysdale <drysdale(a)google.com > <mailto:drysdale@google.com>> wrote: > Hi Fengguang / LKP-folk, > > Quick question -- how easy is it to add extra builds/tests/checks to > your marvellous 0-day kbuild system? > > The reason I ask is that I've recently been exploring syzkaller [1], > which is a system call fuzzer written by some of my colleagues here at > Google (cc'ed). Although it's fairly new, it has uncovered a bunch of > kernel bugs already [2] so I wondered if it might be a good candidate > for inclusion in the 0-day checks at some point. > > (As an aside, I'm in the process of writing an article about syzkaller > for LWN, which might also expose it to more folk.) > > What do you think? > > Thanks, > David > > [1] https://github.com/google/syzkaller > [2] https://github.com/google/syzkaller/wiki/Found-Bugs > >

_______________________________________________ LKP mailing list LKP(a)lists.01.org https://lists.01.org/mailman/listinfo/lkp

On Thu, Apr 28, 2016 at 3:41 AM, Philip Li <philip.li(a)intel.com&gt; wrote: > On Wed, Apr 27, 2016 at 11:03:42AM -0700, Kostya Serebryany wrote: >> Ping... >> (I know that the recent news at Intel might be quite distracting for some >> you...) > hi all, this is Philip from 0day team, sorry for late reply. We are glad to add > syzkaller test suite to 0day, and we will do detail plan of it within this quarter > and share with by end of June including what the first step can be and any support > if needed. > > Is this time frame ok? because right now, all resources are booked, and > formal action may need start next quarter. Hi Philip, The time frame is OK > One quick question is i recall an early discussion is that this enabling requires new gcc, > is this still be true? Syzkaller requires compiler support. I use new gcc. However, there are currently patches in flight that add support for gcc plugins to kernel build system. As far as I understand, with the plugin support no messing with gcc is required (KCOV instrumentation plugin will also be in kernel tree).

> One quick question is i recall an early discussion is that this enabling requires new gcc, > is this still be true? I see Debian just upgraded its gcc packages to 6.1, which makes it more convenient for us to integrate with syzkaller. Since syzkaller is more about a client-server system than a hands on tool like trinity, it may need non-trivial modifications to 0-day and possibly syzkaller to achieve the integration. For example, syzkaller and 0-day each has its own way to run QEMU, which should be resolved somehow. Another question is, the 0-day model relies heavily on bisecting to cover the large number of kernel trees. If a bug cannot be easily reproduced and bisected, we don't know to whom it should be reported to. Trinity offers an option "-sN: use N as random seed." to help reproduce bugs it found. I wonder if syzkaller has similar mechanism to help with reproducing bugs it found?

Hi Dmitry, On Tue, May 03, 2016 at 10:00:04AM +0200, Dmitry Vyukov wrote: > On Tue, May 3, 2016 at 6:01 AM, Fengguang Wu <fengguang.wu(a)intel.com&gt; wrote: > >> One quick question is i recall an early discussion is that this enabling requires new gcc, > >> is this still be true? > > > > I see Debian just upgraded its gcc packages to 6.1, which makes it > > more convenient for us to integrate with syzkaller. > > > > Since syzkaller is more about a client-server system than a hands on > > tool like trinity, it may need non-trivial modifications to 0-day and > > possibly syzkaller to achieve the integration. For example, syzkaller > > and 0-day each has its own way to run QEMU, which should be resolved > > somehow. > > > > Another question is, the 0-day model relies heavily on bisecting to > > cover the large number of kernel trees. If a bug cannot be easily > > reproduced and bisected, we don't know to whom it should be reported > > to. Trinity offers an option "-sN: use N as random seed." to help > > reproduce bugs it found. I wonder if syzkaller has similar mechanism > > to help with reproducing bugs it found? > > > Hi Fengguang, > > Yes, syzkaller setup is more elaborate, but architecture is pretty > modular so I believe we can resolve these issues. Yes syzkaller looks very interesting. I'll try it out in more details and try enabling it in 0day. > What does your infrastructure do? I guess it winds up qemu instances, > copies necessary binaries and then runs some command? How does it > monitor for crashes? By parsing console output? How do you control > duration of testing? By instructing trinity to do fixed number of > iterations, or by killing qemu? Basically each 0day boot server runs a number of QEMU instances with various locally cached QEMU images. Trinity binaries are built into these images. A boot script in the images will run trinity in background, sleep for some time and then kill trinity. If tests run smoothly, the scripts inside the QEMU image will be able to shutdown VM in time. Otherwise if things get stuck inside VM, the host side script which started the QEMU VM and monitors its serial output may decide to kill the QEMU process when either timeout or find kernel panic patterns.

> Syzkaller already has a number of simpler tools. For example, > syz-stress works more or less like trinity -- you can copy it onto > test machine and instruct to run random programs. I can cook another > tool that will fit into your setup. OK, we may sort details out in private emails. Preferably the resulted system should get the most out of what syzkaller offers and excels. > One inherent complication, though, > is that syzkaller is stateful, it needs to accumulate corpus of > interesting programs (based on code coverage) to work more > efficiently. So we will need some way to save corpus from qemu > instances externally and then pass the corpus to new qemu instances. > But on the other hand, that was proven to be super efficient in > finding regressions -- namely, in 15 minutes it can test almost > everything that it discovered previously. That looks very impressive! > Re reproducers: syzkaller can do even better than trinity. Syzkaller > logs all executed programs to stdout (in default mode, or it can also > log to local files or kernel console). If you have the log with > programs that were executed before the crash, then it's possible (1) > to re-execute the log on other kernels, or (2) extract the guilty > program from the log, minimize it and convert to an executable C > program to present to developers > (https://github.com/google/syzkaller/wiki/Crash-reproducer-programs). > Based on my experience (2) is real game changer, developers much more > willing to fix bugs when they have a small reproducer. Cannot agree any more -- it looks the right way to go. Hope we can make it run in 0day soon! Thanks, Fengguang

Hi Fengguang, Any updates on this? How do you see the way forward?

On Fri, May 6, 2016 at 10:08 AM, Dmitry Vyukov <dvyukov(a)google.com&gt; wrote: > On Thu, May 5, 2016 at 4:48 PM, Fengguang Wu <fengguang.wu(a)intel.com&gt; wrote: >> Hi Dmitry, >> >> On Tue, May 03, 2016 at 10:00:04AM +0200, Dmitry Vyukov wrote: >>> On Tue, May 3, 2016 at 6:01 AM, Fengguang Wu <fengguang.wu(a)intel.com&gt; wrote: >>> >> One quick question is i recall an early discussion is that this enabling requires new gcc, >>> >> is this still be true? >>> > >>> > I see Debian just upgraded its gcc packages to 6.1, which makes it >>> > more convenient for us to integrate with syzkaller. >>> > >>> > Since syzkaller is more about a client-server system than a hands on >>> > tool like trinity, it may need non-trivial modifications to 0-day and >>> > possibly syzkaller to achieve the integration. For example, syzkaller >>> > and 0-day each has its own way to run QEMU, which should be resolved >>> > somehow. >>> > >>> > Another question is, the 0-day model relies heavily on bisecting to >>> > cover the large number of kernel trees. If a bug cannot be easily >>> > reproduced and bisected, we don't know to whom it should be reported >>> > to. Trinity offers an option "-sN: use N as random seed." to help >>> > reproduce bugs it found. I wonder if syzkaller has similar mechanism >>> > to help with reproducing bugs it found? >>> >>> >>> Hi Fengguang, >>> >>> Yes, syzkaller setup is more elaborate, but architecture is pretty >>> modular so I believe we can resolve these issues. >> >> Yes syzkaller looks very interesting. I'll try it out in more details >> and try enabling it in 0day. >> >>> What does your infrastructure do? I guess it winds up qemu instances, >>> copies necessary binaries and then runs some command? How does it >>> monitor for crashes? By parsing console output? How do you control >>> duration of testing? By instructing trinity to do fixed number of >>> iterations, or by killing qemu? >> >> Basically each 0day boot server runs a number of QEMU instances >> with various locally cached QEMU images. Trinity binaries are built >> into these images. A boot script in the images will run trinity in >> background, sleep for some time and then kill trinity. If tests run >> smoothly, the scripts inside the QEMU image will be able to shutdown >> VM in time. Otherwise if things get stuck inside VM, the host side >> script which started the QEMU VM and monitors its serial output may >> decide to kill the QEMU process when either timeout or find kernel >> panic patterns. > > > On second thought, what do you think about running syzkaller as is? > Syzkaller does roughly the same thing. Syzkaller is configurable with > kernel, image, qemu-binary, etc. I can provide any missing > functionality for automation (e.g. run for an hour and exit). That > would be the simplest option. > > If it does not work for you for some reason, we would need a way to > copy files to/from a VM with scp. Then we could copy the corpus.zip > into a VM, start fuzzing for some time, then it will finish and save > an updated corpus.zip, then we copy out the corpus to host for > subsequent runs. > > >>> Syzkaller already has a number of simpler tools. For example, >>> syz-stress works more or less like trinity -- you can copy it onto >>> test machine and instruct to run random programs. I can cook another >>> tool that will fit into your setup. >> >> OK, we may sort details out in private emails. Preferably the resulted >> system should get the most out of what syzkaller offers and excels. >> >>> One inherent complication, though, >>> is that syzkaller is stateful, it needs to accumulate corpus of >>> interesting programs (based on code coverage) to work more >>> efficiently. So we will need some way to save corpus from qemu >>> instances externally and then pass the corpus to new qemu instances. >>> But on the other hand, that was proven to be super efficient in >>> finding regressions -- namely, in 15 minutes it can test almost >>> everything that it discovered previously. >> >> That looks very impressive! >> >>> Re reproducers: syzkaller can do even better than trinity. Syzkaller >>> logs all executed programs to stdout (in default mode, or it can also >>> log to local files or kernel console). If you have the log with >>> programs that were executed before the crash, then it's possible (1) >>> to re-execute the log on other kernels, or (2) extract the guilty >>> program from the log, minimize it and convert to an executable C >>> program to present to developers >>> (https://github.com/google/syzkaller/wiki/Crash-reproducer-programs). >>> Based on my experience (2) is real game changer, developers much more >>> willing to fix bugs when they have a small reproducer. >> >> Cannot agree any more -- it looks the right way to go. >> Hope we can make it run in 0day soon! >> >> Thanks, >> Fengguang

On Tue, Jun 7, 2016 at 3:04 PM, Fengguang Wu <fengguang.wu(a)intel.com&gt; wrote: > Hi Dmitry, > > On Mon, Jun 06, 2016 at 07:11:02PM +0200, Dmitry Vyukov wrote: >> Hi Fengguang, >> >> Any updates on this? How do you see the way forward? > > Sorry no progress yet, however syzkaller is in our high priority list > and will be my next major topic to focus on. Great! Ping this thread when you start. We are also interested and ready to do supporting work.