I would be interested in this as well. Justin Bovee ProSupport Master Engineer Dell | Enterprise Support Services Office Number: (512) 723-9536 ext. 7239536 M-F (8:00am – 5:00pm CST) How am I doing? Email my manager Rocky_Salim(a)dell.com with any feedback -----Original Message----- From: HPDD-discuss [mailto:hpdd-discuss-bounces@lists.01.org] On Behalf Of Michael Kluge Sent: Tuesday, October 21, 2014 5:24 AM To: hpdd-discuss(a)lists.01.org Subject: [HPDD-discuss] logstash patterns for Lustre syslog Hi list, does anyone have a set of logstash/grok patterns for Lustre logs that you can/want to share? Regards, Michael -- Dr.-Ing. Michael Kluge Technische Universität Dresden Center for Information Services and High Performance Computing (ZIH) D-01062 Dresden Germany Contact: Willersbau, Room A 208 Phone: (+49) 351 463-34217 Fax: (+49) 351 463-37773 e-mail: michael.kluge(a)tu-dresden.de WWW: http://www.tu-dresden.de/zih _______________________________________________ HPDD-discuss mailing list HPDD-discuss(a)lists.01.org https://lists.01.org/mailman/listinfo/hpdd-discuss

Hi all, We are implementing the ELK stack at NCI and I've been responsible for setting it up. I've just created a Gist on GitHub to share some of the patterns I've been using: https://gist.github.com/joshuar/c37f310249d387aef530 The first section is a set of custom patterns I use to help match elements of the Lustre messages. The second section is the actual messages I'm matching with the grok filter. My method is to match against a bunch of specific message formats we are interested in and failing that, match a generic format. The default behavior of the grok filter is to break on first match, so this works well this way. YMMV. Note this is a work in progress and happy to hear suggestions! Lustre devs, it'd be really nice if you guys to agree on a common message format and use it consistently within the source. This would make using tools like Logstash much easier! Best regards, Joshua Rich HPC Systems Administrator HPC Systems and Cloud Services National Computational Infrastructure The Australian National University 143 Ward Road Acton, ACT, 2601 T +61 2 6125 2360 joshua.rich(a)anu.edu.au http://nci.org.au ________________________________________ From: HPDD-discuss <hpdd-discuss-bounces(a)lists.01.org&gt; on behalf of Brock Palen <brockp(a)umich.edu&gt; Sent: Wednesday, 22 October 2014 12:45 AM To: Justin_Bovee(a)Dell.com Cc: hpdd-discuss(a)lists.01.org Subject: Re: [HPDD-discuss] logstash patterns for Lustre syslog I would also be interested. I use logstash + graphite right now to make graphs of lustre patterns. Brock Palen www.umich.edu/~brockp CAEN Advanced Computing XSEDE Campus Champion brockp(a)umich.edu (734)936-1985 On Oct 21, 2014, at 9:43 AM, <Justin_Bovee(a)Dell.com&gt; <Justin_Bovee(a)Dell.com&gt; wrote: > I would be interested in this as well. > > Justin Bovee > ProSupport Master Engineer > Dell | Enterprise Support Services > Office Number: (512) 723-9536 ext. 7239536 > M-F (8:00am – 5:00pm CST) > How am I doing? Email my manager Rocky_Salim(a)dell.com with any feedback > > -----Original Message----- > From: HPDD-discuss [mailto:hpdd-discuss-bounces@lists.01.org] On Behalf Of Michael Kluge > Sent: Tuesday, October 21, 2014 5:24 AM > To: hpdd-discuss(a)lists.01.org > Subject: [HPDD-discuss] logstash patterns for Lustre syslog > > Hi list, > > does anyone have a set of logstash/grok patterns for Lustre logs that you can/want to share? > > > Regards, Michael > > -- > Dr.-Ing. Michael Kluge > > Technische Universität Dresden > Center for Information Services and > High Performance Computing (ZIH) > D-01062 Dresden > Germany > > Contact: > Willersbau, Room A 208 > Phone: (+49) 351 463-34217 > Fax: (+49) 351 463-37773 > e-mail: michael.kluge(a)tu-dresden.de > WWW: http://www.tu-dresden.de/zih > > _______________________________________________ > HPDD-discuss mailing list > HPDD-discuss(a)lists.01.org > https://lists.01.org/mailman/listinfo/hpdd-discuss _______________________________________________ HPDD-discuss mailing list HPDD-discuss(a)lists.01.org https://lists.01.org/mailman/listinfo/hpdd-discuss

Hey Brock, I'd be really interested in any notes on your Graphite set-up if you are willing to share. I've been interested in whether I could use Graphite as part of a wider monitoring and reporting framework we are looking into. Using it for Lustre though sounds interesting! Best regards, Joshua Rich HPC Systems Administrator HPC Systems and Cloud Services National Computational Infrastructure The Australian National University 143 Ward Road Acton, ACT, 2601 T +61 2 6125 2360 joshua.rich(a)anu.edu.au http://nci.org.au ________________________________________ From: HPDD-discuss <hpdd-discuss-bounces(a)lists.01.org&gt; on behalf of Brock Palen <brockp(a)umich.edu&gt; Sent: Wednesday, 22 October 2014 12:45 AM To: Justin_Bovee(a)Dell.com Cc: hpdd-discuss(a)lists.01.org Subject: Re: [HPDD-discuss] logstash patterns for Lustre syslog I would also be interested. I use logstash + graphite right now to make graphs of lustre patterns. Brock Palen www.umich.edu/~brockp CAEN Advanced Computing XSEDE Campus Champion brockp(a)umich.edu (734)936-1985 On Oct 21, 2014, at 9:43 AM, <Justin_Bovee(a)Dell.com&gt; <Justin_Bovee(a)Dell.com&gt; wrote: > I would be interested in this as well. > > Justin Bovee > ProSupport Master Engineer > Dell | Enterprise Support Services > Office Number: (512) 723-9536 ext. 7239536 > M-F (8:00am – 5:00pm CST) > How am I doing? Email my manager Rocky_Salim(a)dell.com with any feedback > > -----Original Message----- > From: HPDD-discuss [mailto:hpdd-discuss-bounces@lists.01.org] On Behalf Of Michael Kluge > Sent: Tuesday, October 21, 2014 5:24 AM > To: hpdd-discuss(a)lists.01.org > Subject: [HPDD-discuss] logstash patterns for Lustre syslog > > Hi list, > > does anyone have a set of logstash/grok patterns for Lustre logs that you can/want to share? > > > Regards, Michael > > -- > Dr.-Ing. Michael Kluge > > Technische Universität Dresden > Center for Information Services and > High Performance Computing (ZIH) > D-01062 Dresden > Germany > > Contact: > Willersbau, Room A 208 > Phone: (+49) 351 463-34217 > Fax: (+49) 351 463-37773 > e-mail: michael.kluge(a)tu-dresden.de > WWW: http://www.tu-dresden.de/zih > > _______________________________________________ > HPDD-discuss mailing list > HPDD-discuss(a)lists.01.org > https://lists.01.org/mailman/listinfo/hpdd-discuss _______________________________________________ HPDD-discuss mailing list HPDD-discuss(a)lists.01.org https://lists.01.org/mailman/listinfo/hpdd-discuss

Justin that is cool, Joshua, I don't think graphite is good for monitoring (notifications) but is good for reporting. I hope to add more rules to logstash to send alerts for slow_commit and other errors that normally tell me lustre is going to start getting mad when the backend gets slow. For logstash I had to make my own grok {} rules, I made a helper script that takes all the stats and munges them into json so that logstash can parse them. I run two inputs, one for summary (per OST/MDT stats) and one for per clinet stats. Because this turns into a lot of data, for the per client stats I keep them less time, and I don't collect them at the same resolution. I am doing this for a 30 OST 1 MDT, ~1100 client setup. THe grahpite server is not sharded and currently has just one spindle (yep lose everything if the disk dies) command => "/root/logstash.git/helpers/json-stats-wrapper.py /proc/fs/lustre/mdt/*/exports/*/stats" command => "/root/logstash.git/helpers/json-stats-wrapper.py /proc/fs/lustre/mdt/*/md_stats" The end result I am storing on my current graphite server, I hit about 27,000 commits/minute and most of those are from lusture. I have attached some example plots. The read_bytes rate from each OST summary Summed read_bytes rates from all OST's with a moving median over the last 30 points per client MDT open rate (/proc/fs/lustre/mdt/*/exports/*/stats) for the last hour, filtered to the 'most divant' I use this to find the users/jobs who are most of our MDT load. Brock Palen www.umich.edu/~brockp <http://www.umich.edu/~brockp> CAEN Advanced Computing XSEDE Campus Champion brockp(a)umich.edu (734)936-1985 > On Oct 22, 2014, at 2:41 PM, Justin Miller <jupmille(a)iu.edu > <mailto:jupmille@iu.edu>> wrote: > > I have experimented with collectl + graphite a little bit. collectl has > graphite export functionality built in, so in combination with the > Lustre collectl plugins (which I think are maintained by someone from > Terascala), you can send data directly to graphite. > > Lustre collectl plugins: https://github.com/pcpiela/collectl-lustre > > Download those plugins and put them in /usr/share/collectl > > Then specify which plugin to use depending on the type of machine, e.g. > > OSS: collectl --import lustreOSS.ph,sdBC --export graphite,10.10.0.103,d=1 > MDS: collectl --import lustreMDS.ph,sC --export graphite,10.10.0.103,d=1 > CLI: collectl --import lustreClient.ph,sdO --export > graphite,10.10.0.103,d=1 > > The plugins have options you can pass specifying what service data to to > export and more or less detail (the ",sdBC" in the OSS example), > although I can't recall right now what each on means. I seem to recall > specifying a full path on the import option didn't work, it had to be in > /usr/share/collectl > > Setting d=1 will echo the data being sent. Setting d=9 will just echo > data and not send anything to the graphite server. Without the debug > flag, the graphite export won't echo anything. See > http://collectl.sourceforge.net/Graphite.html > > Hopefully this is useful. I am also very interested in any ELK stack or > graphite related projects people are willing to share and discuss. > > Slightly off topic, but my understanding is that the builtin collectl > Lustre plugins are being deprecated in favor of the external Lustre > plugins. (see http://sourceforge.net/p/collectl/mailman/message/31992463/) > > Regards, > > Justin Miller > > On 10/21/14 6:14 PM, Joshua Rich wrote: >> Hey Brock, >> >> I'd be really interested in any notes on your Graphite set-up if you >> are willing to share. I've been interested in whether I could use >> Graphite as part of a wider monitoring and reporting framework we are >> looking into. Using it for Lustre though sounds interesting! >> >> Best regards, >> >> Joshua Rich >> HPC Systems Administrator >> HPC Systems and Cloud Services >> >> National Computational Infrastructure >> The Australian National University >> 143 Ward Road >> Acton, ACT, 2601 >> >> T +61 2 6125 2360 >> joshua.rich(a)anu.edu.au >> http://nci.org.au >> >> ________________________________________ >> From: HPDD-discuss <hpdd-discuss-bounces(a)lists.01.org&gt; on behalf of >> Brock Palen <brockp(a)umich.edu&gt; >> Sent: Wednesday, 22 October 2014 12:45 AM >> To: Justin_Bovee(a)Dell.com >> Cc: hpdd-discuss(a)lists.01.org >> Subject: Re: [HPDD-discuss] logstash patterns for Lustre syslog >> >> I would also be interested. >> >> I use logstash + graphite right now to make graphs of lustre patterns. >> >> Brock Palen >> www.umich.edu/~brockp >> CAEN Advanced Computing >> XSEDE Campus Champion >> brockp(a)umich.edu >> (734)936-1985 >> >> >> >> On Oct 21, 2014, at 9:43 AM, <Justin_Bovee(a)Dell.com&gt; >> <Justin_Bovee(a)Dell.com&gt; wrote: >> >>> I would be interested in this as well. >>> >>> Justin Bovee >>> ProSupport Master Engineer >>> Dell | Enterprise Support Services >>> Office Number: (512) 723-9536 ext. 7239536 >>> M-F (8:00am – 5:00pm CST) >>> How am I doing? Email my manager Rocky_Salim(a)dell.com with any feedback >>> >>> -----Original Message----- >>> From: HPDD-discuss [mailto:hpdd-discuss-bounces@lists.01.org] On >>> Behalf Of Michael Kluge >>> Sent: Tuesday, October 21, 2014 5:24 AM >>> To: hpdd-discuss(a)lists.01.org >>> Subject: [HPDD-discuss] logstash patterns for Lustre syslog >>> >>> Hi list, >>> >>> does anyone have a set of logstash/grok patterns for Lustre logs >>> that you can/want to share? >>> >>> >>> Regards, Michael >>> >>> -- >>> Dr.-Ing. Michael Kluge >>> >>> Technische Universität Dresden >>> Center for Information Services and >>> High Performance Computing (ZIH) >>> D-01062 Dresden >>> Germany >>> >>> Contact: >>> Willersbau, Room A 208 >>> Phone: (+49) 351 463-34217 >>> Fax: (+49) 351 463-37773 >>> e-mail: michael.kluge(a)tu-dresden.de >>> WWW: http://www.tu-dresden.de/zih >>> >>> _______________________________________________ >>> HPDD-discuss mailing list >>> HPDD-discuss(a)lists.01.org >>> https://lists.01.org/mailman/listinfo/hpdd-discuss >> _______________________________________________ >> HPDD-discuss mailing list >> HPDD-discuss(a)lists.01.org >> https://lists.01.org/mailman/listinfo/hpdd-discuss > > > _______________________________________________ > HPDD-discuss mailing list > HPDD-discuss(a)lists.01.org > https://lists.01.org/mailman/listinfo/hpdd-discuss _______________________________________________ HPDD-discuss mailing list HPDD-discuss(a)lists.01.org https://lists.01.org/mailman/listinfo/hpdd-discuss