hyper scan library crash crash.
by tidy＠holonetsecurity.com
Hello!
I have been developing an app with very simple function to compile a pattern and use the pattern to match which needs use Hyperscan library:
After compiling the hyperscan library with following options: -DCMAKE_BUILD_TYPE=debug -DBOOST_ROOT=${BOOST_ROOT} -DBUILD_SHARED_LIBS=1 -DCMAKE_INSTALL_PREFIX=${USR_LIB_PATH}.
The App runs as expected at my compiling machine, and then I copied my app and libhs.so to another machine (Same X86 64bit machine.
however it crashed every time at pattern load function when I execute the app.
The compiling machine and running machine has difference cpuinfo, not sure it has relationship with this.
Please refer detail information from this link: https://github.com/01org/hyperscan/issues/20
Thanks!
6 years, 4 months
Re: [Hyperscan] Hyperscan 4.1.0 building
by Vishal Kotalwar V
Matt & Justin,
As per you advice, I made changes in QEMU to bring SSSE3 cpu flag to guest as well. It solved the issue.
I faced another linking issue which i could solve by following
https://github.com/01org/hyperscan/commit/3dc8bab78fecd9fc69a329c2359cab3...
Now the compilation is complete.
Thanks & regards,
Vishal V. Kotalwar
From: "Vishal Kotalwar V" <vishalkv(a)altencalsoftlabs.com>
To: "hyperscan" <hyperscan(a)lists.01.org>
Sent: Monday, May 9, 2016 1:22:53 PM
Subject: Re: [Hyperscan] Hyperscan 4.1.0 building
Matt,
I tried the cmake command you provided but that did not help. Got the same error.
below is the piece of output from cpuinfo, will it help if I move to hyperscan 4.0.1 (being optimistic but i feel i will hit the same error there as well)
processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 6
model name : QEMU Virtual CPU version 2.0.0
stepping : 3
microcode : 0x1
cpu MHz : 2199.998
cache size : 4096 KB
physical id : 3
siblings : 1
core id : 0
cpu cores : 1
apicid : 3
initial apicid : 3
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl pni vmx cx16 x2apic popcnt hypervisor lahf_lm vnmi ept
bogomips : 4399.99
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
Thanks & regards,
Vishal V. Kotalwar
From: "Matthew Barr" <matthew.barr(a)intel.com>
To: "hyperscan" <hyperscan(a)lists.01.org>
Sent: Monday, May 9, 2016 1:09:19 PM
Subject: Re: [Hyperscan] Hyperscan 4.1.0 building
Hi Vishal,
My guess is that the virtualisation layer isn’t exposing the cpuid flags for SSSE3, which means that gcc doesn’t know what features it can use through the use of -march=native. You can see what processor features are enabled by inspecting the “flags:” line in /proc/cpuinfo
One fix would be to set the processor configuration in QEMU to match the host CPU. Alternately you can specify a target for gcc. Gcc 4.8.5 doesn’t have the “sandybridge” target – they added that later – but you can use the confusingly named “corei7-avx”.
For example, you could run CMake like this:
cmake -DCMAKE_C_FLAGS=’-march=corei7-avx’ -DCMAKE_CXX_FLAGS=’-march=corei7-avx’ -DBUILD_SHARED_LIBS=1 -DBOOST_ROOT=/tmp/boost-1.60 ..
regards,
Matt.
From: Hyperscan [mailto:hyperscan-bounces@lists.01.org] On Behalf Of Vishal Kotalwar V
Sent: Monday, May 09, 2016 5:22 PM
To: hyperscan <hyperscan(a)lists.01.org>
Subject: Re: [Hyperscan] Hyperscan 4.1.0 building
Thanks Justin.
This is Intel Sandy bridge m/c running on QEMU. My VM has 4 cores and 4GB RAM assigned.
Re-attached error log.
Thanks & regards,
Vishal V. Kotalwar
6 years, 5 months
Re: [Hyperscan] Hyperscan 4.1.0 building
by Vishal Kotalwar V
Matt,
I tried the cmake command you provided but that did not help. Got the same error.
below is the piece of output from cpuinfo, will it help if I move to hyperscan 4.0.1 (being optimistic but i feel i will hit the same error there as well)
processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 6
model name : QEMU Virtual CPU version 2.0.0
stepping : 3
microcode : 0x1
cpu MHz : 2199.998
cache size : 4096 KB
physical id : 3
siblings : 1
core id : 0
cpu cores : 1
apicid : 3
initial apicid : 3
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl pni vmx cx16 x2apic popcnt hypervisor lahf_lm vnmi ept
bogomips : 4399.99
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
Thanks & regards,
Vishal V. Kotalwar
From: "Matthew Barr" <matthew.barr(a)intel.com>
To: "hyperscan" <hyperscan(a)lists.01.org>
Sent: Monday, May 9, 2016 1:09:19 PM
Subject: Re: [Hyperscan] Hyperscan 4.1.0 building
Hi Vishal,
My guess is that the virtualisation layer isn’t exposing the cpuid flags for SSSE3, which means that gcc doesn’t know what features it can use through the use of -march=native. You can see what processor features are enabled by inspecting the “flags:” line in /proc/cpuinfo
One fix would be to set the processor configuration in QEMU to match the host CPU. Alternately you can specify a target for gcc. Gcc 4.8.5 doesn’t have the “sandybridge” target – they added that later – but you can use the confusingly named “corei7-avx”.
For example, you could run CMake like this:
cmake -DCMAKE_C_FLAGS=’-march=corei7-avx’ -DCMAKE_CXX_FLAGS=’-march=corei7-avx’ -DBUILD_SHARED_LIBS=1 -DBOOST_ROOT=/tmp/boost-1.60 ..
regards,
Matt.
From: Hyperscan [mailto:hyperscan-bounces@lists.01.org] On Behalf Of Vishal Kotalwar V
Sent: Monday, May 09, 2016 5:22 PM
To: hyperscan <hyperscan(a)lists.01.org>
Subject: Re: [Hyperscan] Hyperscan 4.1.0 building
Thanks Justin.
This is Intel Sandy bridge m/c running on QEMU. My VM has 4 cores and 4GB RAM assigned.
Re-attached error log.
Thanks & regards,
Vishal V. Kotalwar
6 years, 5 months
Re: [Hyperscan] Hyperscan 4.1.0 building
by Vishal Kotalwar V
Thanks Justin.
This is Intel Sandy bridge m/c running on QEMU. My VM has 4 cores and 4GB RAM assigned.
Re-attached error log.
Thanks & regards,
Vishal V. Kotalwar
From: "Justin Viiret" <justin.viiret(a)intel.com>
To: "hyperscan" <hyperscan(a)lists.01.org>
Sent: Monday, May 9, 2016 12:39:29 PM
Subject: Re: [Hyperscan] Hyperscan 4.1.0 building
Hi Vishal,
It looks like the attachment is missing or was scrubbed by the mailing list. Can you paste the error you’re seeing into the body of an email?
It would also be helpful to know what your platform is and which compiler you’re using.
Many thanks,
Justin
From: Hyperscan [mailto:hyperscan-bounces@lists.01.org] On Behalf Of Vishal Kotalwar V
Sent: Monday, May 09, 2016 4:44 PM
To: hyperscan
Subject: [Hyperscan] Hyperscan 4.1.0 building
Hi All,
I am new to hyperscan and want to use it as MPM with Suricata.
I got compilation errors when I tried to build hyperscan library on my m/c
user@ubuntu4:~/downloads/hyperscan-4.1.0/build$ uname -a
Linux ubuntu4 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
Attached is the error log which gives the idea of what I did ...
I googled it and got this
https://github.com/Homebrew/legacy-homebrew/pull/49876
I tried similar thing on my linux by removing "march=native" from c_flags but it did not help.
Please help.
Thanks & regards,
Vishal V. Kotalwar
6 years, 5 months
Hyperscan 4.1.0 building
by Vishal Kotalwar V
Hi All,
I am new to hyperscan and want to use it as MPM with Suricata.
I got compilation errors when I tried to build hyperscan library on my m/c
user@ubuntu4:~/downloads/hyperscan-4.1.0/build$ uname -a
Linux ubuntu4 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
Attached is the error log which gives the idea of what I did ...
I googled it and got this
https://github.com/Homebrew/legacy-homebrew/pull/49876
I tried similar thing on my linux by removing "march=native" from c_flags but it did not help.
Please help.
Thanks & regards,
Vishal V. Kotalwar
6 years, 5 months
hyperscan library, optimizing compilation
by Stanislav Podolsky
Hello!
I have been testing performance of Hyperscan after compiling it with different optimization levels using gcc 4.9.2 and got somewhat unexpected results.
Tests were performed on Ivy Bridge architecture.
Unexpected results:
1. -O2 runs about 5%-10% faster than -O3
2. Compiling with AVX extension gives same or even reduced performance as compiling without AVX extension(using SSE4.2)
If anyone performed similar optimization tuning and can suggest any optimizations, including using different compiler than gcc(for example intel compiler) it would be much appreciated!
Thanks!
Stas
6 years, 5 months
