12:41 p.m.
Hello,
When trying to reproduce the crash I had, I saw that most of the time
when there is no re-ordering set (which can happen even with the default
options) but with loss and delay, the tests can be very slow with the ns4:
ns1 MPTCP -> ns3 (dead:beef:3::2:10013) MPTCP (duration 157ms) [ OK ]
ns1 MPTCP -> ns4 (10.0.3.1:10014 ) MPTCP (duration 33023ms) [ OK ]
ns1 MPTCP -> ns4 (dead:beef:3::1:10015) MPTCP (duration 36485ms) [ OK ]
ns2 MPTCP -> ns1 (10.0.1.1:10016 ) MPTCP (duration 118ms) [ OK ]
(...)
ns2 MPTCP -> ns3 (dead:beef:3::2:10021) MPTCP (duration 126ms) [ OK ]
ns2 MPTCP -> ns4 (10.0.3.1:10022 ) MPTCP (duration 15036ms) [ OK ]
ns2 MPTCP -> ns4 (dead:beef:3::1:10023) MPTCP (duration 11507ms) [ OK ]
ns3 MPTCP -> ns1 (10.0.1.1:10024 ) MPTCP (duration 129ms) [ OK ]
(...)
ns3 MPTCP -> ns2 (dead:beef:2::1:10029) MPTCP (duration 139ms) [ OK ]
ns3 MPTCP -> ns4 (10.0.3.1:10030 ) MPTCP (duration 4317ms) [ OK ]
ns3 MPTCP -> ns4 (dead:beef:3::1:10031) MPTCP (duration 4321ms) [ OK ]
ns4 MPTCP -> ns1 (10.0.1.1:10032 ) MPTCP (duration 48538ms) [ OK ]
ns4 MPTCP -> ns1 (dead:beef:1::1:10033) MPTCP (duration 39943ms) [ OK ]
ns4 MPTCP -> ns2 (10.0.1.2:10034 ) MPTCP (duration 24599ms) [ OK ]
ns4 MPTCP -> ns2 (dead:beef:1::2:10035) MPTCP (duration 23126ms) [ OK ]
ns4 MPTCP -> ns2 (10.0.2.1:10036 ) MPTCP (duration 64512ms) [ OK ]
ns4 MPTCP -> ns2 (dead:beef:2::1:10037) MPTCP (duration 37747ms) [ OK ]
ns4 MPTCP -> ns3 (10.0.2.2:10038 ) MPTCP (duration 4766ms) [ OK ]
ns4 MPTCP -> ns3 (dead:beef:2::2:10039) MPTCP (duration 4772ms) [ OK ]
ns4 MPTCP -> ns3 (10.0.3.2:10040 ) MPTCP (duration 4699ms) [ OK ]
ns4 MPTCP -> ns3 (dead:beef:3::2:10041) MPTCP (duration 4324ms) [ OK ]
Time: 425 seconds
This can be easily reproduced by using:
./mptcp_connect.sh -r 0
When there is re-ordering, the tests are not ~95% of the time that slow.
I guess it depends how much re-ordering we have.
Did you also see this in your tests?
Cheers,
Matt
--
Matthieu Baerts | R&D Engineer
matthieu.baerts(a)tessares.net
Tessares SA | Hybrid Access Solutions
www.tessares.net
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium
4:12 p.m.
Hello,
On 16/11/2019 13:41, Matthieu Baerts wrote:
Hello,
When trying to reproduce the crash I had, I saw that most of the time
when there is no re-ordering set (which can happen even with the default
options) but with loss and delay, the tests can be very slow with the ns4:
In my previous tests, I forgot to mention that I was testing "mptcp: add
basic kselftest for mptcp" commit.
Here are the results I just got when asking the CI to test the latest
origin/export branch:
00:12:45.309 # New MPTCP socket can be blocked via sysctl [ OK ]
00:12:45.327 # setsockopt(..., TCP_ULP, "mptcp", ...) blocked [ OK ]
00:12:47.942 # INFO: validating network environment with pings
00:12:48.530 # INFO: Using loss of 0.34% delay 382 ms on ns3eth4
00:12:48.538 # ns1 MPTCP -> ns1 (10.0.1.1:10000 ) MPTCP (duration
27ms) [ OK ]
00:12:48.600 # ns1 MPTCP -> ns1 (10.0.1.1:10001 ) TCP (duration
24ms) [ OK ]
00:12:48.658 # ns1 TCP -> ns1 (10.0.1.1:10002 ) MPTCP (duration
25ms) [ OK ]
00:12:48.716 # ns1 MPTCP -> ns1 (dead:beef:1::1:10003) MPTCP (duration
25ms) [ OK ]
00:12:48.775 # ns1 MPTCP -> ns1 (dead:beef:1::1:10004) TCP (duration
27ms) [ OK ]
00:12:48.839 # ns1 TCP -> ns1 (dead:beef:1::1:10005) MPTCP (duration
25ms) [ OK ]
00:12:48.899 # ns1 MPTCP -> ns2 (10.0.1.2:10006 ) MPTCP (duration
35ms) [ OK ]
00:12:48.977 # ns1 MPTCP -> ns2 (dead:beef:1::2:10007) MPTCP (duration
32ms) [ OK ]
00:12:49.049 # ns1 MPTCP -> ns2 (10.0.2.1:10008 ) MPTCP (duration
30ms) [ OK ]
00:12:49.112 # ns1 MPTCP -> ns2 (dead:beef:2::1:10009) MPTCP (duration
32ms) [ OK ]
00:12:49.178 # ns1 MPTCP -> ns3 (10.0.2.2:10010 ) MPTCP (duration
41ms) [ OK ]
00:12:49.255 # ns1 MPTCP -> ns3 (dead:beef:2::2:10011) MPTCP (duration
39ms) [ OK ]
00:12:49.330 # ns1 MPTCP -> ns3 (10.0.3.2:10012 ) MPTCP (duration
40ms) [ OK ]
00:12:49.412 # ns1 MPTCP -> ns3 (dead:beef:3::2:10013) MPTCP (duration
43ms) [ OK ]
00:12:49.496 # ns1 MPTCP -> ns4 (10.0.3.1:10014 ) MPTCP (duration
7276ms) [ OK ]
00:12:56.805 # ns1 MPTCP -> ns4 (dead:beef:3::1:10015) MPTCP (duration
43826ms) [ OK ]
00:13:40.667 # ns2 MPTCP -> ns1 (10.0.1.1:10016 ) MPTCP (duration
33ms) [ OK ]
00:13:40.736 # ns2 MPTCP -> ns1 (dead:beef:1::1:10017) MPTCP (duration
33ms) [ OK ]
00:13:40.807 # ns2 MPTCP -> ns3 (10.0.2.2:10018 ) MPTCP (duration
40ms) [ OK ]
00:13:40.887 # ns2 MPTCP -> ns3 (dead:beef:2::2:10019) MPTCP (duration
40ms) [ OK ]
00:13:40.967 # ns2 MPTCP -> ns3 (10.0.3.2:10020 ) MPTCP (duration
38ms) [ OK ]
00:13:41.044 # ns2 MPTCP -> ns3 (dead:beef:3::2:10021) MPTCP (duration
36ms) [ OK ]
00:13:41.116 # ns2 MPTCP -> ns4 (10.0.3.1:10022 ) MPTCP (duration
10156ms) [ OK ]
00:13:51.310 # ns2 MPTCP -> ns4 (dead:beef:3::1:10023) MPTCP (duration
11349ms) [ OK ]
00:14:02.696 # ns3 MPTCP -> ns1 (10.0.1.1:10024 ) MPTCP (duration
41ms) [ OK ]
00:14:02.772 # ns3 MPTCP -> ns1 (dead:beef:1::1:10025) MPTCP (duration
43ms) [ OK ]
00:14:02.851 # ns3 MPTCP -> ns2 (10.0.1.2:10026 ) MPTCP (duration
40ms) [ OK ]
00:14:02.929 # ns3 MPTCP -> ns2 (dead:beef:1::2:10027) MPTCP (duration
43ms) [ OK ]
00:14:03.014 # ns3 MPTCP -> ns2 (10.0.2.1:10028 ) MPTCP (duration
46ms) [ OK ]
00:14:03.102 # ns3 MPTCP -> ns2 (dead:beef:2::1:10029) MPTCP (duration
40ms) [ OK ]
00:14:03.181 # ns3 MPTCP -> ns4 (10.0.3.1:10030 ) MPTCP (duration
4989ms) [ OK ]
00:14:08.206 # ns3 MPTCP -> ns4 (dead:beef:3::1:10031) MPTCP (duration
4606ms) [ OK ]
00:14:12.850 # ns4 MPTCP -> ns1 (10.0.1.1:10032 ) MPTCP (duration
9569ms) [ OK ]
00:14:22.451 # ns4 MPTCP -> ns1 (dead:beef:1::1:10033) MPTCP (duration
18160ms) [ OK ]
00:14:40.647 # ns4 MPTCP -> ns2 (10.0.1.2:10034 ) MPTCP (duration
30587ms) [ OK ]
00:15:11.268 # ns4 MPTCP -> ns2 (dead:beef:1::2:10035) MPTCP (duration
27150ms) [ OK ]
00:15:38.454 # ns4 MPTCP -> ns2 (10.0.2.1:10036 ) MPTCP (duration
17595ms) [ OK ]
00:15:56.085 # ns4 MPTCP -> ns2 (dead:beef:2::1:10037) MPTCP (duration
34875ms) [ OK ]
00:16:30.992 # ns4 MPTCP -> ns3 (10.0.2.2:10038 ) MPTCP (duration
4983ms) [ OK ]
00:16:36.008 # ns4 MPTCP -> ns3 (dead:beef:2::2:10039) MPTCP (duration
4599ms) [ OK ]
00:16:40.641 # ns4 MPTCP -> ns3 (10.0.3.2:10040 ) MPTCP (duration
4605ms) [ OK ]
00:16:45.280 # ns4 MPTCP -> ns3 (dead:beef:3::2:10041) MPTCP (duration
4599ms) [ OK ]
00:16:49.916 # Time: 245 seconds
We can see that there is no re-ordering (Using loss of 0.34% delay 382
ms on ns3eth4). The other tests have re-ordering and no issue (12/22
seconds without and with KASAN)
Cheers,
Matt
--
Matthieu Baerts | R&D Engineer
matthieu.baerts(a)tessares.net
Tessares SA | Hybrid Access Solutions
www.tessares.net
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium
9:22 a.m.
Hello,
On 20/11/2019 17:12, Matthieu Baerts wrote:
Hello,
On 16/11/2019 13:41, Matthieu Baerts wrote:
> Hello,
>
> When trying to reproduce the crash I had, I saw that most of the time
> when there is no re-ordering set (which can happen even with the
> default options) but with loss and delay, the tests can be very slow
> with the ns4:
In my previous tests, I forgot to mention that I was testing "mptcp: add
basic kselftest for mptcp" commit.
Here are the results I just got when asking the CI to test the latest
origin/export branch:
FYI, selftests from last night's CI job failed. But it doesn't look
critical as they failed due to a timeout, probably because of this issue
(no re-ordering set):
01:12:17.147 TAP version 13
01:12:17.160 1..1
01:12:17.211 # selftests: net/mptcp: mptcp_connect.sh
01:12:18.149 [ 53.654519] IPv6: ADDRCONF(NETDEV_CHANGE): ns1eth2: link
becomes ready
01:12:18.434 [ 53.938877] IPv6: ADDRCONF(NETDEV_CHANGE): ns2eth3: link
becomes ready
01:12:18.714 [ 54.219094] IPv6: ADDRCONF(NETDEV_CHANGE): ns3eth4: link
becomes ready
01:12:18.814 # INFO: set ns3-5de1ddd4-0j9d6N dev ns3eth2: ethtool -K
gso off
01:12:18.877 [ 54.382317] IPv6: ADDRCONF(NETDEV_CHANGE): ns2eth1: link
becomes ready
01:12:18.939 # Created /tmp/tmp.Cr7KGLuarg (size 4403797) containing
data sent by client
01:12:19.101 # Created /tmp/tmp.sU9xWEO4c7 (size 7175996) containing
data sent by server
01:12:19.318 # New MPTCP socket can be blocked via sysctl [ OK ]
01:12:19.419 # setsockopt(..., TCP_ULP, "mptcp", ...) blocked [ OK ]
01:12:19.446 # INFO: validating network environment with pings
01:12:22.687 # INFO: Using loss of 0.99% delay 218 ms on ns3eth4
01:12:22.716 # ns1 MPTCP -> ns1 (10.0.1.1:10000 ) MPTCP (duration
171ms) [ OK ]
01:12:23.068 # ns1 MPTCP -> ns1 (10.0.1.1:10001 ) TCP (duration
163ms) [ OK ]
01:12:23.404 # ns1 TCP -> ns1 (10.0.1.1:10002 ) MPTCP (duration
121ms) [ OK ]
01:12:23.700 # ns1 MPTCP -> ns1 (dead:beef:1::1:10003) MPTCP (duration
169ms) [ OK ]
01:12:24.039 # ns1 MPTCP -> ns1 (dead:beef:1::1:10004) TCP (duration
127ms) [ OK ]
01:12:24.343 # ns1 TCP -> ns1 (dead:beef:1::1:10005) MPTCP (duration
131ms) [ OK ]
01:12:24.659 # ns1 MPTCP -> ns2 (10.0.1.2:10006 ) MPTCP (duration
307ms) [ OK ]
01:12:25.155 # ns1 MPTCP -> ns2 (dead:beef:1::2:10007) MPTCP (duration
312ms) [ OK ]
01:12:25.655 # ns1 MPTCP -> ns2 (10.0.2.1:10008 ) MPTCP (duration
312ms) [ OK ]
01:12:26.154 # ns1 MPTCP -> ns2 (dead:beef:2::1:10009) MPTCP (duration
297ms) [ OK ]
01:12:26.623 # ns1 MPTCP -> ns3 (10.0.2.2:10010 ) MPTCP (duration
466ms) [ OK ]
01:12:27.267 # ns1 MPTCP -> ns3 (dead:beef:2::2:10011) MPTCP (duration
366ms) [ OK ]
01:12:27.813 # ns1 MPTCP -> ns3 (10.0.3.2:10012 ) MPTCP (duration
458ms) [ OK ]
01:12:28.445 # ns1 MPTCP -> ns3 (dead:beef:3::2:10013) MPTCP (duration
428ms) [ OK ]
01:12:29.062 # ns1 MPTCP -> ns4 (10.0.3.1:10014 ) MPTCP (duration
34112ms) [ OK ]
01:13:03.333 # ns1 MPTCP -> ns4 (dead:beef:3::1:10015) MPTCP (duration
16252ms) [ OK ]
01:13:19.755 # ns2 MPTCP -> ns1 (10.0.1.1:10016 ) MPTCP (duration
305ms) [ OK ]
01:13:20.253 # ns2 MPTCP -> ns1 (dead:beef:1::1:10017) MPTCP (duration
328ms) [ OK ]
01:13:20.731 # ns2 MPTCP -> ns3 (10.0.2.2:10018 ) MPTCP (duration
430ms) [ OK ]
01:13:21.342 # ns2 MPTCP -> ns3 (dead:beef:2::2:10019) MPTCP (duration
325ms) [ OK ]
01:13:21.845 # ns2 MPTCP -> ns3 (10.0.3.2:10020 ) MPTCP (duration
361ms) [ OK ]
01:13:22.384 # ns2 MPTCP -> ns3 (dead:beef:3::2:10021) MPTCP (duration
332ms) [ OK ]
01:13:22.902 # ns2 MPTCP -> ns4 (10.0.3.1:10022 ) MPTCP (duration
35082ms) [ OK ]
01:13:58.156 # ns2 MPTCP -> ns4 (dead:beef:3::1:10023) MPTCP (duration
39069ms) [ OK ]
01:14:37.397 # ns3 MPTCP -> ns1 (10.0.1.1:10024 ) MPTCP (duration
405ms) [ OK ]
01:14:37.975 # ns3 MPTCP -> ns1 (dead:beef:1::1:10025) MPTCP (duration
434ms) [ OK ]
01:14:38.595 # ns3 MPTCP -> ns2 (10.0.1.2:10026 ) MPTCP (duration
346ms) [ OK ]
01:14:39.118 # ns3 MPTCP -> ns2 (dead:beef:1::2:10027) MPTCP (duration
351ms) [ OK ]
01:14:39.654 # ns3 MPTCP -> ns2 (10.0.2.1:10028 ) MPTCP (duration
363ms) [ OK ]
01:14:40.195 # ns3 MPTCP -> ns2 (dead:beef:2::1:10029) MPTCP (duration
395ms) [ OK ]
01:14:40.770 # ns3 MPTCP -> ns4 (10.0.3.1:10030 ) MPTCP (duration
2908ms) [ OK ]
01:14:43.867 # ns3 MPTCP -> ns4 (dead:beef:3::1:10031) MPTCP (duration
2919ms) [ OK ]
01:14:46.976 # ns4 MPTCP -> ns1 (10.0.1.1:10032 ) MPTCP (duration
52371ms) [ OK ]
01:15:39.533 # ns4 MPTCP -> ns1 (dead:beef:1::1:10033) MPTCP (duration
70078ms) [ OK ]
01:16:49.781 # ns4 MPTCP -> ns2 (10.0.1.2:10034 ) MPTCP (duration
34180ms) [ OK ]
01:17:24.135 # ns4 MPTCP -> ns2 (dead:beef:1::2:10035) MPTCP (duration
46360ms) [ OK ]
01:18:10.673 # ns4 MPTCP -> ns2 (10.0.2.1:10036 ) MPTCP (duration
63281ms) [ OK ]
01:19:14.119 # ns4 MPTCP -> ns2 (dead:beef:2::1:10037) MPTCP
./mptcp_connect.sh: line 104: 1144 Terminated ip netns
exec ${listener_ns} ./mptcp_connect -t $timeout -l -p $port -s
${srv_proto} $extra_args $local_addr < "$sin" > "$sout"
01:19:47.280 # ./mptcp_connect.sh: line 104: 1150 Terminated
ip netns exec ${connector_ns} ./mptcp_connect -t $timeout -p $port -s
${cl_proto} $extra_args $connect_addr < "$cin" > "$cout"
01:19:47.498 not ok 1 selftests: net/mptcp: mptcp_connect.sh # TIMEOUT
The current timeout is 450.
Cheers,
Matt
--
Matthieu Baerts | R&D Engineer
matthieu.baerts(a)tessares.net
Tessares SA | Hybrid Access Solutions
www.tessares.net
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium
605
days inactive
619
days old
2 comments
1 participants
participants (1)
-
Matthieu Baerts