We want to use mptcp_pm_nl_rm_addr_received to deal with both removing an address and removing a subflow. But it not work. Here is the problem: Suppose there are three subflows, 1 local_id=0 remote_id=0 2 local_id=0 remote_id=1 3 local_id=1 remote_id=0. Here we want to remove the local subflow, the No.3 subflow, so we passed msk->pm.rm_id=1 to mptcp_pm_nl_rm_addr_received. According to this logic, if (msk->pm.rm_id != subflow->remote_id && msk->pm.rm_id != subflow->local_id) We removed the wrong subflow, the No.2 subflow. So we need to deal with removing an address and removing a subflow separately. We check subflow->remote_id in mptcp_pm_nl_rm_addr_received to remove an address and check subflow->local_id in mptcp_pm_nl_rm_subflow_received to remove a subflow. Signed-off-by: Geliang Tang <geliangtang(a)gmail.com&gt; --- net/mptcp/pm_netlink.c | 43 +++++++++++++++++++++++++++++++++++++----- net/mptcp/protocol.h | 1 + 2 files changed, 39 insertions(+), 5 deletions(-) diff --git a/net/mptcp/pm_netlink.c b/net/mptcp/pm_netlink.c index 0b1718921cc6..b3771a188a4c 100644 --- a/net/mptcp/pm_netlink.c +++ b/net/mptcp/pm_netlink.c @@ -271,23 +271,56 @@ void mptcp_pm_nl_rm_addr_received(struct mptcp_sock *msk) if (list_empty(&msk->conn_list)) return; - msk->pm.add_addr_accepted--; - msk->pm.subflows--; - WRITE_ONCE(msk->pm.accept_addr, true); + list_for_each_entry_safe(subflow, tmp, &msk->conn_list, node) { + struct sock *ssk = mptcp_subflow_tcp_sock(subflow); + int how = RCV_SHUTDOWN | SEND_SHUTDOWN; + long timeout = 0; + + if (msk->pm.rm_id != subflow->remote_id) + continue; + + spin_unlock_bh(&msk->pm.lock); + mptcp_subflow_shutdown(sk, ssk, how); + __mptcp_close_ssk(sk, ssk, subflow, timeout); + spin_lock_bh(&msk->pm.lock); + + msk->pm.add_addr_accepted--; + msk->pm.subflows--; + WRITE_ONCE(msk->pm.accept_addr, true); + + break; + } +} + +void mptcp_pm_nl_rm_subflow_received(struct mptcp_sock *msk) +{ + struct mptcp_subflow_context *subflow, *tmp; + struct sock *sk = (struct sock *)msk; + + pr_debug("rm_id %d", msk->pm.rm_id); + + if (!msk->pm.rm_id) + return; + + if (list_empty(&msk->conn_list)) + return; list_for_each_entry_safe(subflow, tmp, &msk->conn_list, node) { struct sock *ssk = mptcp_subflow_tcp_sock(subflow); int how = RCV_SHUTDOWN | SEND_SHUTDOWN; long timeout = 0; - if (msk->pm.rm_id != subflow->remote_id && - msk->pm.rm_id != subflow->local_id) + if (msk->pm.rm_id != subflow->local_id) continue; spin_unlock_bh(&msk->pm.lock); mptcp_subflow_shutdown(sk, ssk, how); __mptcp_close_ssk(sk, ssk, subflow, timeout); spin_lock_bh(&msk->pm.lock); + + msk->pm.local_addr_used--; + msk->pm.subflows--; + break; } } diff --git a/net/mptcp/protocol.h b/net/mptcp/protocol.h index 4b8a5308aeed..2849e1c8769b 100644 --- a/net/mptcp/protocol.h +++ b/net/mptcp/protocol.h @@ -476,6 +476,7 @@ void mptcp_pm_nl_fully_established(struct mptcp_sock *msk); void mptcp_pm_nl_subflow_established(struct mptcp_sock *msk); void mptcp_pm_nl_add_addr_received(struct mptcp_sock *msk); void mptcp_pm_nl_rm_addr_received(struct mptcp_sock *msk); +void mptcp_pm_nl_rm_subflow_received(struct mptcp_sock *msk); int mptcp_pm_nl_get_local_id(struct mptcp_sock *msk, struct sock_common *skc); static inline struct mptcp_ext *mptcp_get_ext(struct sk_buff *skb) -- 2.17.1

On Mon, 2020-08-24 at 19:36 +0800, Geliang Tang wrote: Since here the caller can block and have already acquired the msk socket lock, you could simply acquire the pm spin lock and call: mptcp_pm_nl_rm_subflow_received(); No need for an additional work queue scheduling, I think. Cheers, Paolo

This patch added the missing remote_id set in subflow_ulp_clone. And also updated the remote_id set in __mptcp_subflow_connect, to make the code more like local_id set. Signed-off-by: Geliang Tang <geliangtang(a)gmail.com&gt; --- net/mptcp/subflow.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/net/mptcp/subflow.c b/net/mptcp/subflow.c index 82f6d2d9e39e..0a7b80711096 100644 --- a/net/mptcp/subflow.c +++ b/net/mptcp/subflow.c @@ -1042,6 +1042,7 @@ int __mptcp_subflow_connect(struct sock *sk, const struct mptcp_addr_info *loc, struct mptcp_subflow_context *subflow; struct sockaddr_storage addr; int local_id = loc->id; + int remote_id = remote->id; struct socket *sf; struct sock *ssk; u32 remote_token; @@ -1072,7 +1073,6 @@ int __mptcp_subflow_connect(struct sock *sk, const struct mptcp_addr_info *loc, subflow->remote_key = msk->remote_key; subflow->local_key = msk->local_key; subflow->token = msk->token; - subflow->remote_id = remote->id; mptcp_info2sockaddr(loc, &addr); addrlen = sizeof(struct sockaddr_in); @@ -1086,10 +1086,11 @@ int __mptcp_subflow_connect(struct sock *sk, const struct mptcp_addr_info *loc, goto failed; mptcp_crypto_key_sha(subflow->remote_key, &remote_token, NULL); - pr_debug("msk=%p remote_token=%u local_id=%d", msk, remote_token, - local_id); + pr_debug("msk=%p remote_token=%u local_id=%d remote_id=%d", msk, remote_token, + local_id, remote_id); subflow->remote_token = remote_token; subflow->local_id = local_id; + subflow->remote_id = remote_id; subflow->request_join = 1; subflow->request_bkup = !!(loc->flags & MPTCP_PM_ADDR_FLAG_BACKUP); mptcp_info2sockaddr(remote, &addr); @@ -1354,6 +1355,7 @@ static void subflow_ulp_clone(const struct request_sock *req, new_ctx->fully_established = 1; new_ctx->backup = subflow_req->backup; new_ctx->local_id = subflow_req->local_id; + new_ctx->remote_id = subflow_req->remote_id; new_ctx->token = subflow_req->token; new_ctx->thmac = subflow_req->thmac; } -- 2.17.1

This patch added two new mibs for RM_ADDR, named MPTCP_MIB_RMADDR and MPTCP_MIB_RMSUBFLOW, when the RM_ADDR suboption is received, increase the first mib counter, when the local subflow is removed, increase the second mib counter. Suggested-by: Matthieu Baerts <matthieu.baerts(a)tessares.net&gt; Suggested-by: Paolo Abeni <pabeni(a)redhat.com&gt; Signed-off-by: Geliang Tang <geliangtang(a)gmail.com&gt; --- net/mptcp/mib.c | 2 ++ net/mptcp/mib.h | 2 ++ net/mptcp/pm_netlink.c | 5 +++++ 3 files changed, 9 insertions(+) diff --git a/net/mptcp/mib.c b/net/mptcp/mib.c index 056986c7a228..3b076a636228 100644 --- a/net/mptcp/mib.c +++ b/net/mptcp/mib.c @@ -27,6 +27,8 @@ static const struct snmp_mib mptcp_snmp_list[] = { SNMP_MIB_ITEM("OFOMerge", MPTCP_MIB_OFOMERGE), SNMP_MIB_ITEM("NoDSSInWindow", MPTCP_MIB_NODSSWINDOW), SNMP_MIB_ITEM("DuplicateData", MPTCP_MIB_DUPDATA), + SNMP_MIB_ITEM("RmAddr", MPTCP_MIB_RMADDR), + SNMP_MIB_ITEM("RmSubflow", MPTCP_MIB_RMSUBFLOW), SNMP_MIB_SENTINEL }; diff --git a/net/mptcp/mib.h b/net/mptcp/mib.h index 937a177729f1..7f2b6685c034 100644 --- a/net/mptcp/mib.h +++ b/net/mptcp/mib.h @@ -20,6 +20,8 @@ enum linux_mptcp_mib_field { MPTCP_MIB_OFOMERGE, /* Segments merged in OoO queue */ MPTCP_MIB_NODSSWINDOW, /* Segments not in MPTCP windows */ MPTCP_MIB_DUPDATA, /* Segments discarded due to duplicate DSS */ + MPTCP_MIB_RMADDR, /* Received RM_ADDR */ + MPTCP_MIB_RMSUBFLOW, /* Remove a subflow */ __MPTCP_MIB_MAX }; diff --git a/net/mptcp/pm_netlink.c b/net/mptcp/pm_netlink.c index cd37d94a30fe..2b286730c98d 100644 --- a/net/mptcp/pm_netlink.c +++ b/net/mptcp/pm_netlink.c @@ -15,6 +15,7 @@ #include <uapi/linux/mptcp.h> #include "protocol.h" +#include "mib.h" /* forward declaration */ static struct genl_family mptcp_genl_family; @@ -301,6 +302,8 @@ void mptcp_pm_nl_rm_addr_received(struct mptcp_sock *msk) msk->pm.subflows--; WRITE_ONCE(msk->pm.accept_addr, true); + __MPTCP_INC_STATS(sock_net(sk), MPTCP_MIB_RMADDR); + break; } } @@ -334,6 +337,8 @@ void mptcp_pm_nl_rm_subflow_received(struct mptcp_sock *msk) msk->pm.local_addr_used--; msk->pm.subflows--; + __MPTCP_INC_STATS(sock_net(sk), MPTCP_MIB_RMSUBFLOW); + break; } } -- 2.17.1

This patch added the remove addr and subflow test cases and two new functions. The first function run_remove_tests calls do_transfer with two new arguments, rm_nr_ns1 and rm_nr_ns2, for the numbers of addresses should be removed during the transfer process in namespace 1 and namespace 2. If both these two arguments are 0, we do the join test cases with "mptcp_connect -j" command. Otherwise, do the remove test cases with "mptcp_connect -r" command. The second function chk_rm_nr checks the RM_ADDR related mibs's counters. The output of the test cases looks like this: 11 remove single subflow syn[ ok ] - synack[ ok ] - ack[ ok ] rm [ ok ] - sf [ ok ] 12 remove multiple subflows syn[ ok ] - synack[ ok ] - ack[ ok ] rm [ ok ] - sf [ ok ] 13 remove single address syn[ ok ] - synack[ ok ] - ack[ ok ] rm [ ok ] - sf [ ok ] 14 remove subflow and signal syn[ ok ] - synack[ ok ] - ack[ ok ] rm [ ok ] - sf [ ok ] 15 remove subflows and signal syn[ ok ] - synack[ ok ] - ack[ ok ] rm [ ok ] - sf [ ok ] Signed-off-by: Geliang Tang <geliangtang(a)gmail.com&gt; --- .../testing/selftests/net/mptcp/mptcp_join.sh | 142 +++++++++++++++++- 1 file changed, 139 insertions(+), 3 deletions(-) diff --git a/tools/testing/selftests/net/mptcp/mptcp_join.sh b/tools/testing/selftests/net/mptcp/mptcp_join.sh index f39c1129ce5f..5548799bab6c 100755 --- a/tools/testing/selftests/net/mptcp/mptcp_join.sh +++ b/tools/testing/selftests/net/mptcp/mptcp_join.sh @@ -8,6 +8,7 @@ cin="" cout="" ksft_skip=4 timeout=30 +mptcp_connect="" capture=0 TEST_COUNT=0 @@ -132,6 +133,8 @@ do_transfer() cl_proto="$3" srv_proto="$4" connect_addr="$5" + rm_nr_ns1="$6" + rm_nr_ns2="$7" port=$((10000+$TEST_COUNT)) TEST_COUNT=$((TEST_COUNT+1)) @@ -156,14 +159,44 @@ do_transfer() sleep 1 fi - ip netns exec ${listener_ns} ./mptcp_connect -j -t $timeout -l -p $port -s ${srv_proto} 0.0.0.0 < "$sin" > "$sout" & + if [[ $rm_nr_ns1 -eq 0 && $rm_nr_ns2 -eq 0 ]]; then + mptcp_connect="./mptcp_connect -j" + else + mptcp_connect="./mptcp_connect -r" + fi + + ip netns exec ${listener_ns} $mptcp_connect -t $timeout -l -p $port -s ${srv_proto} 0.0.0.0 < "$sin" > "$sout" & spid=$! sleep 1 - ip netns exec ${connector_ns} ./mptcp_connect -j -t $timeout -p $port -s ${cl_proto} $connect_addr < "$cin" > "$cout" & + ip netns exec ${connector_ns} $mptcp_connect -t $timeout -p $port -s ${cl_proto} $connect_addr < "$cin" > "$cout" & cpid=$! + if [ $rm_nr_ns1 -gt 0 ]; then + counter=1 + sleep 1 + + while [ $counter -le $rm_nr_ns1 ] + do + ip netns exec ${listener_ns} ./pm_nl_ctl del $counter + sleep 1 + let counter+=1 + done + fi + + if [ $rm_nr_ns2 -gt 0 ]; then + counter=1 + sleep 1 + + while [ $counter -le $rm_nr_ns2 ] + do + ip netns exec ${connector_ns} ./pm_nl_ctl del $counter + sleep 1 + let counter+=1 + done + fi + wait $cpid retc=$? wait $spid @@ -219,7 +252,24 @@ run_tests() connect_addr="$3" lret=0 - do_transfer ${listener_ns} ${connector_ns} MPTCP MPTCP ${connect_addr} + do_transfer ${listener_ns} ${connector_ns} MPTCP MPTCP ${connect_addr} 0 0 + lret=$? + if [ $lret -ne 0 ]; then + ret=$lret + return + fi +} + +run_remove_tests() +{ + listener_ns="$1" + connector_ns="$2" + connect_addr="$3" + rm_nr_ns1="$4" + rm_nr_ns2="$5" + lret=0 + + do_transfer ${listener_ns} ${connector_ns} MPTCP MPTCP ${connect_addr} ${rm_nr_ns1} ${rm_nr_ns2} lret=$? if [ $lret -ne 0 ]; then ret=$lret @@ -276,6 +326,43 @@ chk_join_nr() fi } +chk_rm_nr() +{ + local rm_addr_nr=$1 + local rm_subflow_nr=$2 + local count + local dump_stats + + printf "%-39s %s" " " "rm " + count=`ip netns exec $ns1 nstat -as | grep MPTcpExtRmAddr | awk '{print $2}'` + [ -z "$count" ] && count=0 + if [ "$count" != "$rm_addr_nr" ]; then + echo "[fail] got $count RM_ADDR[s] expected $rm_addr_nr" + ret=1 + dump_stats=1 + else + echo -n "[ ok ]" + fi + + echo -n " - sf " + count=`ip netns exec $ns2 nstat -as | grep MPTcpExtRmSubflow | awk '{print $2}'` + [ -z "$count" ] && count=0 + if [ "$count" != "$rm_subflow_nr" ]; then + echo "[fail] got $count RM_SUBFLOW[s] expected $rm_subflow_nr" + ret=1 + dump_stats=1 + else + echo "[ ok ]" + fi + + if [ "${dump_stats}" = 1 ]; then + echo Server ns stats + ip netns exec $ns1 nstat -as | grep MPTcp + echo Client ns stats + ip netns exec $ns2 nstat -as | grep MPTcp + fi +} + sin=$(mktemp) sout=$(mktemp) cin=$(mktemp) @@ -363,6 +450,55 @@ ip netns exec $ns2 ./pm_nl_ctl add 10.0.4.2 flags subflow run_tests $ns1 $ns2 10.0.1.1 chk_join_nr "multiple subflows and signal" 3 3 3 +# single subflow, remove +reset +ip netns exec $ns1 ./pm_nl_ctl limits 0 1 +ip netns exec $ns2 ./pm_nl_ctl limits 0 1 +ip netns exec $ns2 ./pm_nl_ctl add 10.0.3.2 flags subflow +run_remove_tests $ns1 $ns2 10.0.1.1 0 1 +chk_join_nr "remove single subflow" 1 1 1 +chk_rm_nr 1 1 + +# multiple subflows, remove +reset +ip netns exec $ns1 ./pm_nl_ctl limits 0 2 +ip netns exec $ns2 ./pm_nl_ctl limits 0 2 +ip netns exec $ns2 ./pm_nl_ctl add 10.0.2.2 flags subflow +ip netns exec $ns2 ./pm_nl_ctl add 10.0.3.2 flags subflow +run_remove_tests $ns1 $ns2 10.0.1.1 0 2 +chk_join_nr "remove multiple subflows" 2 2 2 +chk_rm_nr 2 2 + +# single address, remove +reset +ip netns exec $ns1 ./pm_nl_ctl limits 0 1 +ip netns exec $ns1 ./pm_nl_ctl add 10.0.2.1 flags signal +ip netns exec $ns2 ./pm_nl_ctl limits 1 1 +run_remove_tests $ns1 $ns2 10.0.1.1 1 0 +chk_join_nr "remove single address" 1 1 1 +chk_rm_nr 0 0 + +# subflow and signal, remove +reset +ip netns exec $ns1 ./pm_nl_ctl limits 0 2 +ip netns exec $ns1 ./pm_nl_ctl add 10.0.2.1 flags signal +ip netns exec $ns2 ./pm_nl_ctl limits 1 2 +ip netns exec $ns2 ./pm_nl_ctl add 10.0.3.2 flags subflow +run_remove_tests $ns1 $ns2 10.0.1.1 1 1 +chk_join_nr "remove subflow and signal" 2 2 2 +chk_rm_nr 1 1 + +# subflows and signal, remove +reset +ip netns exec $ns1 ./pm_nl_ctl limits 0 3 +ip netns exec $ns1 ./pm_nl_ctl add 10.0.2.1 flags signal +ip netns exec $ns2 ./pm_nl_ctl limits 1 3 +ip netns exec $ns2 ./pm_nl_ctl add 10.0.3.2 flags subflow +ip netns exec $ns2 ./pm_nl_ctl add 10.0.4.2 flags subflow +run_remove_tests $ns1 $ns2 10.0.1.1 1 2 +chk_join_nr "remove subflows and signal" 3 3 3 +chk_rm_nr 2 2 + # single subflow, syncookies reset_with_cookies ip netns exec $ns1 ./pm_nl_ctl limits 0 1 -- 2.17.1