bf940a84c8
We encountered an unexpected system reboot during a stress test, and based on the kernel warning backtrace, it looks like a race condition in the RCU subsystem caused this issue. It is similar to the issue reported at https://lore.kernel.org/all/20210917211148.GU4156@paulmck-ThinkPad-P17-Gen-1/#t Guillaume Morin applied two patches, then he can not reproduce it again. commit 2431774f04 [rcu: Mark accesses to rcu_state.n_force_qs] had been in upstream Linux kernel. https://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git/ commit/?h=rcu/next&id=325a2030b90376d179a129794e2fae2b24d73923 [rcu: Tighten rcu_advance_cbs_nowake() checks] Paul E. McKenney provided, which was not pushed to the mainline now. (It would be slated for the v5.17 merge window by default) Basically the rcu_advance_cbs() == true warning in rcu_advance_cbs_nowake() is firing then everything eventually gets stuck on RCU synchronization. WARNING: CPU: 35 PID: 2743975 at kernel/rcu/tree.c:1589 rcu_advance_cbs_nowake+0x78/0x80 ...... Call Trace: call_rcu+0x173/0x5c0 task_work_run+0x6d/0xa0 exit_to_user_mode_prepare+0x130/0x140 syscall_exit_to_user_mode+0x27/0x1d0 entry_SYSCALL_64_after_hwframe+0x44/0xa9 Verification: The formal regression tests were carried out by colleagues in the test team at Wind River, which include userspace packages, ltp and posix， basic networking test etc. Closes-Bug: #1952710 Signed-off-by: Jiping Ma <jiping.ma2@windriver.com> Change-Id: I4ee7d3d007edced81c9ac43c5850f941f1d393ee
46 lines
1.8 KiB
Diff
46 lines
1.8 KiB
Diff
From 21e034adb9df3581fda926a29b3a11bda38ba93b Mon Sep 17 00:00:00 2001
|
|
From: "Paul E. McKenney" <paulmck@kernel.org>
|
|
Date: Fri, 17 Sep 2021 15:04:48 -0700
|
|
Subject: [PATCH] rcu: Tighten rcu_advance_cbs_nowake() checks
|
|
|
|
Currently, rcu_advance_cbs_nowake() checks that a grace period is in
|
|
progress, however, that grace period could end just after the check.
|
|
This commit rechecks that a grace period is still in progress the lock.
|
|
The grace period cannot end while the current CPU's rcu_node structure's
|
|
->lock is held, thus avoiding false positives from the WARN_ON_ONCE().
|
|
|
|
As Daniel Vacek noted, it is not necessary for the rcu_node structure
|
|
to have a CPU that has not yet passed through its quiescent state.
|
|
|
|
Tested-By: Guillaume Morin <guillaume@morinfr.org>
|
|
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
(Cherry pick from https://git.kernel.org/pub/scm/linux/kernel/git/paulmck/
|
|
linux-rcu.git/commit/?h=rcu/next&id=325a2030b90376d179a129794e2fae2b24d73923)
|
|
Signed-off-by: Jiping Ma <jiping.ma2@windriver.com>
|
|
---
|
|
kernel/rcu/tree.c | 7 ++++---
|
|
1 file changed, 4 insertions(+), 3 deletions(-)
|
|
|
|
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
|
|
index ef8d36f580fc..8706b30c2ac8 100644
|
|
--- a/kernel/rcu/tree.c
|
|
+++ b/kernel/rcu/tree.c
|
|
@@ -1590,10 +1590,11 @@ static void __maybe_unused rcu_advance_cbs_nowake(struct rcu_node *rnp,
|
|
struct rcu_data *rdp)
|
|
{
|
|
rcu_lockdep_assert_cblist_protected(rdp);
|
|
- if (!rcu_seq_state(rcu_seq_current(&rnp->gp_seq)) ||
|
|
- !raw_spin_trylock_rcu_node(rnp))
|
|
+ if (!rcu_seq_state(rcu_seq_current(&rnp->gp_seq)) || !raw_spin_trylock_rcu_node(rnp))
|
|
return;
|
|
- WARN_ON_ONCE(rcu_advance_cbs(rnp, rdp));
|
|
+ // The grace period cannot end while we hold the rcu_node lock.
|
|
+ if (rcu_seq_state(rcu_seq_current(&rnp->gp_seq)))
|
|
+ WARN_ON_ONCE(rcu_advance_cbs(rnp, rdp));
|
|
raw_spin_unlock_rcu_node(rnp);
|
|
}
|
|
|
|
--
|
|
2.31.1
|
|
|