The purpose of these symbols is for getting irq/softirq latency. Currently we only have symbols for irq_handler_entry and irq_handler_exit. We would use these latency for further monitoring and debugging.
The related tracepoints were defined in trace/events/irq.h. We would use these tracepoints in vendor kernel module.
Bug: 227809911
Signed-off-by: Ziyi Cui <ziyic@google.com>
Change-Id: Idf4ccdede5232689b2752283539aee54a5f67866
(cherry picked from commit e09aff6074e10262d503418e599f1a18fc047bd8)
Export irq_handle_exit tracepoint, so that vendor modules
can register probes for this tracepoint.
Bug: 205928005
Change-Id: I8e1eaffb7dd2f257e9c09412aad54ecca62bf019
Signed-off-by: Changki Kim <changki.kim@samsung.com>
Signed-off-by: Chanho Park <chanho61.park@samsung.com>
This reverts commit 2cadcbac1a, as the
merge conflicts have been resolved.
Bug: 168521633
Change-Id: Ib17f36be69a9112cc5ea2a5a2d940b82b480c8f0
Signed-off-by: Lingutla Chandrasekhar <clingutla@codeaurora.org>
[elavila: Port to mainline]
Signed-off-by: J. Avila <elavila@google.com>
Signed-off-by: Shaleen Agrawal <shalagra@codeaurora.org>
This reverts commit 4196c1dafc, as the
merge conflicts have been resolved.
Bug: 31752786
Bug: 168521633
Change-Id: I6cb3fc698d567e03c67e2c4373ce75cc71cdfe9c
Signed-off-by: John Dias <joaodias@google.com>
[elavila: Amend commit text for AOSP, port to mainline]
Signed-off-by: J. Avila <elavila@google.com>
[ashayj@codeaurora.org: update usage of __IRQ_STAT and minor conflicts]
Signed-off-by: Ashay Jaiswal <ashayj@codeaurora.org>
Signed-off-by: Shaleen Agrawal <shalagra@codeaurora.org>
Remove broken task->state references and let wake_up_process() DTRT.
The anti-pattern in these patches breaks the ordering of ->state vs
COND as described in the comment near set_current_state() and can lead
to missed wakeups:
(OoO load, observes RUNNING)<-.
for (;;) { |
t->state = UNINTERRUPTIBLE; |
smp_mb(); ,-----> | (observes !COND)
| /
if (COND) ---------' | COND = 1;
break; `- if (t->state != RUNNING)
wake_up_process(t); // not done
schedule(); // forever waiting
}
t->state = TASK_RUNNING;
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Davidlohr Bueso <dbueso@suse.de>
Acked-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Acked-by: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20210611082838.160855222@infradead.org
Pull RCU updates from Ingo Molnar:
- Support for "N" as alias for last bit in bitmap parsing library (eg
using syntax like "nohz_full=2-N")
- kvfree_rcu updates
- mm_dump_obj() updates. (One of these is to mm, but was suggested by
Andrew Morton.)
- RCU callback offloading update
- Polling RCU grace-period interfaces
- Realtime-related RCU updates
- Tasks-RCU updates
- Torture-test updates
- Torture-test scripting updates
- Miscellaneous fixes
* tag 'core-rcu-2021-04-28' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (77 commits)
rcutorture: Test start_poll_synchronize_rcu() and poll_state_synchronize_rcu()
rcu: Provide polling interfaces for Tiny RCU grace periods
torture: Fix kvm.sh --datestamp regex check
torture: Consolidate qemu-cmd duration editing into kvm-transform.sh
torture: Print proper vmlinux path for kvm-again.sh runs
torture: Make TORTURE_TRUST_MAKE available in kvm-again.sh environment
torture: Make kvm-transform.sh update jitter commands
torture: Add --duration argument to kvm-again.sh
torture: Add kvm-again.sh to rerun a previous torture-test
torture: Create a "batches" file for build reuse
torture: De-capitalize TORTURE_SUITE
torture: Make upper-case-only no-dot no-slash scenario names official
torture: Rename SRCU-t and SRCU-u to avoid lowercase characters
torture: Remove no-mpstat error message
torture: Record kvm-test-1-run.sh and kvm-test-1-run-qemu.sh PIDs
torture: Record jitter start/stop commands
torture: Extract kvm-test-1-run-qemu.sh from kvm-test-1-run.sh
torture: Record TORTURE_KCONFIG_GDB_ARG in qemu-cmd
torture: Abstract jitter.sh start/stop into scripts
rcu: Provide polling interfaces for Tree RCU grace periods
...
On RT a task which has soft interrupts disabled can block on a lock and
schedule out to idle while soft interrupts are pending. This triggers the
warning in the NOHZ idle code which complains about going idle with pending
soft interrupts. But as the task is blocked soft interrupt processing is
temporarily blocked as well which means that such a warning is a false
positive.
To prevent that check the per CPU state which indicates that a scheduled
out task has soft interrupts disabled.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Tested-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Tested-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://lore.kernel.org/r/20210309085727.527563866@linutronix.de
tasklet_unlock_spin_wait() spin waits for the TASKLET_STATE_SCHED bit in
the tasklet state to be cleared. This works on !RT nicely because the
corresponding execution can only happen on a different CPU.
On RT softirq processing is preemptible, therefore a task preempting the
softirq processing thread can spin forever.
Prevent this by invoking local_bh_disable()/enable() inside the loop. In
case that the softirq processing thread was preempted by the current task,
current will block on the local lock which yields the CPU to the preempted
softirq processing thread. If the tasklet is processed on a different CPU
then the local_bh_disable()/enable() pair is just a waste of processor
cycles.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Tested-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://lore.kernel.org/r/20210309084241.988908275@linutronix.de
tasklet_kill() spin waits for TASKLET_STATE_SCHED to be cleared invoking
yield() from inside the loop. yield() is an ill defined mechanism and the
result might still be wasting CPU cycles in a tight loop which is
especially painful in a guest when the CPU running the tasklet is scheduled
out.
tasklet_kill() is used in teardown paths and not performance critical at
all. Replace the spin wait with wait_var_event().
Signed-off-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://lore.kernel.org/r/20210309084241.890532921@linutronix.de
tasklet_unlock_wait() spin waits for TASKLET_STATE_RUN to be cleared. This
is wasting CPU cycles in a tight loop which is especially painful in a
guest when the CPU running the tasklet is scheduled out.
tasklet_unlock_wait() is invoked from tasklet_kill() which is used in
teardown paths and not performance critical at all. Replace the spin wait
with wait_var_event().
There are no users of tasklet_unlock_wait() which are invoked from atomic
contexts. The usage in tasklet_disable() has been replaced temporarily with
the spin waiting variant until the atomic users are fixed up and will be
converted to the sleep wait variant later.
Signed-off-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://lore.kernel.org/r/20210309084241.783936921@linutronix.de
If there is heavy softirq activity, the softirq system will attempt
to awaken ksoftirqd and will stop the traditional back-of-interrupt
softirq processing. This is all well and good, but only if the
ksoftirqd kthreads already exist, which is not the case during early
boot, in which case the system hangs.
One reproducer is as follows:
tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration 2 --configs "TREE03" --kconfig "CONFIG_DEBUG_LOCK_ALLOC=y CONFIG_PROVE_LOCKING=y CONFIG_NO_HZ_IDLE=y CONFIG_HZ_PERIODIC=n" --bootargs "threadirqs=1" --trust-make
This commit therefore adds a couple of existence checks for ksoftirqd
and forces back-of-interrupt softirq processing when ksoftirqd does not
yet exist. With this change, the above test passes.
Reported-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Reported-by: Uladzislau Rezki <urezki@gmail.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
[ paulmck: Remove unneeded check per Sebastian Siewior feedback. ]
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
To avoid include recursion hell move the do_softirq_own_stack() related
content into a generic asm header and include it from all places in arch/
which need the prototype.
This allows architectures to provide an inline implementation of
do_softirq_own_stack() without introducing a lot of #ifdeffery all over the
place.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
Link: https://lore.kernel.org/r/20210210002513.289960691@linutronix.de
This reverts commit 8d19443b0b as the
softirq code is rewritten in 5.11-rc1 and massive merge conflicts are
happening.
If this change is still needed, please work with upstream to get the
patches accepted so they can then come into this tree automatically.
Bug: 31752786
Bug: 168521633
Cc: John Dias <joaodias@google.com>
Cc: J. Avila <elavila@google.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
Change-Id: I6875407b586f505c2045e4cf40682831b4fceac1
This reverts commit 0578248bed as the
softirq code is rewritten in 5.11-rc1 and massive merge conflicts are
happening.
If this change is still needed, please work with upstream to get the
patches accepted so they can then come into this tree automatically.
Bug: 168521633
Cc: Pavankumar Kondeti <pkondeti@codeaurora.org>
Cc: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
Cc: J. Avila <elavila@google.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
Change-Id: I421cc2ca29b635d7ac39d52aa486e20cbdd8ca45
This reverts commit ea05d25c78 as the
softirq code is rewritten in 5.11-rc1 and massive merge conflicts are
happening.
If this change is still needed, please work with upstream to get the
patches accepted so they can then come into this tree automatically.
Bug: 168521633
Cc: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
Cc: J. Avila <elavila@google.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
Change-Id: I4ab5adcf06824b0b7d052ea07e8c851c9052fe6b
This reverts commit 1bcefd3883 as the
softirq code is rewritten in 5.11-rc1 and massive merge conflicts are
happening.
If this change is still needed, please work with upstream to get the
patches accepted so they can then come into this tree automatically.
Bug: 168521633
Cc: Lingutla Chandrasekhar <clingutla@codeaurora.org>
Cc: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
Cc: J. Avila <elavila@google.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
Change-Id: Idb615a68b4e892b1b0391a8511c9338f6202368b
This reverts commit 9889f08de4 as the
softirq code is rewritten in 5.11-rc1 and massive merge conflicts are
happening.
If this change is still needed, please work with upstream to get the
patches accepted so they can then come into this tree automatically.
Bug: 168521633
Cc: Lingutla Chandrasekhar <clingutla@codeaurora.org>
Cc: J. Avila <elavila@google.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
Change-Id: Ibd116aef3140df9a896415e521ef4039c74b4819
Export irq_handle_entry tracepoint, so that vendor modules
can register probes for this tracepoint.
Bug: 175806230
Change-Id: Ieb276e1938ccf819a847cde37d5a25bfa00efe05
Signed-off-by: Prasad Sodagudi <psodagud@codeaurora.org>
Now that account_hardirq_enter() is called after HARDIRQ_OFFSET has
been incremented, there is nothing left that prevents us from also
moving tick_irq_enter() after HARDIRQ_OFFSET is incremented.
The desired outcome is to remove the nasty hack that prevents softirqs
from being raised through ksoftirqd instead of the hardirq bottom half.
Also tick_irq_enter() then becomes appropriately covered by lockdep.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Link: https://lore.kernel.org/r/20201202115732.27827-6-frederic@kernel.org
IRQ time entry is currently accounted before HARDIRQ_OFFSET or
SOFTIRQ_OFFSET are incremented. This is convenient to decide to which
index the cputime to account is dispatched.
Unfortunately it prevents tick_irq_enter() from being called under
HARDIRQ_OFFSET because tick_irq_enter() has to be called before the IRQ
entry accounting due to the necessary clock catch up. As a result we
don't benefit from appropriate lockdep coverage on tick_irq_enter().
To prepare for fixing this, move the IRQ entry cputime accounting after
the preempt offset is incremented. This requires the cputime dispatch
code to handle the extra offset.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://lore.kernel.org/r/20201202115732.27827-5-frederic@kernel.org
Tasklets are supposed to finish their work quickly and
should not block the current running process, but it is not
guaranteed that. Currently softirq_entry/exit can be used to
know total tasklets execution time, but not helpful to track
individual tasklet's execution time. With that we can't find
any culprit tasklet function, which is taking more time.
Add {hi}-tasklet_entry/exit trace point support to track
individual tasklet execution.
Bug: 168521633
Change-Id: I3496d15f64d020916774e673ccb4a8116ea2f2c9
Signed-off-by: Lingutla Chandrasekhar <clingutla@codeaurora.org>
[elavila: Port to mainline]
Signed-off-by: J. Avila <elavila@google.com>
Ksfotirqd is a normal priority CFS task. It can experience higher
scheduling latency under heavy load conditions. Currently once
asynchronous softirq processing is deferred to ksoftirqd, softirqs
are not processed further until ksoftirqd task gets a chance to run.
High latencies for softirqs like TIMER, HI TASKLET is not acceptable.
So revert 'commit 4cd13c21b2 ("softirq: Let ksoftirqd do its job")'.
Bug: 168521633
Change-Id: I38a1a88b5f42dd534c65d739dbb7e4321a7904db
Signed-off-by: Lingutla Chandrasekhar <clingutla@codeaurora.org>
[satyap@codeaurora.org: Fix trivial merge conflicts]
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
[elavila: Port to mainline]
Signed-off-by: J. Avila <elavila@google.com>
This reverts commit 3c53776e29 because it
makes HI_SOFTIRQ and TASKLET_SOFTIRQ run immediately i.e. not get
deferred to ksfotirqd. The commit text calls out that this is a stopgap
until a better solution is deviced and that it should have included
TIMER_SOFTIRQ as well, but chose not to.
Patch ebc7b75d772647813cdaf085c7f5adf3c90b033b from Qualcomm is a more
comprehensive solution which defers long running tasklets to ksfotirqd
when a rt task is interrupted.
We cannot use both these together as they conflict; e.g. the original
makes TASKLET softirq run immediately while the Qualcomm one defers it
if rt gets interrupted. Choose to use the Qualcomm one.
Bug: 168521633
Change-Id: I4af64cd7e2c4291dda5f503bf2d74ede459a76c6
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
[elavila: port to mainline, resolve conflicts, add commit text]
Signed-off-by: J. Avila <elavila@google.com>
Defer the softirq processing to ksoftirqd if a RT task is running
or queued on the current CPU. This complements the RT task placement
algorithm which tries to find a CPU that is not currently busy with
softirqs.
Currently NET_TX, NET_RX, BLOCK and TASKLET softirqs are only deferred
as they can potentially run for long time.
Bug: 168521633
Change-Id: Id7665244af6bbd5a96d9e591cf26154e9eaa860c
Signed-off-by: Pavankumar Kondeti <pkondeti@codeaurora.org>
[satyap@codeaurora.org: trivial merge conflict resolution.]
Signed-off-by: Satya Durga Srinivasu Prabhala <satyap@codeaurora.org>
[elavila: Port to mainline, squash with bugfix]
Signed-off-by: J. Avila <elavila@google.com>
The scheduling change to avoid putting RT threads on cores that
are handling softint's was catching cases where there was no reason
to believe the softint would take a long time, resulting in unnecessary
migration overhead. This patch reduces the migration to cases where
the core has a softint that is actually likely to take a long time,
as opposed to the RCU, SCHED, and TIMER softints that are rather quick.
Bug: 31752786
Bug: 168521633
Change-Id: Ib4e179f1e15c736b2fdba31070494e357e9fbbe2
Signed-off-by: John Dias <joaodias@google.com>
[elavila: Amend commit text for AOSP, port to mainline]
Signed-off-by: J. Avila <elavila@google.com>
__raise_softirq_irqoff() must be called with interrupts disabled to protect
the per CPU softirq pending state update against an interrupt and soft
interrupt handling on return from interrupt.
Add a lockdep assertion to validate the calling convention.
[ tglx: Massaged changelog ]
Signed-off-by: Jiafei Pan <Jiafei.Pan@nxp.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Link: https://lore.kernel.org/r/20200814045522.45719-1-Jiafei.Pan@nxp.com
Pull tasklets API update from Kees Cook:
"These are the infrastructure updates needed to support converting the
tasklet API to something more modern (and hopefully for removal
further down the road).
There is a 300-patch series waiting in the wings to get set out to
subsystem maintainers, but these changes need to be present in the
kernel first. Since this has some treewide changes, I carried this
series for -next instead of paining Thomas with it in -tip, but it's
got his Ack.
This is similar to the timer_struct modernization from a while back,
but not nearly as messy (I hope). :)
- Prepare for tasklet API modernization (Romain Perier, Allen Pais,
Kees Cook)"
* tag 'tasklets-v5.9-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/kees/linux:
tasklet: Introduce new initialization API
treewide: Replace DECLARE_TASKLET() with DECLARE_TASKLET_OLD()
usb: gadget: udc: Avoid tasklet passing a global
Nowadays, modern kernel subsystems that use callbacks pass the data
structure associated with a given callback as argument to the callback.
The tasklet subsystem remains one which passes an arbitrary unsigned
long to the callback function. This has several problems:
- This keeps an extra field for storing the argument in each tasklet
data structure, it bloats the tasklet_struct structure with a redundant
.data field
- No type checking can be performed on this argument. Instead of
using container_of() like other callback subsystems, it forces callbacks
to do explicit type cast of the unsigned long argument into the required
object type.
- Buffer overflows can overwrite the .func and the .data field, so
an attacker can easily overwrite the function and its first argument
to whatever it wants.
Add a new tasklet initialization API, via DECLARE_TASKLET() and
tasklet_setup(), which will replace the existing ones.
This work is greatly inspired by the timer_struct conversion series,
see commit e99e88a9d2 ("treewide: setup_timer() -> timer_setup()")
To avoid problems with both -Wcast-function-type (which is enabled in
the kernel via -Wextra is several subsystems), and with mismatched
function prototypes when build with Control Flow Integrity enabled,
this adds the "use_callback" member to let the tasklet caller choose
which union member to call through. Once all old API uses are removed,
this and the .data member will be removed as well. (On 64-bit this does
not grow the struct size as the new member fills the hole after atomic_t,
which is also "int" sized.)
Signed-off-by: Romain Perier <romain.perier@gmail.com>
Co-developed-by: Allen Pais <allen.lkml@gmail.com>
Signed-off-by: Allen Pais <allen.lkml@gmail.com>
Reviewed-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Acked-by: Thomas Gleixner <tglx@linutronix.de>
Co-developed-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Kees Cook <keescook@chromium.org>
Currently all IRQ-tracking state is in task_struct, this means that
task_struct needs to be defined before we use it.
Especially for lockdep_assert_irq*() this can lead to header-hell.
Move the hardirq state into per-cpu variables to avoid the task_struct
dependency.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Ingo Molnar <mingo@kernel.org>
Link: https://lkml.kernel.org/r/20200623083721.512673481@infradead.org
