[原创] chrt: failed to set pid xxxx's policy: Operation not permitted
KEYWORDS: scheduler, scheduling, sysctl, sched_fifo, sched_rr, chrt, deadline, sched_deadline
HISTORY:
- Created at 09:50:43 on 2019-03-20.
chrt: failed to set pid xxxx's policy: Operation not permitted
First reported on physical machine (Dell Inc. OptiPlex 790) with CentOS 7.6.
kernel-3.10.0-957.5.1.el7.x86_64.util-linux-2.23.2-59.el7.x86_64.
# cat /proc/sys/kernel/sched_rt_runtime_us
950000
# ulimit -a | grep priority
scheduling priority (-e) 0
real-time priority (-r) 0
# sleep 300 &
[1] 9222
# chrt -f -p 20 9222
chrt: failed to set pid 9222's policy: Operation not permitted
# chrt -f 20 whoami
chrt: failed to set pid 0's policy: Operation not permitted
Analysis
According to [1] (see below), after committing following command, chrt works.
# sysctl -w kernel.sched_rt_runtime_us=-1
kernel.sched_rt_runtime_us = -1
# chrt -f 20 whoami
root
/usr/share/doc/kernel-doc-3.10.0/Documentation/scheduler/sched-rt-group.txt
A run time of -1 specifies runtime == period, ie. no limit.
For more information about sched_rt_runtime_us, refer to Terminology/scheduling_policies.md.
But this is not a issue on virtual machine with the same packages version, physical host or virtual machine with CentOS 6.x.
- Is it a bug?
- Does this issue relate to control groups?
- There must be something different from physical machine and virtual machine. But what are they?
Digging Deeper
-
On Physical Machine with CentOS 7.x (
kernel-3.10.0-957.5.1.el7.x86_64andutil-linux-2.23.2-59.el7.x86_64)# sysctl -a | grep sched_rt_runtime_us kernel.sched_rt_runtime_us = 950000 # strace -fe sched_setattr,sched_setscheduler chrt -f 10 whoami sched_setattr(0, {size=48, sched_policy=SCHED_FIFO, sched_flags=0, sched_nice=0, sched_priority=10, sched_runtime=0, sched_deadline=0, sched_period=0}, 0) = -1 EPERM (Operation not permitted) chrt: failed to set pid 0's policy: Operation not permitted +++ exited with 1 +++ # sysctl -w kernel.sched_rt_runtime_us=-1 kernel.sched_rt_runtime_us = -1 # strace -fe sched_setattr,sched_setscheduler chrt -f 10 whoami sched_setattr(0, {size=48, sched_policy=SCHED_FIFO, sched_flags=0, sched_nice=0, sched_priority=10, sched_runtime=0, sched_deadline=0, sched_period=0}, 0) = 0 root +++ exited with 0 +++
man 2 sched_setattr(On “Mint 19.1”, there is nosched_setattrman page can be found on CentOS 7.x)EPERM. The caller does not have appropriate privileges.EPERM. The CPU affinity mask of the thread specified by pid does not include all CPUs in the system (seesched_setaffinity(2)).
These system calls first appeared in Linux 3.14.
-
On Virtual Machine with CentOS 7.x (
kernel.sched_rt_runtime_us = 950000)Different results with different package versions.
-
kernel-3.10.0-862.6.3.el7.x86_64andutil-linux-2.23.2-52.el7.x86_64# strace -fe sched_setattr,sched_setscheduler chrt -f 10 whoami sched_setattr(0, {size=48, sched_policy=SCHED_FIFO, sched_flags=0, sched_nice=0, sched_priority=10, sched_runtime=0, sched_deadline=0, sched_period=0}, 0) = -1 ENOSYS (Function not implemented) sched_setscheduler(0, SCHED_FIFO, [10]) = 0 root +++ exited with 0 +++ -
kernel-3.10.0-957.5.1.el7.x86_64andutil-linux-2.23.2-59.el7.x86_64# strace -fe sched_setattr,sched_setscheduler chrt -f 10 whoami sched_setattr(0, {size=48, sched_policy=SCHED_FIFO, sched_flags=0, sched_nice=0, sched_priority=10, sched_runtime=0, sched_deadline=0, sched_period=0}, 0) = 0 root +++ exited with 0 +++
-
-
On Virtual Machine with CentOS 6.x (
kernel-2.6.32-696.20.1.el6.x86_64andutil-linux-ng-2.17.2-12.28.el6.x86_64,kernel.sched_rt_runtime_us = 950000)# strace -fe sched_setattr,sched_setscheduler chrt -f 10 whoami strace: invalid system call `sched_setattr' # strace -fe sched_setscheduler chrt -f 10 whoami sched_setscheduler(0, SCHED_FIFO, { 10 }) = 0 root -
On Physical Machine with Mint 19.1 (
linux-image-4.15.0-45-genericandutil-linux-2.31.1-0.4ubuntu3.3) or Ubuntu 16.04 (linux-image-4.15.0-39-genericandutil-linux-2.27.1-6ubuntu3.2)# strace -fe sched_setattr,sched_setscheduler chrt -f 10 whoami sched_setscheduler(0, SCHED_FIFO, [10]) = 0 root +++ exited with 0 +++
I guess the key of this issue is sched_setattr. And it’s probably a bug, see below [2].
It is a bug. Relative issue you can find on URL EPERM running chrt from shell with positive niceness #359.
Though no one mentioned about SCHED_FIFO, and in my example I was the root user.
As the developer said “Well, both syscalls have priority as argument and it should be zero for SCHED_BATCH. It’s strange and inconsistent API that sched_setattr() interprets zero priority as attempt to change task priority. It makes the syscall useless.”
So, I guess this problem is related to two parameters sched_runtime (as kernel.sched_rt_runtime_us=-1 makes chrt work well and this parameter is used by SCHED_DEADLINE too [4]) and sched_priority of sched_setattr().
Following examples may prove that.
# sleep 3000 &
[1] 7000
# sysctl -w kernel.sched_rt_runtime_us=-1
kernel.sched_rt_runtime_us = -1
# strace -fe sched_setattr,sched_setscheduler chrt --verbose -f -p 1 7000
pid 7000's current scheduling policy: SCHED_OTHER
pid 7000's current scheduling priority: 0
sched_setattr(7000, {size=48, sched_policy=SCHED_FIFO, sched_flags=0, sched_nice=0, sched_priority=1, sched_runtime=0, sched_deadline=0, sched_period=0}, 0) = 0
pid 7000's new scheduling policy: SCHED_FIFO
pid 7000's new scheduling priority: 1
+++ exited with 0 +++
# sysctl -w kernel.sched_rt_runtime_us=950000
kernel.sched_rt_runtime_us = 950000
# chrt -f -p 2 7000
chrt: failed to set pid 7000's policy: Operation not permitted
# renice -n 0 -p 7000
7000 (process ID) old priority -2, new priority 0
# renice -n 2 -p 7000
7000 (process ID) old priority 0, new priority 2
# renice -n -2 -p 7000
7000 (process ID) old priority 2, new priority -2
# strace -fe sched_setattr,sched_setscheduler chrt -f -p 2 7000
sched_setattr(7000, {size=48, sched_policy=SCHED_FIFO, sched_flags=0, sched_nice=0, sched_priority=2, sched_runtime=0, sched_deadline=0, sched_period=0}, 0) = -1 EPERM (Operation not permitted)
chrt: failed to set pid 7000's policy: Operation not permitted
+++ exited with 1 +++
Remained Questions
But by now, I still have no idea about why chrt works well on virtual machine.
Intermediate Measure
You should exactly know what you are doing before using it.
# On the Fly
sysctl -w kernel.sched_rt_runtime_us=-1
# For Persistent
echo "# Added by dreamer_catcher on 2019-03-20"
echo "kernel.sched_rt_runtime_us = -1" >> /etc/sysctl.conf
sysctl -p
Solution (Not Tested)
You may need to download the source code (latest released) of util-linux and compile it manually.
References
- [1]. FIFI Policy.
- [2]. chrt from shell scripts: operation not permitted.
- [3]. EPERM running chrt from shell with positive niceness #359.
- [4]. Deadline Task Scheduling.