Balance CPU utilization between services using Cgroup
On 2020/05/17 at 12:23
I have an application which has two CPU intensive components which I call them as service A and service B. Service A will spawn a new process to deal with a new request. However service B only has fixed processes to deal with the workload. So when service A is under high load, it will spawn lots of cpu-intensive processes to deal with the requests. This behavior will cause service A to consume more than 90% of total CPU resource which will result service B to be very slow and unresponsive.
To deal with this issue, I would like to distribute the CPU resouce equally between the two services. Using CPU Cgroup to fix the situation comes to my head.
Cgroup uses a tree hierarchy to define how many resource its children can use under the root. For example, if we construct a CPU cgroup like this:
test / \ t1 t2
Each group (test, t1 and t2) has 1024 (default value) cpu share. If t1 and t2 both contain cpu-intensive processes, no matter how many processces are under t1, the maximum cpu utilization t1 can get is
(1024 / (1024 + 1024)) which is
cgcreate -g cpu:test cgcreate -g cpu:test/t1 cgcreate -g cpu:test/t2 cgexec -g cpuset:test -g cpu:test/t1 ./busy11 & # 25% cpu cgexec -g cpuset:test -g cpu:test/t1 ./busy12 & # 25% cpu cgexec -g cpuset:test -g cpu:test/t2 ./busy2 & # 50% cpu
An interesting fact to know is we can also run processes under to test group which is the root of t1 and t2. If we run a process under the test group, it is the same as creating an individual subgroup which has the same cpu share as root group's setting under the test group and run the process in it.
cgexec -g cpuset:test -g cpu:test ./busy01 & # 25% cpu cgexec -g cpuset:test -g cpu:test ./busy02 & # 25% cpu cgexec -g cpuset:test -g cpu:test/t1 ./busy11 & # 12.5% cpu cgexec -g cpuset:test -g cpu:test/t1 ./busy12 & # 12.5% cpu cgexec -g cpuset:test -g cpu:test/t2 ./busy2 & # 50% cpu
The calculation behavior explaination can be found at PROCESS BEHAVIOR IN THE ROOT CONTROL GROUP.