Linux Kernel security mitigations protect systems from hardware security vulnerabilities, but they often come at the cost of performance. In this article, I will note possible performance impact of kernel mitigations, particularly in the context of Docker.

Docker

Docker/containeralization is extremely useful for deployment orchestrations by packaging applications into simple container images that utilize linux kernel namespaces to achieve sandboxing rather than virtualization. However, in the low-latency world, anything that can impact the performance of an application can be unacceptable.

Theoretically the overhead of containeralization even in low-latency applications should be negligible as there’s no virtualization or heavy abstractions running around the application to achieve containerization. However, does this pan out in practice? We’ll see.

The problem

Recently, while evaluating the performance of a highly CPU bound workload, I noticed a significant overhead when running the application on a container vs. directly on the host.

As usual, I’ll use my orderbook as a test bed to evaluate the impact for the purpose of this post.

Tests running using current main branch at b4b7c7c

Benchmark results for AddOrder;

bench -duration=30

It looks like latency and throughput are both down significantly. So what happened?

Turns out, this has to do with Kernel security mitigations.

To check what mitigations have been enabled on your host:

1
2
3
4
5

$ lscpu | grep Vulnerability 
Vulnerability Retbleed:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence

It turns out in this case the biggest culprit here is the Spec store bypass. By default, this is set to:

1

spec_store_bypass_disable=auto

Docker uses kernel security features such as seccomp to sandbox containers and limit their access to system resources. The seccomp syscall, which Docker uses by default, seems to enable the Speculative Store Bypass and branch prediction mitigations unless they were explicitly disabled at the host level.

Running the same test with nospec_store_bypass_disable gives the following results inside docker:

The results are similar with running docker with --security-opt seccomp=unconfined

Results

Looking at the above, it looks like the performance (both latency and throughput) on bare metal and with Docker is comparable as long as SSB is disabled. With SSB optional, docker’s seccomp syscall results in spec store bypass enabled, in turn causing a degradation in both latency and throughput.

This is because according to the seccomp documentation, seccomp includes a flag called SECCOMP_FILTER_FLAG_SPEC_ALLOW. This flag allows a process to opt-out of the Speculative Store Bypass mitigation, which unless set enables it.

That said, SSB is not the only mitigation that causes performance impact.

In another workload I was working on, SSB and spectre_v2 were both responsible for degradation in latency, and retbleed mitigation was responsible for adding a significant amount of jitter.

The impact of these mitigations will also change with the CPU, kernel version being used and whether you have hyperthreading enabled. If the performance impact of these mitigations is a point of concern, the prudent thing to do is test, test, and test some more to get the appropriate data for your specific workload, CPU, and kernel version combinations. It is essential to be aware of these potential impacts and consider the trade-offs when deciding whether to enable or disable specific mitigations based on your application’s latency requirements, threat models, and risk profile.

Possible Solutions

The best solution is to identify and disable the relevant security mitigations explicitly so that there is no way to accidentally enable them.

Recent updates in runc and the Linux kernel have introduced changes that may help reduce the performance impact of certain kernel security mitigations. For example, runc added support for SECCOMP_FILTER_FLAG_SPEC_ALLOW in #3390 and enabled it by default in PR #3588. This allows seccomp syscall by runc to allow speculation.

Additionally, in Linux kernel version 5.16, the default behavior was changed to disable these mitigations for seccomp (source). This change further reduces the performance impact of kernel security mitigations when using seccomp, and consequently docker containers.

Disclaimer

These mitigations are built into the kernel for a reason. Consider a variety of factors such as your threat model, risk profile and the actual performance tradeoffs for your specific use case before deciding to risk accept and disable any relevant security mitigations.