42% of vulnerabilities exploited after patch already released - Live patching helps keep you secure
28 Aug 2024, by Slade Baylis
When it comes to keeping your systems secure, deploying patches as soon as possible after they’re available should be at the top of your priorities. Whilst all software updates are important - as they can contain bug fixes, new features, optimisations and more - patches are different in that they are usually smaller and aim to just fix security vulnerabilities as they are discovered.
As with most things in the cybersecurity space, keeping your systems secure is an ever-lasting arms race against those who would seek to break in. This being the case, the speed at which these updates are deployed is a critical factor in determining how good your security posture is. With some updates this is relatively easy, just requiring you to apply them without any disruption to your services. However, some others, such as kernel updates, usually require systems to be restarted for the patches to be applied.
This can be a problem for production systems, as in most cases, any systems running production workloads have limited windows of time within which downtime is acceptable. This results in a difficult choice between either applying a necessary security patch and accepting the downtime, or alternatively keeping systems online but remaining vulnerable until your next allowable outage window. With reports like the FireEye Mandiant Threat Intelligence Report1 back in 2018 and 2019 showing that the exploitation of 42% of attacks occurred after a patch was already released, it’s vitally important to apply patches as quickly as possible.
That’s why this month we’re talking about KernelCare, which is a service that provides the ability to apply security patches and bug fixes to Linux kernels without requiring any reboot or downtime.
What are kernels?
Before we can understand why kernels are more difficult to update in real-time than other pieces of software, we’ll need to cover what they are and what function they perform.
In simple terms, a “kernel” is the core of any operating system, such as Microsoft Windows or the various distributions of Linux. The kernel is used to handle all interactions between software and the underlying hardware that software is running on. Whenever a piece of software needs to use any hardware resources - such as CPUs, RAM, disks, etc – the kernel and device drivers provide an abstraction layer allowing for those resources to be used. They also manage the allocation of these resources to different processes, as well as resolve any conflicts between different processes requesting the same resources at the same time.
As you can imagine, with the kernel having direct access to the underlying system hardware, it’s critical for this to remain secure. If it was to be compromised, not only would it allow for all types of almost undetectable data breaches, but it would also allow bad actors to easily damage the hardware itself if desired. Due to this, the kernel is usually protected in many ways. Some of the ways include running the kernel entirely segregated from any applications run by the user, as well as running it entirely out of separate areas of the system’s memory that are inaccessible to unauthorised software and less critical parts of the operating system.
Due to the fact that all other software and parts of the operating system rely on the kernel to function, it’s extremely difficult to update whilst the system is still running. Given the kernel is continually being used by the system in some form or another, the question then arises as to how is it possible to upgrade it without disrupting those other systems? Normally it isn’t, but the good news is that elegant solutions to this problem have been created in the form of various kernel live patching services.
How does live patching work?
Regardless of your choice of kernel live patching service, the fundamental approach is similar. In each case, the provider will keep an eye for any new kernel updates that are released, then set to work on creating a “live patch” version of those updates to address the associated vulnerability. After the live patch has been created, these are then made available for clients to deploy onto their own systems.
For deploying these patches, there are usually several main components that are involved2 which include:
- Patch Servers
A “patch server” is used to store all the various patches that are available for each kernel version. This can either be an external server or an internally-managed server that only holds pre-approved patches to be applied. - Agent Programs
The “agent” is a piece of software that runs on the server in the background, periodically checking in with the patch server to see if any new patches are available that need to be applied. - Kernel Modules
The “kernel module” is the actual software that handles the patching of the kernel, performing functions such as waiting for processes using parts of the kernel to finish before replacing them, pausing and restarting kernel processes as necessary, and setting up relocations to the new kernel code.
With these providers allowing “live patches” of kernels to be applied, users of these servers are able to get the best of both worlds - they remain secure against newly discovered vulnerabilities whilst also reducing the amount of downtime that would otherwise be required to update their systems.
In our article from last month, where we touched on the updates from CrowdStrike that took many critical services around the world offline, we mentioned how automated updates are something that should be avoided, and that proper change management (CM) is something that IT providers should put in place instead. That’s still the case and is our recommendation with kernel live-update services also. However, it should be noted, that with these kernel updates being vetted by their original developers, as well as the live-update providers who adapt them, it’s less likely that similar issues will escape the quality control processes they have in place.
KernelCare – Our recommendation for Linux kernel live patching
Until recently, live kernel updates for Windows servers weren’t available at all, and even now it’s restricted to Azure users. Currently, any security patches to the kernel for most Windows users are applied just like any other update, being contained within the regular OS updates that require downtime of your systems to apply. Not only that, for most of our clients the operating system of choice is some flavour of Linux, and it’s for this reason that in this article we’ve primarily been talking about it in relation to the Linux kernel specifically.
Of the different services that are available for the different Linux distributions, our own choice and recommendation for our clients is KernelCare. One of the reasons is that it was developed by CloudLinux, the same people who developed the popular operating system for shared web hosting (SWH) services by the same name. CloudLinux have also developed other popular and industry leading systems, such as the security software suite for web hosting servers called Imunify360, as well as the replacement for the previously available and widely used Centos operating system in the form of AlmaLinux.
It’s due to this backing and expertise that we’re happy to have them be our 'go to' recommendation. As stated on their website, they have over 500+ combined years of Linux expertise across their staff! Our recommendation is also based on the fact that KernalCare is available for a large amount of the most popular Linux distributions, such as AlmaLinux, CentOS, Debian, Oracle, Proxmox, RHEL, Ubuntu, and of course their own CloudLinux OS.
Want to know more about KernelCare?
If you have more questions about KernelCare, let us know! We’re happy to let you know what’s involved to get it deployed onto your systems and help improve your security posture.
You can reach out to us via email at sales@micron21.com or via phone on 1300 769 972.
Sources
1, FireEye Mandiant Threat Intelligence, “Trends in time to exploit”, <https://jp.stage.mandiant.com/resources/blog/time-between-disclosure-patch-release-and-vulnerability-exploitation>
2, KernelCare, “Three Ways To Patch a Linux Kernel”, <https://medium.com/@KernelCare/three-ways-to-patch-a-linux-kernel-f3e6acd80737>
