What’s all the fuss about Ksplice? – Oracle Linux 8 – Patching Doesn’t Have to Mean Rebooting

What’s all the fuss about Ksplice?

The first question is, Why is this important to IT, and more importantly, to defending the applications running on the servers? Before we get to that, you first need to understand the process that hackers use to compromise systems.

The basic process includes three main stages: reconnaissance, gaining access, and then maintaining access. Once a system is compromised, the process is repeated on other systems on the network:

Figure 7.1 – Security life cycle

In the first phase, reconnaissance, hackers use a variety of methods to scan the system for vulnerabilities. Often, this attacking phase against servers starts after hackers have gained access to a low-level support account that has access to the server. Once there, it is a trivial task to look at the kernel version and identify what vulnerabilities the kernel is susceptible to. Oracle offers a free site (https://ksplice.oracle.com/inspector) that lets you easily check the vulnerabilities of the kernel. The results of such a check are shown in Figure 7.2:

Figure 7.2 – Ksplice Inspector

Note

The MITRE Corporation is a non-profit company funded by the US federal government that researches ways to improve safety across multiple industries, including aviation, healthcare, homeland security, defense, and cybersecurity. The Common Vulnerabilities and Exposures (CVE) program was launched in 1999 and quickly became the go-to location to track cybersecurity vulnerabilities. More information about the CVE program can be found at https://cve.org.

Not only will the system identify the CVE that the kernel is susceptible to but it also links to the MITRE site with the CVE details. In the example report, you can see the kernel is susceptible to several CVEs, including a few that would allow the attacker to gain root access. Surprisingly, unlike other similar technologies, Ksplice also helps defend against this phase with its ability to detect attacks against the running kernel, seeking to gain privileged escalation using the Known Exploit Detection technology. This effectively turns every system using Ksplice into a free intrusion detection system for the kernel.

In the next phase, gaining access, the hacker exploits the vulnerability, gaining access to the data. This is often done weeks after the initial discovery phase. Hackers take their time; they do this to run under the radar, and they know that most environments are only patched on a quarterly basis at best. Additionally, many attacks come from inside the organization, from people who have limited access to systems as part of their normal role. Ksplice disrupts this pattern, enabling the system admin to easily patch daily, closing the window of opportunity for the hacker.

In the last phase, the hackers work to maintain access, often using the same vulnerability to continually mine the system for data. Ksplice allows this to also be shut down, as it closes the vulnerability not only for exploits post-patch but also shuts down the exploit for the attack while in progress. This is an important feature of Ksplice: the ability to patch the kernel and close the exploit while the exploit is being used. This shuts down the exploit mid-flight, significantly enhancing the system’s security.

Now, about that patching effort – once a quarter at best? Kernel patches are released almost daily, yet systems can remain unpatched for almost a quarter of the year or more. This gives attackers a huge window of opportunity to compromise your systems. Not only is scheduling the patch a challenge for most mission-critical systems but the patch process itself is very time- and labor-intensive. Let’s look at why this is the case. As seen in the following diagram, there are many steps required to patch a server:

Figure 7.3 – Legacy patch process

The following list provides a short description of each of these steps:

1. Block users from accessing the application: Disable load balancers or disable user logins to the application so users do not accidentally corrupt data.
2. Shut down the application: Shut down the databases, application servers, schedulers, load balancers, and so on. This is to prepare for the operating system reboot.
3. Sanity reboot the servers: Perform a reboot of the servers, with no other changes made. This is to verify that the servers can perform a clean reboot. Often in this step, a snapshot is made of the app servers.
4. Restart the application: Start the application back up to verify that it is working. This is to verify the sanity reboot.
5. Test post sanity reboot: Often, some changes get made to the application and/or its infrastructure that can cause issues with the application. This step verifies that there is a good system before the patches are applied.
6. Shut down the application: Shut everything down again to prepare for the patching.
7. Patch the servers: Apply the operating system patches.
8. Reboot post patch: Reboot so the patches take effect.
9. Restart the application: Restart the application again.
10. Test the application: Test the application, and verify all is good.
11. Let users back in: Allow users back into the application.

Note

Not every organization follows the full path process, skipping steps to save some time. While this can save some time in the short term, all it takes is one bad patch cycle where changes were made to the application and/or its infrastructure between patch cycles to make for a very bad, long night, or worse, long days.

As you can see, properly patching takes a lot of time, and the more complex the application, the more people and time are required for each patch cycle. Most of this time and effort is caused due to the reboots required.

When you avoid the reboot, the patch process is significantly simplified; as shown in the following diagram, there are only two steps!

Figure 7.4 – Ksplice patching process

The following list provides a short description of these two steps:

• Ksplice the servers: Apply the kernel and user space patches with ksplice.
• Testing: Test the application

By using Ksplice, your team can save a ton of time, reduce late nights, and improve security. Best of all, it’s easy to set up and use!