Rootkits

Introduction

A rootkit is a collection of software tools designed to provide unauthorized access to a computer while actively concealing its presence from users, administrators, and security software. The name derives from the Unix term "root" (the superuser account with full system privileges) and "kit" (the set of tools used). Rootkits represent one of the most insidious categories of malware because their primary purpose is stealth -- they do not just compromise a system, they make the compromise invisible.

Unlike viruses or worms that spread aggressively, rootkits prioritize persistence and concealment. They are frequently deployed as a component of a larger attack, providing a hidden platform from which other malware can operate undetected. A rootkit might hide the files, processes, network connections, and registry entries of a trojan, spyware, or cryptocurrency miner.

"The defining characteristic of a rootkit is not what it does, but what it hides. A rootkit turns the operating system itself into an accomplice." -- Greg Hoglund, co-author of Rootkits: Subverting the Windows Kernel

Types of Rootkits

Rootkits operate at different levels of the system architecture. The deeper the rootkit embeds itself, the more powerful its concealment capabilities become -- but also the more complex it is to develop and the more likely it is to cause system instability.

User-Mode Rootkits

User-mode rootkits operate in the same privilege level as normal applications. They hide their presence by intercepting and modifying the results of system API calls. On Windows, a user-mode rootkit might hook functions in system DLLs (such as ntdll.dll or kernel32.dll) using techniques like Import Address Table (IAT) hooking or inline hooking (patching the first bytes of a function to redirect execution). When a security tool calls FindFirstFile to list directory contents, the rootkit intercepts the result and removes its own files from the listing.

User-mode rootkits are easier to develop but also easier to detect, since they operate at the same privilege level as security software. A kernel-mode detector can see past user-mode hooks.

Kernel-Mode Rootkits

Kernel-mode rootkits run with the highest operating system privileges (Ring 0 on x86 processors). They can modify kernel data structures, hook system calls at the kernel level, and manipulate the very mechanisms that security software relies on for visibility. On Windows, kernel-mode rootkits commonly modify the System Service Descriptor Table (SSDT) to intercept system calls, or manipulate the EPROCESS linked list (Direct Kernel Object Manipulation, or DKOM) to unlink a process from the process list entirely -- making it invisible to Task Manager and all user-mode tools.

Kernel-mode rootkits are loaded as device drivers on Windows or kernel modules on Linux. Modern Windows versions require kernel-mode drivers to be digitally signed (Driver Signature Enforcement), which has raised the bar significantly. However, attackers have circumvented this through stolen code-signing certificates, vulnerable signed drivers that can be exploited (Bring Your Own Vulnerable Driver, or BYOVD), and exploitation of kernel vulnerabilities.

Bootkits

A bootkit infects the boot process itself -- the Master Boot Record (MBR), Volume Boot Record (VBR), or boot manager -- loading before the operating system kernel. Because the bootkit executes before any security software, it can intercept and control the entire boot chain. Notable bootkits include TDL4 (Alureon), which infected the MBR and was one of the most advanced rootkits of its era, and Rovnix, which infected the VBR.

The introduction of UEFI Secure Boot was specifically designed to counter bootkits by verifying the digital signature of each component in the boot chain. Only signed, trusted code is allowed to execute during boot.

Firmware and UEFI Rootkits

Firmware rootkits embed themselves in hardware firmware -- the UEFI/BIOS, network card firmware, hard drive controller firmware, or other embedded systems. Because firmware persists independently of the operating system, these rootkits survive OS reinstallation, disk formatting, and even hard drive replacement (if targeting non-disk firmware). They represent the most persistent and difficult-to-remediate form of rootkit.

In 2018, ESET discovered LoJax, the first UEFI rootkit found in the wild, attributed to the Russian APT group Fancy Bear (APT28). LoJax wrote a malicious UEFI module to the SPI flash chip on the motherboard, ensuring persistence even after a complete OS reinstallation.

Hypervisor-Level Rootkits

Hypervisor rootkits (also called virtual machine-based rootkits, or VMBRs) insert a thin hypervisor beneath the running operating system, effectively moving the OS into a virtual machine without its knowledge. The rootkit then controls all hardware access from below. Proof-of-concept implementations include Blue Pill (by Joanna Rutkowska, 2006) and SubVirt (by Microsoft Research and University of Michigan). While theoretically devastating, practical hypervisor rootkits remain rare in the wild due to extreme development complexity.

Rootkit Techniques

Rootkits employ a variety of techniques to hide their presence and maintain control:

Notable Rootkits in History

The Sony BMG Rootkit Scandal

In 2005, security researcher Mark Russinovich discovered that Sony BMG music CDs contained hidden copy-protection software called Extended Copy Protection (XCP), developed by First 4 Internet. When a user inserted a Sony BMG CD into their Windows computer, the software silently installed a rootkit that hid any file, process, or registry key beginning with "$sys$". It modified the Windows kernel's system call table to filter its presence from all applications, including security software.

The implications were severe. The rootkit created a security vulnerability that any malware author could exploit -- malware simply needed to name its files with the "$sys$" prefix to become invisible. Multiple worm variants were discovered exploiting this exact mechanism within weeks of the disclosure. The rootkit also transmitted listening data back to Sony's servers, raising privacy concerns.

"The entire episode illustrates what happens when a company decides that its interests in protecting its intellectual property override its customers' interests in maintaining the security of their computers." -- Bruce Schneier

The scandal resulted in class-action lawsuits, a recall of millions of CDs, settlements with the Federal Trade Commission and multiple state attorneys general, and lasting damage to Sony BMG's reputation. It remains one of the most prominent examples of a legitimate company deploying rootkit technology against its own customers.

Detection Methods

Detecting rootkits is inherently challenging because they specifically target the mechanisms used for detection. Effective rootkit detection typically relies on cross-view comparison -- examining the system from multiple vantage points and looking for discrepancies.

• Cross-View Detection: Compare results from high-level APIs (which rootkits can hook) with raw disk or memory analysis (which bypasses hooks). If the API reports 200 files but raw disk analysis finds 205, the 5 hidden files are likely rootkit-related. Tools like GMER and RootkitRevealer use this approach.
• Integrity Checking: Compare current system files, kernel objects, and boot sectors against known-good baselines. Tripwire and OSSEC implement this approach. UEFI Secure Boot performs integrity checking on the boot chain.
• Memory Forensics: Analyze a memory dump offline using tools like Volatility or Rekall. Memory analysis can reveal hidden processes, injected code, hooked functions, and kernel modifications that the rootkit would normally conceal from the running OS.
• Behavioral Analysis: Monitor for rootkit installation behaviors: loading unsigned drivers, modifying the MBR/VBR, writing to SPI flash, hooking system calls. EDR solutions can detect these actions in real time.
• Boot from Trusted Media: Boot the system from a known-clean USB drive or CD and scan the installed operating system from the outside. Since the rootkit is not running, it cannot hide its files.
• Hardware-Based Detection: Intel Boot Guard and AMD Platform Secure Boot verify firmware integrity using hardware root-of-trust mechanisms that firmware rootkits cannot subvert.

Prevention and Mitigation

Preventing rootkit installation is far preferable to detecting and removing one after the fact:

• Enable UEFI Secure Boot: Ensures only digitally signed boot components can execute, blocking bootkits and many firmware rootkits
• Enable Driver Signature Enforcement: Prevents unsigned kernel drivers from loading on Windows. Do not disable this feature.
• Keep Firmware Updated: Apply UEFI/BIOS updates from your hardware vendor to patch firmware vulnerabilities
• Use Virtualization-Based Security (VBS): Windows features like Hypervisor-Protected Code Integrity (HVCI) use the hypervisor to protect kernel memory from modification
• Principle of Least Privilege: Rootkit installation typically requires administrator or root privileges. Minimize the number of accounts with elevated access.
• Monitor for BYOVD Attacks: Block known vulnerable drivers using Microsoft's Vulnerable Driver Blocklist or custom driver-loading policies
• Hardware Root of Trust: Use TPM 2.0 for measured boot, which records the integrity of each boot component and can detect unauthorized modifications

If a rootkit is confirmed, the safest remediation is to reformat and reinstall the operating system from trusted media. For firmware rootkits, reflashing the UEFI/BIOS firmware may be necessary, and in extreme cases, hardware replacement is the only guaranteed solution.

Modern Firmware Threats

UEFI firmware rootkits represent the frontier of rootkit development. The 2023 discovery of BlackLotus -- the first publicly known bootkit capable of bypassing UEFI Secure Boot on fully patched Windows 11 systems -- demonstrated that even modern security mechanisms have exploitable weaknesses. BlackLotus exploited CVE-2022-21894, a vulnerability in the Windows Boot Manager, and was sold on underground forums for approximately $5,000.

The increasing complexity of UEFI firmware (which can contain millions of lines of code) has expanded the attack surface significantly. Security researchers have identified hundreds of vulnerabilities in UEFI implementations from major vendors, and the firmware supply chain involves multiple parties (chipset vendors, BIOS vendors, OEMs), making comprehensive security difficult to achieve.

For a broader understanding of how rootkits fit into the malware landscape, see our pages on malware analysis and spyware, which often employ rootkit techniques for concealment.

Summary

Rootkits remain among the most technically sophisticated and dangerous forms of malware. Key takeaways:

• Rootkits operate at multiple system levels, from user-mode API hooking to UEFI firmware modification
• The deeper a rootkit embeds itself, the more difficult it is to detect and remove
• Modern defenses like Secure Boot, driver signing, and VBS have raised the bar significantly but are not impenetrable
• Detection relies on cross-view comparison, memory forensics, and integrity verification
• Firmware rootkits represent the current frontier, surviving OS reinstallation and disk replacement
• Prevention through security feature enablement and least privilege is more effective than post-infection remediation

References

• Hoglund, G., & Butler, J. (2005). Rootkits: Subverting the Windows Kernel. Addison-Wesley.
• Russinovich, M. (2005). "Sony, Rootkits and Digital Rights Management Gone Too Far." Mark's Blog, Microsoft.
• Rutkowska, J. (2006). "Introducing Blue Pill." Black Hat USA.
• ESET Research. (2018). "LoJax: First UEFI Rootkit Found in the Wild." ESET White Paper.
• ESET Research. (2023). "BlackLotus UEFI Bootkit: Myth Confirmed." ESET White Paper.
• Kaspersky GReAT. (2022). "CosmicStrand: The Discovery of a Sophisticated UEFI Firmware Rootkit." Kaspersky Securelist.
• Microsoft. (2023). "Guidance for Investigating Attacks Using CVE-2022-21894: The BlackLotus Campaign." Microsoft Security Blog.
• Szor, P. (2005). The Art of Computer Virus Research and Defense. Addison-Wesley.
• Schneier, B. (2005). "Sony's DRM Rootkit: The Real Story." Schneier on Security.
• NIST. (2011). SP 800-155: "BIOS Integrity Measurement Guidelines." National Institute of Standards and Technology.