Intrusion Detection Systems (IDS)

Introduction

An Intrusion Detection System (IDS) is a security technology that monitors network traffic or system activity for signs of malicious behavior, policy violations, or unauthorized access. Unlike a firewall, which controls traffic based on rules about what is permitted, an IDS analyzes traffic and system events to identify activity that should not be happening -- even if it passes through the firewall.

The key distinction is that an IDS is fundamentally a passive, detective control. It observes and alerts but does not block traffic. When the IDS detects suspicious activity, it generates an alert that is sent to security analysts or a Security Information and Event Management (SIEM) system for investigation. The actual response -- blocking an IP, isolating a host, or taking other action -- is left to human operators or automated response systems.

This passive nature gives IDS an important advantage: it cannot cause service disruptions through false positives. If the IDS misidentifies legitimate traffic as malicious, the only consequence is a false alert. Compare this with an Intrusion Prevention System (IPS), which actively blocks traffic and can therefore disrupt legitimate services when it makes mistakes.

"An intrusion detection system is like a burglar alarm for your network. It does not prevent break-ins, but it ensures you know when one is happening -- and that knowledge is the first step in any effective response." -- Stephen Northcutt, SANS Institute

History and Development

The concept of intrusion detection dates back to James Anderson's 1980 report for the U.S. Air Force, "Computer Security Threat Monitoring and Surveillance," which proposed using audit trails to detect unauthorized computer use. This foundational work established the idea that system logs could be analyzed to identify suspicious behavior.

In 1987, Dorothy Denning published "An Intrusion-Detection Model," which formalized the theoretical framework for intrusion detection and introduced the concept of anomaly detection based on statistical profiling. This paper is widely considered the birth of intrusion detection as a field of study.

NIDS vs. HIDS

Intrusion detection systems are broadly categorized by where they monitor: at the network level or at the host level. Each approach has distinct strengths and limitations, and most mature security programs deploy both.

Network-Based IDS (NIDS)

A Network-Based IDS (NIDS) monitors network traffic by capturing and analyzing packets as they traverse a network segment. The NIDS sensor is typically connected to a network tap or SPAN (mirror) port on a switch, giving it visibility into all traffic on that segment.

How NIDS works:

1. The sensor captures raw network packets from the monitored segment
2. Packets are reassembled into streams and sessions (TCP stream reassembly)
3. The reassembled data is analyzed against signatures, rules, or behavioral models
4. When a match is found, the system generates an alert with details (source, destination, rule matched, packet payload)
5. Alerts are forwarded to a console, log server, or SIEM for review

Advantages: A single sensor can monitor an entire network segment. NIDS is transparent to endpoints -- no software installation required on monitored hosts. It can detect network-level attacks (port scans, DoS, lateral movement) that HIDS would miss.

Limitations: Cannot inspect encrypted traffic (unless decryption is performed upstream). High-speed networks (10Gbps+) can overwhelm sensors. Cannot see activity that stays within a single host (local privilege escalation, malicious scripts).

Host-Based IDS (HIDS)

A Host-Based IDS (HIDS) is installed on individual hosts (servers, workstations, or endpoints) and monitors activity on that specific system. HIDS examines:

• System logs: Authentication logs, application logs, kernel messages
• File integrity: Detects unauthorized changes to critical system files, configuration files, and binaries
• Registry monitoring: On Windows systems, detects changes to registry keys commonly targeted by malware
• Process monitoring: Identifies suspicious processes, unexpected services, or abnormal resource usage
• Rootkit detection: Scans for signs of rootkit installation (hidden processes, hidden files, kernel module anomalies)

Advantages: Can detect attacks that NIDS misses (local exploits, insider abuse, encrypted traffic after decryption at the host). Provides detailed context about what happened on a specific machine. Can monitor file integrity and system configuration changes.

Limitations: Must be installed and maintained on every monitored host, creating management overhead. Consumes resources on the host. If the host is compromised, the HIDS itself may be disabled or subverted by the attacker.

Detection Methods

Signature-Based Detection

Signature-based detection (also called misuse detection) compares observed activity against a database of known attack patterns, called signatures or rules. Each signature describes a specific pattern -- a sequence of bytes in a packet payload, a particular combination of TCP flags, a known exploit string, or a suspicious HTTP request pattern.

For example, a Snort rule to detect a simple SQL injection attempt:

alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS ( msg:"SQL Injection attempt - UNION SELECT"; flow:to_server,established; content:"UNION"; nocase; content:"SELECT"; nocase; distance:0; sid:1000001; rev:1;)

Advantages: Very low false positive rate for known attacks. Fast and efficient -- pattern matching is computationally inexpensive. Easy to understand what was detected and why.

Limitations: Cannot detect novel (zero-day) attacks that have no existing signature. Requires constant signature updates to remain effective. Can be evaded through minor modifications to attack payloads (polymorphic attacks, encoding variations).

Anomaly-Based Detection

Anomaly-based detection establishes a baseline of "normal" behavior for a network or system, then alerts when observed behavior deviates significantly from that baseline. The system learns what normal looks like during a training period and flags statistical outliers.

Anomaly detection can identify:

• Unusual traffic volumes or patterns (potential DDoS or data exfiltration)
• Connections to previously unseen destinations
• Unusual protocols or ports being used
• Login attempts at abnormal hours
• Deviation from established user behavior patterns

Advantages: Can detect novel, zero-day attacks and previously unknown threats. Does not require signatures for every possible attack. Can identify slow, stealthy attacks that signature-based systems miss.

Limitations: Higher false positive rate -- legitimate changes in behavior (new applications, infrastructure changes, seasonal traffic patterns) trigger alerts. Requires a clean training period (if malicious activity exists during baseline establishment, it becomes "normal"). More computationally expensive than signature matching.

Hybrid Detection

Modern IDS platforms combine both approaches. Signature-based detection catches known threats with high confidence, while anomaly-based detection provides coverage for unknown threats. This combination, along with protocol analysis and behavioral heuristics, provides the most comprehensive detection coverage.

Major IDS Tools

Snort is the most widely deployed open-source NIDS, created by Martin Roesch in 1998 and now maintained by Cisco (following the acquisition of Sourcefire in 2013). Snort operates in three modes: packet sniffer, packet logger, and full NIDS. It uses a flexible, powerful rule language that has become a de facto standard -- many commercial IDS products can import Snort rules. Snort 3, the latest major version, features a modular architecture with multi-threading support.

Suricata, developed by the Open Information Security Foundation (OISF) and released in 2010, is a high-performance, multi-threaded IDS/IPS engine designed for modern high-speed networks. Suricata supports Snort-compatible rules but adds native support for protocol detection (independent of port), file extraction, TLS/SSL certificate logging, and Lua scripting for custom detection logic. Its multi-threaded architecture allows it to scale across multiple CPU cores, addressing the performance limitations of Snort's single-threaded design.

Zeek (formerly Bro) takes a fundamentally different approach from Snort and Suricata. Rather than focusing on signature matching, Zeek is a network analysis framework that provides deep protocol analysis and generates structured logs describing network activity. Zeek's output includes detailed records of every connection, DNS query, HTTP transaction, SSL certificate, and file transfer -- creating a rich data source for security analysis, forensics, and threat hunting.

OSSEC is the leading open-source HIDS, providing log analysis, file integrity monitoring (FIM), rootkit detection, and real-time alerting. OSSEC uses an agent-server architecture: lightweight agents run on monitored hosts and forward events to a central server for analysis. Wazuh, a fork of OSSEC, extends it with a modern web interface, RESTful API, compliance reporting (PCI DSS, HIPAA, GDPR), and integration with Elastic Stack for visualization.

False Positives and Tuning

The most persistent operational challenge in intrusion detection is managing false positives -- alerts triggered by legitimate activity that is incorrectly identified as malicious. Excessive false positives lead to alert fatigue, where analysts become desensitized to alerts and begin ignoring them, potentially missing real attacks buried in the noise.

Equally dangerous are false negatives -- real attacks that the IDS fails to detect. The tension between false positives and false negatives is fundamental: tuning the IDS to be more sensitive (catching more real attacks) inevitably increases false positives, while reducing false positives (making the IDS less sensitive) increases the risk of missing real attacks.

Strategies for reducing false positives without increasing false negatives:

• Baseline and tune: Deploy the IDS in monitoring mode first. Analyze the alert output for several weeks to identify false positives, then tune rules to suppress them.
• Suppress by context: Disable rules that are not relevant to your environment. If you have no Windows servers, disable Windows-specific attack signatures.
• Threshold alerts: Instead of alerting on every occurrence, set thresholds (e.g., alert only if 10 failed logins occur within 60 seconds).
• Whitelist known-good: Suppress alerts for known legitimate activity (e.g., vulnerability scanners during scheduled scan windows).
• Correlate with context: Integrate IDS alerts with asset inventory and vulnerability data. An alert for a Linux exploit targeting a Windows server is clearly a false positive.
• Use SIEM correlation: Combine IDS alerts with other data sources (firewall logs, authentication logs, endpoint detection) to confirm or dismiss alerts.

"An IDS that generates thousands of unactionable alerts per day is worse than no IDS at all, because it creates a false sense of security while consuming analyst time that could be spent on actual threat hunting." -- Richard Bejtlich, The Practice of Network Security Monitoring

Deployment Architecture

Effective IDS deployment requires strategic sensor placement to maximize visibility:

• Perimeter: Place a NIDS sensor between the internet-facing firewall and the DMZ to detect inbound attacks before they reach public-facing services.
• Behind the firewall: Place a sensor on the internal side of the firewall to detect attacks that pass through (zero-days, permitted protocols used for attack).
• Between segments: Place sensors between network segments (DMZ-to-internal, between VLANs) to detect lateral movement.
• Core switch: Monitor traffic at the network core to detect internal threats and east-west traffic.
• Critical servers: Install HIDS agents on databases, domain controllers, web servers, and other high-value targets.

All IDS sensors should forward alerts to a centralized management console or SIEM. This centralized view enables correlation across sensors, trend analysis, and efficient incident response.

IDS vs. IPS

An Intrusion Prevention System (IPS) extends IDS by adding the ability to actively block detected threats, rather than merely alerting on them. While an IDS passively monitors a copy of network traffic (via tap or SPAN), an IPS sits inline in the traffic path and can drop, modify, or reset connections in real time.

Many modern platforms (Snort, Suricata, commercial NGFWs) can operate in either IDS mode (passive monitoring) or IPS mode (inline blocking). The choice between IDS and IPS depends on the organization's risk tolerance, the maturity of their detection rules, and whether they can afford the risk of false-positive-induced service disruptions.

A common deployment strategy is to run new rules in IDS mode first, tune them to reduce false positives, and then promote well-tested rules to IPS mode for active blocking.

Best Practices

• Deploy both NIDS and HIDS for comprehensive visibility across network and host layers
• Place NIDS sensors at strategic chokepoints -- perimeter, between segments, and at the network core
• Keep signatures and rules up to date -- subscribe to rule feeds (Emerging Threats, Snort community rules, vendor feeds)
• Tune aggressively to reduce false positives -- an untuned IDS quickly becomes ignored
• Forward all IDS alerts to a centralized SIEM for correlation, trending, and incident response
• Establish a formal alert triage process with defined severity levels and escalation procedures
• Conduct periodic detection testing -- run known attacks against the IDS to verify it detects them
• Consider deploying Zeek alongside Snort/Suricata for complementary detection -- Zeek provides protocol analysis and logging, while Snort/Suricata provide signature-based alerting
• Monitor IDS sensor health and performance -- an overloaded sensor that drops packets provides a false sense of security
• Integrate with IPS and firewall infrastructure for automated response where appropriate

References

• Anderson, J. P. (1980). Computer Security Threat Monitoring and Surveillance. James P. Anderson Co., Fort Washington, PA.
• Denning, D. E. (1987). "An Intrusion-Detection Model." IEEE Transactions on Software Engineering, SE-13(2), 222-232.
• Roesch, M. (1999). "Snort: Lightweight Intrusion Detection for Networks." Proceedings of LISA '99, USENIX.
• Bejtlich, R. (2013). The Practice of Network Security Monitoring: Understanding Incident Detection and Response. No Starch Press.
• Northcutt, S., & Novak, J. (2002). Network Intrusion Detection, 3rd Edition. New Riders.
• Scarfone, K., & Mell, P. (2007). NIST SP 800-94: Guide to Intrusion Detection and Prevention Systems (IDPS).
• OISF (2024). Suricata User Guide. Open Information Security Foundation.
• Paxson, V. (1999). "Bro: A System for Detecting Network Intruders in Real-Time." Computer Networks, 31(23-24), 2435-2463.
• Bray, R., et al. (2008). OSSEC Host-Based Intrusion Detection Guide. Syngress.
• SANS Institute (2023). Intrusion Detection FAQ. SANS Information Security Reading Room.