Exclusive Webinar: Your NDR is not doing enough! Find out what you need to supercharge it!


Leveraging Machine Learning for Network Traffic Anomaly Detection: Unveiling Hidden Threats

Many threats lurk in your network, hiding in external (north-south) or internal (east-west) traffic. So, this is where we come in. We leverage machine learning models and techniques, both supervised and unsupervised, to detect network anomalies hiding in your network traffic. Our network anomaly detection solutions combine detection with mitigation capabilities, enhancing overall network security by swiftly addressing potential threats.

Introduction to Network Anomaly Detection

Network anomaly detection is a cornerstone of network security, enabling the identification of unusual patterns or behaviors in network traffic that may signal potential security threats. Anomaly detection systems leverage machine learning algorithms and statistical analysis to pinpoint data points that deviate from normal network behavior. By continuously monitoring network traffic, these systems can identify unusual patterns that might indicate malicious activities, allowing for timely intervention and mitigation.

What are your cyber threats attempting in your network traffic?

To begin, threats hiding in external (north-south) traffic are attempting to do three things:

  1. Break into an enterprise
  2. Communicate with an infected host within an enterprise, OR
  3. Steal data

However, the malware activities that leave a footprint in internal network traffic are attempting:

  1. To observe and discover an enterprise’s systems and internal network
  2. To attempt lateral movement to control remote systems such as network-attached storage, and
  3. To collect information needed to follow through on the adversary’s objectives, such as stealing or exfiltrating sensitive data
Fidelis NDR: Deep Visibility & Advanced Threat Detection

This data sheet covers:

What is normal on a network?

Anomaly detection using network traffic has a long history. Traditionally, it has been done for network performance monitoring and diagnostics. Adapting this approach for threat detection presents three main challenges:

  1. Building representative baseline models for normal or benign network activities
  2. Preventing a deluge of false alarms
  3. Interpreting anomalies as threat-related activities to enable response
Five Key Contexts of Fidelis NDR Anomaly Detection for Comprehensive Threat Coverage

The Fidelis Network Detection and Response (NDR) Anomaly Detection addresses the first two challenges using two strategies. Number one, it casts a wide net by analyzing network behavior using five different contexts. These are:

  1. External
  2. Internal
  3. Application Protocols
  4. Data Movement
  5. Events detected using rules and signatures

Using Machine Learning to Detect Threats in an External Context

In an external context, we focus on properties of external or north-south traffic that is independent of the application protocol. Using Unsupervised Machine Learning, statistical anomaly detection, and advanced analytics, we flag three types of suspicious activities that involve internal assets controlled by an enterprise:

  1. Traffic going out to a new location or country
  2. Increase in the volume of traffic going out to a location or a domain
  3. External services that only a small number of clients are communicating with, particularly using less well-known or a larger number of ports

With all of this, these models protect against threats mapped by the MITRE ATT&CK framework to the Initial Access tactics. In particular, Drive-by Compromise (T1189), and Data Exfiltration, plus the techniques related to Exfiltration Over Alternative Protocol (T1048), Exfiltration Over Web Service (T1567), and Automated Exfiltration (T1020). Unsupervised anomaly detection is crucial for real-time network monitoring and analysis, allowing for the autonomous identification of anomalies without prior knowledge of what constitutes an anomaly. 

Many organizations also deploy external-facing services hosted in a demilitarized zone (DMZ) that is open to the Internet. Fidelis NDR has anomaly models targeted at DMZ services. This can detect an increase in traffic to DMZ servers or traffic originating from a new location. Such anomalies often indicate that an enterprise might be the target of a new threat vector, campaign, or adversary.

Using Machine Learning to Detect Threats in an Internal Context

In an internal context, we focus on internal traffic patterns along three dimensions to identify anomalies. This includes:

  • Who is talking to whom (I.e., connection patterns between assets)
  • Remote access and login behavior patterns
  • Volume of traffic exchanged between assets

Outlier detection is crucial in identifying unusual data points that deviate from the norm. Specifically, we flag five different types of suspicious activities. Intrusion detection systems play a significant role in enhancing network security by analyzing network behavior and recognizing unusual activity. 

Potential ThreatBehavioral FootprintAnomaly ModelMITRE ATT&CK
Proxy Circumvention (Web, DNS, Mail)Web/DNS/Mail servers used by only a small number of assets.Baseline models learn the access pattern for Web/DNS/Mail servers by different types of assets. Rarely used servers are flagged as anomalies.Proxy (T1090)
Stolen CredentialsNew or abnormal SSH or RDP login pattern.Baseline models learn who-connects-to-whom and when (work hours vs. late night, weekday vs. weekend).Credential Access (TA0006), Lateral Movement (TA0008)
Password spraying, Brute Force attackHigh rate of login failuresBaseline models learn the normal level of login failures between different asset types and services.Lateral Movement (TA0008)
DiscoveryAn asset attempting to connect to all the IP addresses within a subnet, i.e. high fan-out.Baseline models learn the normal connectivity pattern between different asset types and services.Lateral Movement (TA0008)
Data CollectionIncrease in the amount of traffic from an internal server to an asset. This can be indicative of Data Collection prior to exfiltration.Baseline models learn the data transfer patterns between different asset types and file servers. These models capture both the traffic volume as well as transfer of different file types (Microsoft Office documents, PDFs, etc.)Collection (TA0009)
Fidelis DSI: Enhanced Threat Detection with Deep Session Inspection

This data sheet covers:

Deep Learning for Anomaly Detection

Deep learning techniques, including convolutional neural networks (CNNs) and recurrent neural networks (RNNs), have become instrumental in anomaly detection. These advanced methods enable the analysis of high-dimensional data and the identification of complex patterns within network traffic. Deep learning-based anomaly detection systems have demonstrated significant promise in detecting anomalies and identifying potential security threats. However, these systems require substantial amounts of labeled data for training and validation to achieve optimal performance. By leveraging deep learning, organizations can enhance their ability to detect sophisticated and subtle anomalies in network traffic.

Evaluating Anomaly Detection

Evaluating anomaly detection systems is essential to ensure their effectiveness in identifying potential security threats. The performance of these systems is typically assessed using metrics such as accuracy, precision, recall, and F1-score. The evaluation process involves comparing the detected anomalies with the actual anomalies present in the dataset. This comparison provides valuable insights into the strengths and weaknesses of the anomaly detection system, highlighting areas for improvement. Continuous evaluation and updating are crucial for maintaining the effectiveness of anomaly detection systems, ensuring they can adapt to and detect emerging security threats.

So what does Fidelis do to combat these threats lurking in your network?

Fidelis Network® Detection and Response (NDR) uses a combination of these machine learning capabilities and advanced analytics, including anomaly detection algorithms, to detect suspicious activities on an enterprise network. Network anomaly detection techniques are crucial for identifying and mitigating threats in the increasingly complex landscape of network traffic. In a previous blog on Using Machine Learning for Threat Detection, we talked about the advantages of using Machine Learning to detect patterns of cyberattacks hiding in large amounts of network traffic data. We defined the different approaches based on Supervised and Unsupervised Machine Learning algorithms. We also released a webinar hosted by SANS where we discuss this topic in more detail. Machine learning models are particularly effective in detecting complex patterns in network traffic, enhancing the overall security posture.

Conclusion

The Fidelis NDR Anomaly Detection framework involves five contexts. They include External, Internal, Application Protocol, Data Movement, and Events detected using rules and signatures. As mentioned earlier, these contexts capture what is normal baseline behavior on the network, which then helps detect any anomalies.

Frequently Ask Questions

What is machine learning network anomaly detection?

Machine learning network anomaly detection uses algorithms to analyze network traffic and identify patterns that deviate from normal behavior, helping detect hidden threats in real time.

How does anomaly detection improve network security?

By continuously monitoring traffic and detecting unusual activity, anomaly detection helps identify potential attacks early, enabling faster response and reducing the risk of data breaches.

What types of threats can be detected using machine learning?

Machine learning can detect threats like data exfiltration, lateral movement, stolen credentials, and proxy circumvention, both in external (north-south) and internal (east-west) network traffic.

Why is high-quality data important in anomaly detection?

Accurate and comprehensive data helps build reliable baseline models of normal network behavior, which is essential for effectively identifying anomalies and reducing false positives.

About Author

Jon Belanger

Jon Belanger is a seasoned Sr. Analyst in Threat Research with a passion for unraveling the intricate world of cybersecurity. Over the years, Jon has honed his skills through hands-on experience and a commitment to staying ahead of the ever-evolving threat landscape.

Related Readings

One Platform for All Adversaries

See Fidelis in action. Learn how our fast and scalable platforms provide full visibility, deep insights, and rapid response to help security teams across the World protect, detect, respond, and neutralize advanced cyber adversaries.