MITRE ATT&CK Coverage by Security Tools Is Inconsistent, Incomplete: Researchers

Cybersecurity tools are inconsistent and incomplete in their coverage of the MITRE ATT&CK framework, according to research presented at the fifth MITRE ATT&CKcon conference in McLean, Virginia today.

The MITRE ATT&CK framework identifies tactics and techniques that indicate a cyberattack is in progress, and is often used by security vendors, analysts and researchers as a framework for detecting and investigating incidents. “ATT&CK” stands for “Adversarial Tactics, Techniques, & Common Knowledge.”

The researchers – led by Apurva Virkud, a PhD student in computer science at the University of Illinois Urbana-Champaign, who presented the research – looked at endpoint security and security information and event management (SIEM) tools in conducting the research, which dates from 2022-2023.

What they found was that the tools examined – Carbon Black, Splunk, Elastic and the Sigma open source tool – had at least one detection technique for about half of the ATT&CK framework, and lower-risk detections could further dilute that value, Virkud said.

Virkud noted that MITRE doesn’t position ATT&CK as a marketing tool, even though vendors often tout their ATT&CK coverage. She said ATT&CK coverage is “too high level of a metric to really be meaningful.”

MITRE ATT&CK Coverage: Same Threats, Different Techniques

Virkud and colleagues found that the products were consistent in which techniques are covered (slide below).

Security tools’ MITRE ATT&CK technique coverage (Apurva Virkud)

“Even when products are trying to detect the same threat, they’re not using the same attack techniques to describe it,” Virkud said. Those variations may be reasonable, she said, because an ATT&CK technique can cover multiple behaviors.

The researchers also looked at 53 techniques that weren’t implemented in any of the tools, and found the top three reasons for not implementing a technique were:

Ineffective detection method: MITRE itself notes that some behaviors are difficult to detect.
Targets non-host infrastructure: Internet scanning is beyond the scope of these tools.
Client-specific: Detection requires specific knowledge of a customer environment.

“Many of these techniques are difficult if not impossible to implement,” Virkud said (slides below).

Inconsistent ATT&CK Application

Virkud compared rules from Elastic and Splunk for named pipe impersonation and malicious DNS activity (slides below) and noted that “security analysts may attribute the same system log activity to completely different motivations depending on which tool they are using.”

MITRE ATT&CK inconsistency case studies

Perhaps most surprisingly, Virkud and colleagues found that products disagree on the appropriate ATT&CK technique about half the time.

As Virkud’s abstract noted, “even when attempting to detect the same malicious entity, products completely disagree about the appropriate ATT&CK technique annotations 51% of the time, while fully agreeing just 2.7% of the time. Put another way, ‘covering’ one technique may not even suggest protection from the same threat across different products. These findings underscore the dangers of coverage-based ATT&CK assessments.”

The researchers recommended ongoing guidance, evaluations and education from MITRE, and caution and nuance among vendors and practitioners:

Source: Read More