In the last couple of weeks, you may have heard or seen cybersecurity vendors pumping out marketing content about their solutions surrounding the latest results from the MITRE Engenuity ATT&CK® Evaluations for Managed Services. Some quotes from these campaigns include “X tool provided nearly 100% coverage,” “we scored 100%,” or “X tool leads the latest MITRE ATT&CK Evaluations with 100% automated prevention.”
We all know there is no silver bullet in cybersecurity, so it is important to recognize that these rankings and announcements can be greatly misleading as they fail to take a holistic or nuanced view of the results, and how the assessment was run. This phenomenon is largely understood by professionals in the industry and is even typically mocked. However, many organizations investigating new solutions can be easily misled or fall into these traps to ultimately pick a tool that does not fit their actual needs from a threat-informed perspective.
For each evaluation, you will find the following statement in bold print on MITRE Engenuity’s page: “Remember, we don’t compare or rate providers or tools. We only evaluate individual tools.”
The ATT&CK Evaluations are an extremely powerful dataset that can be closely examined to assess solutions, enhance current configurations, and discover defensive capabilities.
For those unfamiliar, the MITRE Corporation is a non-profit federally funded organization dedicated to focusing on cybersecurity. They publish and maintain the classification scheme, MITRE ATT&CK, which is now integrated into almost every defensive solution enabling us to define adversarial actions. Further, the ATT&CK evaluations use offensive techniques in a live fire exercise against a set of vendor solutions by emulating real adversary groups observed in the world. The stage and scale of the test is impressive, because they somehow convinced top competing private organizations to collaborate and publish the results to the public. This is a feat that should be celebrated, given how open and visible each solution is to a potential consumer.
Vendors are unaware of the procedures the MITRE team would execute prior to an assessment and must rely on the effectiveness of the tool they deployed. The managed services evaluations this year were focused on the OilRig adversary group, an Iranian threat group which targets international companies in ranging industries. All evaluations are meant to leverage publicly available CTI to test tools against realistic and relevant threats.
OilRig Scenario
These simulations are also publicly available can be re-performed by any interested party. The scenarios include all the binaries, procedures, step-by-step instructions, and tools used to carry out the evaluation. At DenSecure, we typically refer to these exercises as a threat emulation that can range in complexity from an atomic test to compounding attack chains, or a full purple team exercise.
A helpful guide published by MITRE Engenuity can be found here in assessing the evaluations. Red Canary has an excellent resource with a methodology of how to interpret the most recent results as well.
MITRE Engenuity Tips
Each evaluation can help Organizations compare solutions and fully understand the capabilities of their tools by putting them to the test rather than rely on promises. Below, is an example of searching the ATT&CK Evaluations page:
Searching for Vendor
In this example, I selected the results from the CrowdStrike Managed Services Assessment (2022). The following page allows you to download a complete archive of the assessment results which includes e-mails, screenshots, and reports from each testing day. You can even view any configuration changes made by the vendor so you can implement them yourself:
Obtaining Results
Once downloaded, you can assess the results yourself to determine if the tool can fit your model and environment. If you already have the tool, you can perform your own emulation to validate your own assumptions about the tool.
Outside of the marketing tactics employed, all vendors that participated and solutions presented are extremely powerful and can fill a specific need. There are certain limitations of how the test was performed that should be understood to give fair credit to each vendor.
This style of testing is trending to fully capture and understand our defensive environments as an intricate machine involving humans and technology. Defensive tools can rarely be implemented without proper continuous tuning, time, and resources. The ATT&CK Evaluations are a great resource and a step in the right direction that enables offensive security testing to provide tangible results for defenders. Use the results and assessments as part of your due diligence when selecting new solutions or investigate how you can implement a similar methodology to test your own environment!
