Skip to main content

DARPA Transparent Computing Engagement 5 (TC-E5)

The Transparent Computing (TC) program by DARPA concluded its series of exercises with Engagement #5 in May 2019. This dataset provides high-fidelity visibility into system interactions to detect Advanced Persistent Threats (APTs).

🎯 Program Goal​

To eliminate the "black box" nature of modern computing by recording the provenance of all system components and tracking causal dependencies. This allows for "connecting the dots" between activities to identify root causes of attacks.

📂 Dataset Overview​

  • Timeline: May 2019 (The last of five planned engagements).
  • Scale: Three instantiations of each TA1 performer were set up on separate host platforms.
  • Phases:
    1. Benign Phase: Scripted activities to establish baseline behavior.
    2. Attack Phase: Sophisticated APT behaviors instantiated by the Adversarial Challenge Team (TA5.1).
  • Format: Data is stored in Avro binary format using the CDM 20 schema.

🏗️ Technical Areas (TAs)​

  • TA1 (Tagging & Tracking): Performers include cadets, clearscope, fivedirections, marple, theia, and trace.
  • TA3 (Architecture): Performer starc provided helpful annotations for each data stream to identify specific attack events.
  • TA5.1 (Adversarial Challenge Team): Conducted the attacks and provided the Ground Truth documentation.

🛠️ Key Components & Files​

  • Ground Truth: tc_ground_truth_report_e5_update.pdf contains detailed logs of attacker actions and Indicators of Compromise (IoCs).
  • Schema: TCCDMDatum.avsc (Avro schema) and cdm.pdf (human-readable guide).
  • Tools: ta5-java-consumer.tar.gz contains Java code (v1.8) and scripts to parse Avro files into JSON.
warning

This data represents a very large repository of semantically rich and structured data released for research purposes. As it comes from research prototypes, some imperfections may exist.