Skip to main content

DARPA Transparent Computing Engagement 3 (TC-E3)

The Transparent Computing (TC) program by DARPA aims to provide forensic and real-time detection of Advanced Persistent Threats (APTs) by making system interactions "transparent." This dataset was generated during Engagement #3 in April 2018.

🎯 Program Goal​

To develop technologies that record and preserve the provenance of system elements (processes, files, inputs) and track their causal dependencies to "connect the dots" between individually legitimate activities that indicate malice.

📂 Dataset Overview​

  • Timeline: April 2018.
  • Phases:
    1. Benign Phase: Scripted background activity.
    2. Attack Phase: Realistic APT attacks conducted by the Adversarial Challenge Team (TA5.1).
  • Format: Data is stored in Avro binary format using the Common Data Model (CDM) schema.

🏗️ Technical Areas (TAs)​

  • TA1 (Tagging & Tracking): The data providers. Systems include cadets, clearscope, fivedirections, theia, and trace.
  • TA5.1 (Adversarial Challenge Team): Responsible for the attacks and providing the Ground Truth reports.

🛠️ Key Components & Files​

  • Ground Truth: tc_ground_truth_report_e3_update.pdf contains the specific actions taken by the attackers and Indicators of Compromise (IoCs).
  • Schema: TCCDMDatum.avsc (Avro schema) and cdm.pdf (human-readable documentation).
  • Tools: Includes a Java-based parser (ta3-java-consumer) to convert Avro binary data into JSON.

✅ High-Quality Data Topics​

The following "topics" are identified as containing the most complete and valid data:

  • ta1-cadets-e3-official (0, 1, 2)
  • ta1-clearscope-e3-official (0, 1, 2)
  • ta1-fivedirections-e3-official (1, 2, 3)
  • ta1-theia-e3-official (1r, 3, 5m, 6r)
  • ta1-trace-e3-official (0, 1)
warning

This data represents a very large repository of semantically rich and structured data released for research purposes. As it comes from research prototypes, some imperfections may exist.

Data​

You can find the data on the Google Drive Engagement 3 Folder

Download​

First nagivate to the dataset json file you want to download (data -> dataset name -> json file), and pick the "file Id" of the file. (For example: 1XLCEhf5DR8xw3S-Fimcj32IKnfzHFPJW)

CADETS (JSON)​

# https://drive.google.com/drive/folders/179uDuz62Aw61Ehft6MoJCpPeBEz16VFy
gdown --no-cookies 1AcWrYiBmgAqp7DizclKJYYJJBQbnDMfb # official.json.tar.gz
gdown --no-cookies 1XLCEhf5DR8xw3S-Fimcj32IKnfzHFPJW # official-1.json.tar.gz
gdown --no-cookies 1EycO23tEvZVnN3VxOHZ7gdbSCwqEZTI1 # official-2.json.tar.gz

CLEARSCOPE (JSON)​

# https://drive.google.com/drive/folders/1cbOHa5_dlu0XF8od5YKKqCGOawHzqaT_
gdown --no-cookies 1z1KtBvS-XBwCt3DlkI6yS1UO7_1sVvB6 # official.json.tar.gz (424M)
gdown --no-cookies 1cEdDdmDACkeTmC04vUwW8WXqn36t4DE4 # official-1.json.tar.gz (7.05G)
gdown --no-cookies 11bVukedxRn8lVPmbBD2S6juUlqL4cI6b # official-2.json.tar.gz (10.4G)

THEIA (JSON)​

# https://drive.google.com/drive/folders/1AWXy7GFGJWeJPGzvkT935kTfwBYzjhfC
gdown --no-cookies 10cecNtR3VsHfV0N-gNEeoVeB89kCnse5 # official-1r.json.tar.gz (1.17G)
gdown --no-cookies 1dWJecuLXZMksKAPo8348Q6L5DiccsS1u # official-3.json.tar.gz (38.2M)
gdown --no-cookies 1zbgWJgF7F0fI6JhViqQZoo6AWdoV5YFK # official-5m.json.tar.gz (28.5M)
gdown --no-cookies 1Kadc6CUTb4opVSDE4x6RFFnEy0P1cRp0 # official-6r.json.tar.gz (1.55G)

We can follow the same pattern for fivedirections and trace datasets.