Audit Framework

The Audit Framework is a specialized subsystem within CAPE designed to verify the correctness and reliability of the sandbox’s analysis capabilities. It allows operators to define specific test cases (“Audit Packages”) that run known samples with expected behavioral outcomes (“Objectives”). This is particularly useful for validating that CAPE is correctly capturing specific behaviors (e.g., shellcode injection, network beacons) after updates or configuration changes.

Concepts

Available Test: A test case definition stored on the disk. It consists of a payload (e.g., a malware sample) and a Python script defining the success criteria.
Test Session: A collection of test runs. You can group multiple tests into a session to validate a specific aspect of the system (e.g., “Weekly Regression Test”).
Test Run: A single execution of an Available Test within a Test Session. It links to a standard CAPE Task ID.
Objective: A specific criterion that must be met for a test to pass (e.g., “DNS request to evil.com observed”, “File dropped in AppData”).

Configuration

To enable the Audit Framework, ensure the feature is enabled in your web configuration.

Edit conf/web.conf:

[audit_framework]
enabled = yes

The framework looks for test packages in tests/audit_packages/ by default.

Creating Audit Packages

Audit packages are directory-based. Each package must be a subdirectory inside tests/audit_packages/ (or the configured path) containing at least two files:

payload.zip: A zip file containing the sample to be analyzed. * Note: If the zip contains a single file, that file is treated as the payload. If it contains multiple files, the extracted directory is treated as the payload (useful for packages requiring dependencies).
test.py: A Python script defining the test metadata, objectives, and evaluation logic.

Directory Structure Example

tests/audit_packages/
├── Emotet_Network_Beacon/
│   ├── payload.zip
│   └── test.py
└── AsyncRAT_Config_Extract/
    ├── payload.zip
    └── test.py

The `test.py` Structure

The Python script must define a class named CapeDynamicTest that implements the following methods:

get_metadata(): Returns a dictionary of test settings.
get_objectives(): Returns a list of objective objects.
evaluate_results(task_dir): Analyzes the analysis results.
get_results(): Returns the final status of objectives.

Example `test.py`:

import os
import json

class TestObjective:
    def __init__(self, name, requirement, children=None):
        self.name = name
        self.requirement = requirement
        self.children = children or []

class CapeDynamicTest:
    def __init__(self):
        self._results = {}

    def get_metadata(self):
        """
        Define high-level test information.
        """
        return {
            "Name": "Emotet Beacon Test",
            "Description": "Verifies that CAPE detects the C2 network connection.",
            "Package": "exe",          # CAPE analysis package to use
            "Timeout": 200,            # Analysis timeout in seconds
            "Zip Password": "infected" # Password for payload.zip (optional)
        }

    def get_objectives(self):
        """
        Define the criteria for success.
        """
        return [
            TestObjective("network_c2", "Must connect to C2 server 1.2.3.4"),
            TestObjective("dropped_payload", "Must drop the second stage loader")
        ]

    def evaluate_results(self, task_dir):
        """
        Parse the CAPE report to verify objectives.
        task_dir: Path to the storage directory for this task (contains report.json, etc.)
        """
        report_path = os.path.join(task_dir, "reports", "report.json")

        # Default state
        self._results = {
            "network_c2": {"state": "failure", "state_reason": "IP not found"},
            "dropped_payload": {"state": "failure", "state_reason": "File not found"}
        }

        if not os.path.exists(report_path):
            return

        with open(report_path, "r") as f:
            report = json.load(f)

        # Check Network
        for host in report.get("network", {}).get("hosts", []):
            if host == "1.2.3.4":
                self._results["network_c2"] = {"state": "success", "state_reason": "Connection found"}

        # Check Dropped Files
        if "dropped" in report:
            self._results["dropped_payload"] = {"state": "success", "state_reason": "Dropped files present"}

    def get_results(self):
        """
        Return the dictionary of results calculated in evaluate_results.
        Keys must match the Objective names.
        """
        return self._results

Web Interface Usage

Access the Audit interface via the sidebar menu or at /audit/.

Manage Tests: The main dashboard lists all available tests. * If you have added new tests to the disk, click Reload Tests to update the database.
Create Session: * Select the checkboxes next to the tests you wish to run. * Click Create Session. * You will be redirected to the Session view.
Run Audit: * In the Session view, you can see the status of each test (Unqueued, Queued, Running, Complete). * Click Queue All to submit all unqueued tests to CAPE. * The status will update automatically as CAPE processes the tasks.
View Results: * Once a test is Complete, the framework automatically runs the evaluate_results logic from your test.py. * The UI will display a Pass (Green) or Fail (Red) badge for each objective. * You can expand a test row to see detailed reasons for failure or success.