Executive summary.
The gap isn’t capability. It’s the data model. Runfile indexes the right object.
In 2025 and 2026, regulated firms moved AI agents into production. The questions that follow have not changed in twenty years of financial-services audit: who acted, on whose behalf, when, on what basis — and can you prove the record has not been altered.
The tools the engineering teams bought were built for a different question.
Built to debug, not to defend.
Datadog, Langfuse, LangSmith, Arize. 14–15 day retention by default. Mutable spans. No control mapping. Optimised for finding why a prompt regressed last Tuesday.
Built for posture, not for runs.
Vanta, Drata, OneTrust. Index (control_id, evidence_artifact, owner). Excellent for SOC 2 evidence rooms. Cannot represent an agent run as a first-class object.
Built for policy, not for proof.
Credo AI, Holistic AI, ValidMind. Index the AI system as a registered object. Good for the model-risk function. Not the runtime record of what the agent did on Tuesday at 11:42.
Built for the auditor’s question.
Indexes the agent execution graph. Hash-chained, signed, externally anchored, control-mapped, retained for the obligation in scope. Verifiable offline, with a binary we publish.
The auditor’s question.
The auditor’s job is to ask the question. Runfile’s job is to make the answer producible in a form they’ll accept.
A useful way to test any compliance tool: write down the question the auditor will ask, then check whether the tool can answer it.
Take a credit-decisioning agent at a UK retail bank. It pulls a bureau file, computes debt-to-income, runs a policy threshold, and either approves, declines, or escalates. Three months after go-live, a declined customer files with the Ombudsman. Internal Audit is asked to reconstruct.
Show me every action this agent took on behalf of customer 8041 between 11:42 and 11:43 GMT on 14 March 2026. Include the prompt, the model version, the retrieved bureau response, the policy that fired, the human approval if any, and the final decision. Prove the log has not been modified since the action was taken. Map each event to GDPR Article 22, SS1/23, and the Consumer Duty fair-value test.
— Internal Audit reconstruction request, paraphrasedEvery element is concrete. No judgement calls. The answers either exist or they do not. The auditor expects a citation, not a dashboard.
The vocabulary changes by regulator. The shape does not. A SOX §404 question on a claims-routing agent. A DORA Article 17 incident reconstruction at an EU asset manager. An FDA 21 CFR Part 11 protocol-amendment audit at a pharma. All ask the same thing of the same object: the agent execution graph, signed, complete, control-mapped, retained, and verifiable.
Why existing tools do not answer it.
Three categories. Three indexes. None of them is the auditor’s object.
Three adjacent categories each solve a real problem. None of them solves the one in §02. The reason is the shape of their data model, not their capability.
Dev observability · the OTel trace
Datadog, Langfuse, LangSmith, Braintrust, Arize. Data model: (trace_id, span_id, input, output, latency, model, tokens). Three blockers as evidence:
- Retention. 14–15 day default. SOX wants 7 years. HIPAA wants 6. EU AI Act floors at 6 months and routinely longer under sector law.
- Mutability. Spans are editable by design — debugging tools should let you fix a label. Mutability and integrity are opposed primitives.
- No control mapping. The OTel schema has no notion of regulatory control. Tags are ad hoc, editable, and don’t travel with the evidence.
We’re not trying to be a better Datadog. We feed off their OTel emissions where they’re already running.
GRC platforms · the control-evidence pair
Vanta hit $300M ARR in April 2026. Drata at $100M. OneTrust at half a billion. Excellent for SOC 2 evidence rooms, where the unit is a state-of-the-world at a point in time. An agent run isn’t a point in time. It’s a sequence. The GRC model cannot represent a sequence as a first-class object.
AI governance · the system posture
Credo AI, Holistic AI, ValidMind. Index the AI system itself — its policies, its risk assessment, its bias evaluation, its registry mapping. The right product for the model-risk function. Not the runtime record.
The three-by-four matrix
| Layer | Object indexed | Vendors | Audit-grade for runs? |
|---|---|---|---|
| Engineering debug | OpenTelemetry trace | Datadog · Langfuse · LangSmith · Arize | No |
| GRC | (control, evidence_artifact) | Vanta · Drata · OneTrust | No |
| AI governance | AI system posture | Credo AI · Holistic AI · ValidMind | No |
| Agent assurance | Agent execution graph | Runfile | Yes — built for it |
The agent execution graph.
Spans are lists. Runs are graphs. You can’t traverse a list to answer the auditor’s question.
Every agent invocation is recorded as one bounded object: the run. A run has a beginning, an end, an outcome, and a directed graph of events. The graph is the unit of evidence.
Eight event classes
Within each run, Runfile records the following event classes. Each event is structured, typed, and validated against a schema before it is hashed into the chain. Open any one for the detail.
01Identity & provenance
02LLM calls
03Tool calls
04Retrieval events
05Approvals & human-in-the-loop
06Refusals & guardrail activations
07External side effects
08Outcome
Graph, not trace
A tool call may be triggered by an LLM call, which may be triggered by an orchestrator step. Parent-child edges are first-class. The auditor’s question is a graph traversal. A trace is a list; a run is a graph.
- span · llm.invoke
- span · tool.fetch_credit_bureau
- span · retrieval.policy_docs
- span · policy.high_value_threshold
- span · human.approval_granted
- span · agent.complete
- run_8f2c41 · credit_review
- orchestrator.step · assess
- llm.invoke
- tool.fetch_credit_bureau
- retrieval.policy_docs
- policy.high_value_threshold · escalate
- human.approval_granted · j.tomlin
- llm.invoke
- agent.complete · denied_with_review
- orchestrator.step · assess
Schema, versioning, wire format
Schema authored in TypeScript + Zod, single source of truth. Generates JSON Schema (Python/Pydantic), Go structs (event processor, verifier CLI), TS types. Wire format is canonical JSON per RFC 8785 — the precondition for hashing. Auditor-facing exports emit JSON-LD with a stable context. Schema migrations are additive only for the lifetime of the retention obligation.
Cryptographic chain of custody.
Boring primitives, conservatively combined. An evidence package signed today remains verifiable if Runfile ceased to exist tomorrow.
Trust here means cryptographic, not contractual. The auditor should not need to trust Runfile’s policies or our staff. They should be able to verify the data alone, offline, with a binary they downloaded from our public GitHub releases.
The architecture, in one diagram
Three independent integrity properties
Tamper with one payload → its hash changes → every later chain entry diverges → the day’s Merkle root mismatches the signed manifest. Detected from any later event onward.
The Merkle root is signed by a per-tenant KMS key (FIPS 140-2 Level 3 HSM). Re-checkable offline with the public key we publish.
The weekly meta-root is anchored to Sigstore Rekor. Even Runfile, with full database access, structurally cannot alter a past root without producing a meta-root that contradicts the public log. The auditor verifies us without us.
Storage: S3 Object Lock, compliance mode
Compliance mode is the stricter Object Lock variant. Not even the AWS root account can delete or modify an object within the retention window. Runfile is removed from the threat model. An insider with full root credentials cannot alter a single payload. Verifiable independently via the customer’s CloudTrail.
Customers ask: can we hold the signing key and sign our own logs? No. If the agent runtime can sign its own logs, the logs aren’t credible evidence against the customer — same problem as a defendant signing their own affidavit. The separation between the runtime and the signing infrastructure is the chain-of-custody claim. We won’t break it on request.
SDK & trust boundary.
The SDK is open. The PII stays put. The signing key is on our side of the wall by design.
The capture SDK is the only Runfile component that runs inside the customer’s environment. Open source, Apache 2.0.
What the SDK does
Full coverage
LangGraph, OpenAI Agents SDK, Anthropic Claude SDK, Model Context Protocol (MCP).
Partial coverage
LangGraph.js, Claude SDK TS. Mastra and Vercel AI SDK land in v1.5.
Decorator wrap
One @capture decorates an agent function. Every framework-native event captured automatically.
OTel subscriber
If you already emit OTel GenAI semantic conventions, we ride alongside — no agent code changes.
PII never leaves the customer’s environment
Deterministic tokenisation at the SDK boundary. The mapping lives in the Runfile Token Vault — separate service, separate KMS key, separate IAM boundary, separate audit log from the event store. Three properties follow:
Separate key, separate access boundary, separate audit log. Turning a token back into a name is logged and rate-limited, and on Enterprise it needs your signature.
- Our event store never holds cleartext PII. Ingest sees only tokens. Most auditor questions are at the agent-action layer (did the agent call this tool with these arguments) and answer fine from the tokenised store.
- Reidentification is per-resolution, audit-logged. Rate-limited, scoped, justification required. The auditor sees cleartext when they need it; the platform engineer querying for latency does not.
- Enterprise customers hold the reidentification key. Even Runfile with full Vault access cannot resolve a token to cleartext without the customer’s cross-account signature.
The Vault is its own service deliberately. Combining auth, PII reidentification, and signing behind one IAM boundary would concentrate three different secret classes with one blast radius. Different access patterns, different audit requirements, different threat models.
Regulatory mapping.
DORA is live. April 2026 MRM is live. The AI Act high-risk window opens 2 Dec 2027 — fixed, not standards-conditional. The window to be ready is now.
Some frameworks are dateable. Some are principles-based. We lead the sales motion with the dateable ones and offer the principles-based ones as supported, not promised.
Dateable obligations — we lead with these
| Framework | Date | What it demands | Retention |
|---|---|---|---|
| DORA | Live 17 Jan 2025 |
48-hour ICT incident report; ICT third-party register; audit trail. BaFin (mid-2025) brought AI explicitly into scope; UK CTP regime parallel from 1 Jan 2025. | ~ 5 years |
| Fed / FDIC / OCC MRM | Live 17 Apr 2026 |
Principles-based, risk-tiered, technology-neutral interagency guidance. SR 11-7’s three pillars — independent validation, ongoing monitoring, documentation — remain the operating template. | n/a |
| GDPR Art. 22 & 30 | Live May 2018 |
Human oversight on solely-automated decisions with legal or significant effect; records of processing under Article 30. | Sector law |
| SOX §404 | Live 2002 |
When an agent triggers or affects a financial control — revenue recognition, journal entries, period close — the agent action becomes SOX-relevant. | 7 years |
| HIPAA OCR audit logs | Live 1996 |
When an agent is the actor touching PHI, the covered entity needs an identifiable agent principal, the action, the PHI fields touched, and any human approval. | 6 years |
| EU AI Act Art. 12 & 26(6) |
Standalone 2 Dec 2027 Embedded 2 Aug 2028 |
Automatic recording of events over the lifetime of the system; deployers retain logs. Fixed dates per the May 2026 Digital Omnibus — no longer standards-conditional. | ≥ 6 mo floor |
| TRAIGA (Texas) | Effective 1 Jan 2026 |
Intentional-misuse focus rather than broad high-risk categorisation. Audit-ready evidence on AG investigation. | n/a |
Colorado AI Act has moved three times in nine months — effective 1 Feb 2026, delayed to 30 Jun 2026 (SB 25B-004), then stayed 27 Apr 2026 by federal magistrate order in the xAI matter. SB 189 would push to 1 Jan 2027 if signed. We support whatever the final Act requires. We don’t market a Colorado date.
Penalty stack — EU AI Act, Article 99
Principles-based — supported, not led
- NIST AI RMF 1.0 + GenAI Profile (AI 600-1). De facto US vocabulary. CSA Agentic Profile reframes for agents. Our event-graph feeds the Measure / Manage outputs.
- ISO/IEC 42001:2023. AI management system. Procurement pull is real. We hold the cert ourselves by H2 2027.
- UK FCA / PRA. Technology-neutral. We support SS1/23 model-documentation pack generation.
- Singapore MAS FEAT / APRA / AIDA. Principles-based, lower GTM priority. Supported on request.
- FINRA Notice 24-09. Existing rules applied to AI-assisted communications. We capture the comms side.
How the mappings are held
YAML repository, version-controlled, signed at release. Public to customers. The mapping version applied to a run is recorded in that run’s manifest, so evidence produced today can be re-verified two years from now against the mapping in force at capture.
Evidence package & how it’s verified.
Internal Audit doesn’t buy the platform because they trust Runfile. They buy it because they don’t need to.
A third party — auditor, regulator, opposing counsel — can verify a Runfile evidence package without trusting Runfile. The package is designed for that property from the inside out.
What’s in the package
Signed PDF cover
Scope, event count, integrity status, controls in scope, Merkle root, Rekor entry #, QR to permanent identifier.
JSON-LD event graph
Full canonical-JSON event graph, one run per file, parent-child edges preserved.
Signed manifests + Rekor proofs
Every relevant daily Merkle manifest. The inclusion proof against the public Sigstore Rekor log. The public key.
Mapping repo + README
Versioned control-mapping repository as applied. README in plain English on how to verify.
The verifier CLI
Go binary, single statically-linked executable, public GitHub release. Zero dependency on Runfile’s running infrastructure.
Output: a signed PDF + a JSON detail file. Attach both to the workpaper.
Offline matters
Air-gapped audit environments can’t depend on the internet. Runfile’s verifier doesn’t require it — we ship the Rekor inclusion proofs with the package. The auditor is verifying using the public key, canonical JSON, SHA-256, the Merkle construction (RFC 6962), and the Rekor format — all open standards. Runfile is not in the trust loop.
What Runfile is not.
Selling to a buyer for whom Runfile is wrong is worse than not selling at all. The disqualifiers are how we filter our pipeline.
A whitepaper that doesn’t list its disqualifiers shouldn’t be trusted. Here’s where we’re the wrong answer.
You need real-time enforcement.
Runfile records; it does not gate. Buy Lakera or Galileo. Their refusal events show up in our chain as first-class evidence of effective oversight.
You’re debugging an agent.
High-cardinality, mutable, 14-day spans is the right shape for that job. Datadog, Langfuse, LangSmith, Arize, Braintrust. Runfile rides alongside.
You need a SOC 2 evidence room.
Vanta, Drata, OneTrust. Runfile feeds into them. We’re not them.
You’re a consumer app, no regulated workload.
Most early-stage AI products do not need Runfile yet. Buy LangSmith, ship the product, come back when the auditor calls.
No regulated obligation within 12 months.
The case for paying for tamper-evident evidence today is weaker if the demand-driving regulation is more than a year out. We’ll say so on the call.
You want us to AI-generate the audit.
We don’t. The category was burned recently by a startup faking AI-generated evidence. Agents act, we record, auditors verify.
What changes in v1.5.
v1 is shippable today. v1.5 is additive, not a rewrite.
The v1 architecture is designed to permit v1.5 capabilities as additive changes — not rewrites. Schema fields exist now and are null. IAM boundaries are already in place. Evidence signed today remains verifiable through v1.5 and beyond.
| Capability | v1 | v1.5 |
|---|---|---|
| Nitro Enclave signer | KMS HSM signing | Attestation document with code provenance |
| Multi-region | One region per tenant | Multi-region replication via Terraform stack |
| BYO-cloud | Runfile AWS account | Ingest in customer’s AWS account, cross-account roles |
| TypeScript SDK | Partial (LangGraph.js, Claude SDK TS) | Parity (Mastra, Vercel AI SDK) |
| Self-service signup | Sales-led provisioning | Starter tier self-service |
| Auth | Scoped API keys | OAuth2 client credentials, mTLS |
| SIEM streaming | Runfile audit-of-audit only | Customer-side Splunk / Sentinel / Datadog Security |
| Two-approver workflows | Single approver | Multi-approver for Enterprise tier |
The argument, in one paragraph.
The argument: the agent execution graph is the right data object for AI compliance, and nobody ships it today.
It’s contestable. Other models exist — OTel traces, control-evidence pairs, AI-system-as-object — and any of them may turn out to be the right shape too. We’ve made our case; the market will adjudicate.
What is not contestable is the question. The auditor will ask: what did this agent do, on behalf of this customer, at this time, and can you prove the record is intact. The question is twenty years old in financial services. It doesn’t become a different question when the actor is an LLM with a tool belt.
Runfile is what we’re building to answer it. If you’re the CCO, Head of AI Risk, or Head of Internal Audit at a bank, insurer, or asset manager in the US, UK, or EU, we’d like to hear from you. Internal Audit signs off on what we ship; bring them to the call.