agent-audit

Transparent, traceable observability for agentic AI pipelines.

Every LLM call. Every extracted fact. Every score. Fully auditable.

Works with LangChain, LangGraph, LangSmith, and any custom agent — or standalone with zero dependencies.

---

Install

pip install agent-audit                      # core only
pip install "agent-audit[langsmith]"          # + LangSmith tracing
pip install "agent-audit[langchain]"          # + LangChain callback handler
pip install "agent-audit[all]"                # everything

---

Quick start

from agent_audit import AuditTrail

trail = AuditTrail(run_name="deal-evaluation")

# Manual step
step = trail.begin_step("extraction", "Fact Extraction")
step.log("ARR: $18.2M  [Slide 4]")
step.log("NRR: 118%    [Slide 6]")
step.end("23 facts extracted")

# Context manager
with trail.trace("gate", "Mandate Gate") as step:
    step.log("✓ ARR ≥ $10M — passed")
    step.end("Gate passed")

# Get all entries
for entry in trail.get_entries():
    print(f"[{entry.stage}] {entry.label}: {entry.summary}")

---

With LangSmith

from agent_audit import AuditTrail, LangSmithConfig

trail = AuditTrail(
    run_name="deal-evaluation",
    langsmith=LangSmithConfig(
        api_key="ls__...",
        project_name="my-agent",
    ),
)

Each AuditTrail creates a parent run in LangSmith. Each step becomes a child span with inputs, outputs, tags, and timing — visible in the LangSmith trace viewer.

---

With LangChain / LangGraph

from agent_audit import AuditTrail

trail = AuditTrail(run_name="my-agent")
handler = trail.as_langchain_handler()

# LangChain
chain.invoke(inputs, config={"callbacks": [handler]})

# LangGraph
from langchain_core.runnables import RunnableConfig
graph.invoke(state, config=RunnableConfig(callbacks=[handler]))

The callback handler automatically captures:

• on_chain_start/end → chain steps
• on_llm_start/end → LLM calls with token estimates
• on_tool_start/end → tool invocations

---

API

AuditTrail

Method	Description
begin_step(stage, label)	Start a step, returns AuditStep
trace(stage, label)	Context manager — auto-ends step
get_entries()	All completed AuditEntry objects
elapsed_ms()	Total run time so far
flush(**metadata)	Flush to LangSmith
as_langchain_handler()	Returns LangChain callback handler

AuditStep

Method	Description
log(line)	Append a detail line
end(summary, metadata?)	Commit the entry

AuditEntry (Pydantic model)

class AuditEntry(BaseModel):
    id: str
    stage: str
    label: str
    summary: str
    lines: list[str]
    started_at: float
    completed_at: float
    duration_ms: float
    metadata: dict

---

Architecture

AuditTrail
├── begin_step(stage, label) → AuditStep
│     ├── .log(line)
│     └── .end(summary) → AuditEntry
├── trace(stage, label) → context manager
├── get_entries() → list[AuditEntry]
├── as_langchain_handler() → BaseCallbackHandler
└── flush(**metadata) → LangSmith parent run update
      └── LangSmithAdapter
            ├── Parent run  (one per AuditTrail)
            └── Child spans (one per step)

---

Reference implementation

[nexus] — AI-powered VC deal evaluation. Uses agent-audit across a 6-stage pipeline:

PDF → Fact Extraction → [Gate Check + Enrichments] → Category Scoring ×N → Summary → IC Brief

Each stage writes to an AuditTrail. The final get_entries() is returned to the UI and rendered as a collapsible audit tab where every score can be traced to the source slide quote.

---

Compatibility

Framework	Support
LangChain	as_langchain_handler()
LangGraph	as_langchain_handler() via RunnableConfig
LangSmith	Native Client integration
Anthropic SDK	Manual begin_step / trace
OpenAI SDK	Manual begin_step / trace
Any agent	Zero-dependency core