Exploiter API

Exploiter API is a local document-ingestion and analyst-enrichment framework. It parses common office/document formats, normalizes the extracted text, stores the source material in SQLite, deduplicates files by SHA256, indexes content with SQLite FTS5, and can optionally run LLM translation or analyst enrichment.

The project is designed for a data exploitation analyst who receives mixed files from a collection, case, or inbox and needs to quickly:

• ingest supported files into a searchable case database
• avoid reprocessing exact duplicate files
• search across PDFs, Word documents, spreadsheets, slides, and text files
• translate foreign-language material
• generate analyst-reviewable summaries, indicators, entities, briefs, document type classifications, and timelines
• export saved enrichment results to Markdown

The original extracted document content is preserved separately from generated LLM output. LLM output should be treated as reviewable derived analysis, not as source evidence.

Processing Flow

file or directory
  -> parser selected by extension
  -> normalize extracted text
  -> clean whitespace/unicode artifacts
  -> store in SQLite documents table
  -> deduplicate by SHA256
  -> index with SQLite FTS5
  -> optional translation
  -> optional LLM enrichment
  -> optional terminal display or Markdown export

Repository Layout

Exploiter_API/
  base.py                    Shared REST client base class
  pipeline.py                CLI and ingestion/enrichment entry point
  requirements.txt           Python dependencies
  README.md                  Project overview
  USAGE.md                   Detailed command reference

  Enrichment/
    __init__.py
    enricher.py              Enrichment modes, prompts, JSON parsing

  LLM/
    openai_api.py            OpenAI REST wrapper
    anthropic_api.py         Anthropic REST wrapper

  Parsers/
    pdf_parser.py            PDF text/table/metadata extraction
    docx_parser.py           DOCX paragraph/table/metadata extraction
    excel_parser.py          XLSX/XLS sheet extraction
    pptx_parser.py           PPTX slide/table/metadata extraction
    txt_parser.py            Plain text extraction
    json_parser.py           Standalone JSON-to-DataFrame helper
    normalize.py             Converts parser text shapes to one string
    clean.py                 Tidies extracted text

  Persistence/
    dbCreate.py              SQLite schema, ingest, search, enrichment storage

  Examples/
    openai_usage.py
    anthropic_usage.py

  tests/
    test_pipeline.py

Supported Input Files

The CLI can ingest these file types:

Extension	Parser	Notes
.pdf	PDFParser	Text and tables via pdfplumber, metadata via pypdf
.docx	DOCXParser	Paragraphs, tables, headings, core properties
.xlsx	ExcelParser	All sheets as tab-separated text and tables
.xls	ExcelParser	Legacy Excel support through pandas/xlrd
.pptx	PPTXParser	Text and tables grouped by slide
.txt	TXTParser	Plain UTF-8 text with replacement for invalid bytes

Unsupported files are skipped during directory ingestion. If an unsupported file is passed directly, the CLI raises an unsupported file type error.

Database Output

The primary output is a SQLite database. By default the file is:

exploiter.db

Use --db to choose a different database:

python .\pipeline.py -d .\Inbox --db .\cases\case001.db

Important tables:

Table	Purpose
documents	Source document records, hashes, cleaned original text, optional translation
documents_fts	SQLite FTS5 index over title, original content, translated content
document_enrichments	LLM enrichment outputs stored separately from source content

The documents table includes:

Field	Meaning
id	Document id
file_path	Absolute source path at ingest time
file_type	Short parser type such as pdf, docx, xlsx, txt
title	Parser-provided title when available
Content_Original	Cleaned extracted source text
Content_Translated	Optional translation output
MD5	MD5 file hash
SHA256	SHA256 file hash, unique for deduplication
Notes	Reserved note field
last_modified_by	Parser-provided author/editor metadata when available
ingested_at	SQLite insertion timestamp

The document_enrichments table includes:

Field	Meaning
id	Enrichment id
document_id	Source document id
enrichment_type	summary, entities, indicators, brief, document_type, or timeline
provider	Client class name, such as OpenAIAPI
model	Model id used for the run
prompt_version	Versioned prompt identifier
output_text	Raw/readable model output
output_json	Parsed JSON for structured modes when available
confidence	Model-provided confidence when available
review_status	Defaults to needs_review
created_at	SQLite insertion timestamp
reviewed_by, reviewed_at	Reserved review fields

Installation

From the project root:

python -m pip install -r requirements.txt

For LLM-backed commands, create a .env file in the project root:

OPENAI_API_KEY=...
ANTHROPIC_API_KEY=...

Do not commit real API keys.

Basic CLI Usage

Ingest one file:

python .\pipeline.py .\Inbox\russian_cyber_01.pdf

Ingest multiple files:

python .\pipeline.py .\Inbox\russian_cyber_01.pdf .\Inbox\russian_cyber_01.txt

Ingest a directory recursively:

python .\pipeline.py -d .\Inbox

You can also pass a directory as a positional path:

python .\pipeline.py .\Inbox

Use a case-specific output database:

python .\pipeline.py -d .\Inbox --db .\cases\russian_cyber.db

Search an existing database:

python .\pipeline.py --db .\cases\russian_cyber.db --search "vpn OR password"

Ingest and search in one command:

python .\pipeline.py -d .\Inbox --search "server OR admin"

Translation

Translation is optional. It writes output to documents.Content_Translated and does not overwrite documents.Content_Original.

Translate with OpenAI:

python .\pipeline.py .\Inbox\foreign_report.pdf --translate openai --model gpt-4o-mini

Translate with Anthropic:

python .\pipeline.py .\Inbox\foreign_report.pdf --translate anthropic --model claude-sonnet-4-6

Translate a whole directory:

python .\pipeline.py -d .\Inbox --translate openai --model gpt-4o-mini

--translate requires --model.

LLM Enrichment

Enrichment runs analyst-focused prompts over stored source text and saves the result in document_enrichments.

Supported modes:

Mode	Output type	Intended use
summary	Text	Source-faithful analyst summary
entities	JSON	Named entities with evidence snippets and confidence
indicators	JSON	IPs, domains, URLs, emails, hashes, CVEs, filenames, hostnames, ports
brief	Text	Bottom line, findings, evidence, gaps, confidence, follow-up
document_type	JSON	Likely document type, confidence, rationale, handling notes
timeline	JSON	Dated events with precision, actors, evidence, confidence

Run one enrichment mode:

python .\pipeline.py .\Inbox\russian_cyber_01.pdf --provider openai --model gpt-4o-mini --enrich summary

Run multiple enrichment modes:

python .\pipeline.py .\Inbox\russian_cyber_01.pdf --provider openai --model gpt-4o-mini --enrich summary --enrich indicators --enrich entities

Run enrichment over a directory:

python .\pipeline.py -d .\Inbox --provider openai --model gpt-4o-mini --enrich summary

Run document classification and timeline extraction:

python .\pipeline.py -d .\Inbox --provider anthropic --model claude-sonnet-4-6 --enrich document_type --enrich timeline

--enrich requires --provider and --model, unless --translate is also supplied. If both are used and --provider is omitted, enrichment reuses the translation provider.

Showing and Exporting Enrichments

Show saved summaries in the terminal:

python .\pipeline.py --show-enrichments summary

Show all saved enrichment records:

python .\pipeline.py --show-enrichments

Export saved summaries to Markdown:

python .\pipeline.py --show-enrichments summary --export-enrichments .\summaries.md

Export all saved enrichment records to Markdown:

python .\pipeline.py --export-enrichments .\enrichments.md

Use --db with display/export commands when working with a non-default database:

python .\pipeline.py --db .\cases\russian_cyber.db --show-enrichments summary --export-enrichments .\cases\russian_cyber_summaries.md

Note: in the current CLI, --export-enrichments exports all enrichment records unless it is paired with --show-enrichments <mode> to select a mode.

Example Analyst Workflow

1. Ingest an inbox into a case database:

python .\pipeline.py -d .\Inbox --db .\cases\russian_cyber.db

2. Search for immediate terms of interest:

python .\pipeline.py --db .\cases\russian_cyber.db --search "vpn OR admin OR credential"

3. Generate summaries and indicators:

python .\pipeline.py -d .\Inbox --db .\cases\russian_cyber.db --provider openai --model gpt-4o-mini --enrich summary --enrich indicators

4. Export summaries:

python .\pipeline.py --db .\cases\russian_cyber.db --show-enrichments summary --export-enrichments .\cases\russian_cyber_summaries.md

5. Export all enrichment records:

python .\pipeline.py --db .\cases\russian_cyber.db --export-enrichments .\cases\russian_cyber_enrichments.md

Python API

Ingest a file:

from Persistence.dbCreate import DBCreator
from pipeline import ingest_file

db = DBCreator("exploiter.db")
doc_id = ingest_file("Inbox/russian_cyber_01.pdf", db)
print(doc_id)

Search:

from Persistence.dbCreate import DBCreator

db = DBCreator("exploiter.db")
for hit in db.search("vpn OR password"):
    print(hit["id"], hit["title"], hit["snippet"])

Translate during ingest:

from Persistence.dbCreate import DBCreator
from LLM.openai_api import OpenAIAPI
from pipeline import ingest_file

db = DBCreator("exploiter.db")
client = OpenAIAPI()
doc_id = ingest_file("Inbox/report.pdf", db, client=client, model="gpt-4o-mini")

Run enrichment against an existing document:

from Persistence.dbCreate import DBCreator
from LLM.openai_api import OpenAIAPI
from Enrichment import enrich_document

db = DBCreator("exploiter.db")
client = OpenAIAPI()
enrichment_id = enrich_document(
    db=db,
    doc_id=1,
    client=client,
    model="gpt-4o-mini",
    mode="summary",
)
print(enrichment_id)

List enrichment records:

from Persistence.dbCreate import DBCreator

db = DBCreator("exploiter.db")
for row in db.list_enrichments(enrichment_type="summary"):
    print(row["id"], row["document_id"], row["output_text"])

Development and Validation

Run tests:

python -m pytest -v

Run a syntax check:

python -m compileall Enrichment pipeline.py Persistence\dbCreate.py tests\test_pipeline.py

Show CLI help:

python .\pipeline.py --help

Known Limitations

• OCR is not implemented. Scanned/image-only PDFs may produce little or no text.
• Table data is extracted by parsers but not stored in dedicated relational table structures.
• Large documents are sent to LLMs as one prompt; chunking is not implemented.
• Enrichment output is model-generated and should be reviewed by an analyst.
• --export-enrichments currently writes Markdown only.
• json_parser.py is a standalone helper and is not registered in the main ingestion pipeline.

More Usage Examples

See USAGE.md for a command-focused runbook with additional examples, troubleshooting notes, and output interpretation.