Hakiri

Hakiri is a single static binary that pulls records from your SaaS tools, internal databases, and APIs into a local-first context store agents can query through MCP. Everything lives as Parquet on disk, a SQLite catalog, and TOML manifests — cat-able from a shell, queryable with the duckdb CLI even if Hakiri is uninstalled, syncable to any S3-compatible bucket (R2, S3, MinIO, B2) with no relay in between. Connectors are WASM Components an agent can author in one MCP conversation. Every read is attributable; every memory blob is yours.

What you can do today

• Run a pipeline on a laptop in five minutes. brew install hakiri && hakiri run github-issues writes Parquet under ./.hakiri/ and exits — no signup, no daemon, no cloud account, fully offline.
• Query the store with an agent over MCP. hakiri serve exposes a context.query tool to Claude Code, Cursor, or any MCP client. Structured filters and semantic search land in one call; the runtime plans the join.
• Author a new connector by talking to an LLM. Point an agent at an OpenAPI spec; it produces a working WASM Component connector, dry-runs it against a recorded fixture, and installs it — without leaving the MCP conversation.
• Sync a project across laptops, edge, and cloud through plain object storage. Two Hakiri instances never talk to each other; they talk to the bucket. The bucket is the team-shared context.
• Scope an agent’s reads with capability tokens. Tokens encode {subject, actions, tables, row filter, redact list, ttl}, signed by the project key, verifiable offline. Sub-agents inherit attenuated children without contacting an issuer.
• Deploy the same manifest to a single VM, a 3-node cluster, Cloudflare Workflows + Containers, or AWS Step Functions + Fargate. One binary, three build profiles (hakiri-core, hakiri-full, hakiri-coord), six topologies. Kubernetes is a wrapper, never a requirement.
• Pin where inference runs. Tag a table with inference_zone_allowed = ["local:device", "on-prem:*"] and Hakiri refuses to serve those rows to a cloud zone. An Incognito switch in the MCP client pins a session locally for ad-hoc sensitive work.

How it works

Three commitments shape everything else.

One binary, no orchestrator — laptop to cluster, same artifact

Hakiri ships as a single static Rust binary with no runtime dependencies. No JVM, no Python, no Node, no system packages beyond libc — verified by ldd in CI on a fresh Alpine container. The binary is the install.

Same artifact, six topologies:

• Laptop: brew install hakiri && hakiri run github-issues — fully offline, no daemon, no signup.
• Single VM: hakiri serve behind Caddy or Tailscale Serve, with a systemd unit in the box.
• Team mode (Cloudflare): hakiri team init --cloudflare generates wrangler.toml for a Worker + Durable Object + Cron + Workflows + Containers control plane. An Electron desktop app and a web SPA share a Loro-CRDT-backed config doc; pipelines fire on schedule against the team’s CF account.
• Self-hosted control plane: the hakiri-control Rust binary runs the same control plane offline. The Electron client signs in via OAuth against either.
• Cluster: copy the binary to N VMs, point them at a bundled hakiri coord (Raft-replicated, modeled on ClickHouse Keeper). No external ZooKeeper / etcd.
• Cloud (first-class): hakiri deploy cloudflare lifts the manifest onto Workflows + Containers; hakiri deploy aws onto Step Functions + Fargate. Same TOML, two clouds.

The binary ships in three profiles, hard-gated in CI:

Profile	Target	Budget	Includes
hakiri-core	Lambda zips, edge runtimes, smallest install	≤ 50 MB compressed, ≤ 60 MB idle RSS	Built-in connectors + DuckDB + LWW sync + catalog + Polars
hakiri-full	Daemon serving agent retrieval	≤ 150 MB compressed, ≤ 180 MB idle RSS	core + Wasmtime + Tantivy + HNSW + MCP server
hakiri-coord	Raft coordinator for cluster topologies	≤ 50 MB compressed, ≤ 50 MB RSS per node	Raft KV only — no query engine, no WASM host

A new operator goes from “downloaded binary” to “production sync running on three VMs” in under an hour, without touching Kubernetes, Helm, Terraform, or a cloud console. See specs/06-deployment.md, ADR-0008, ADR-0012.

Agent-authorable, sandboxed connectors via WIT + WASM

Connectors implement a small typed WIT interface — discover, pull, schema, commit-cursor. They run in wasmtime with capability-declared host imports: a connector that needs network access declares it; the operator grants per-connector. An agent-authored connector for “internal payroll API” cannot exfiltrate to evil.com even if the agent was prompt-injected.

• Any language, one artifact. Rust, TinyGo, JavaScript (ComponentizeJS), Python (componentize-py) all compile to a .wasm Component. No subprocess soup.
• Hot-swap, no redeploy. Drop a new .wasm into the catalog; the daemon picks it up on next run.
• Agent-authored is first-class. The built-in MCP server exposes connector.scaffold-from-openapi, connector.build, connector.test, connector.install as tools an LLM calls. The host emits WIT from a fixed template; the agent fills only the Rust impl.
• Provenance recorded. Author (human or agent://claude-research), model, prompt, source spec, fixture, sandbox capability list — all in the catalog, surfaced at run time.
• Drift detection. The catalog stores the connector’s source spec version (OpenAPI hash, SDK commit). hakiri connector check-drift produces a structured diff an agent fixes in one MCP call.
• Inspectable end-to-end. Manifests are TOML against a published JSON Schema; state is SQLite; data is Parquet; traces are OTel. Nothing is a hidden binary blob.

The same WIT contract extends to transforms (ADR-0010) — Polars expressions for the common case, WASM Components for irregular per-batch logic. Detail: specs/02-connectors.md, ADR-0001.

MCP-native context store you can cat and duckdb

The destination is the agent’s context, not an analyst’s dashboard or a vendor’s opaque memory blob. The store is boring storage — Parquet files, a SQLite catalog, JSON/TOML manifests, all cat-able from a shell — surfaced through one MCP tool that unifies structured filters with vector and full-text retrieval.

• One disk layout, no proprietary format. State is SQLite. Data is Parquet. Manifests are TOML/JSON. Indexes are sidecars next to Parquet. A duckdb CLI reads the whole store with Hakiri uninstalled.
• MCP is the only mandatory agent interface. Claude Code, Cursor, custom MCP clients, OpenAI’s MCP support — all see the same tools, schemas, provenance metadata. No claude-only or openai-only code paths.
• Memory lives in the store, not in a vendor blob. Whatever an agent “remembers” lives as Parquet rows — inspectable, queryable with duckdb, exportable as a tar file, deletable per-row. Switching from Claude to Codex changes the model and the MCP client; it does not change the memory.
• Indexes are first-class artifacts, declared in the manifest and rebuilt alongside compaction. Vector (HNSW), FTS (Tantivy), zone-map, PK lookup. The embedding model identifier travels with each vector so a model swap is a single-command rebuild.
• Provenance survives retrieval. Every retrieved row carries lineage edges back to the run, connector, and (if applicable) the agent that authored the connector. The agent cites, not hallucinates.
• No prompt construction inside Hakiri. Hakiri surfaces tools and data via MCP; the agent harness (Claude Code’s, your custom client) constructs prompts. CI-enforced: no Hakiri crate imports a model-vendor SDK.

A team swaps Anthropic → OpenAI → local Llama over a weekend with zero schema migration or data re-ingest. Detail: specs/04-context-store.md, specs/05-agent-interface.md, ADR-0002, ADR-0004.

What follows from those three

The properties below fall out of the three commitments — they are not separate features to maintain.

• Local-first sync over plain object storage. No relay, no sync server. Every install works fully offline; the bucket is the team-shared context. LWW with cursor-kind awareness (ADR-0005); single-writer leases stored in the bucket itself via atomic conditional puts. hakiri sync diagnose probes the bucket and refuses multi-writer mode where conditional puts aren’t supported (R2 ✓, S3 ✓, MinIO partial, B2 ✗). Detail: specs/04-context-store.md.
• Capability-based access control with agents as first-class principals. Capability tokens encode subject tuples (agent, host, on-behalf-of, task-id), signed offline, attenuable, with write-time redaction and read-time row filters. Every read emits an OTel span tagged with the full subject tuple, the token id, and the rows touched — the span trail is the audit log. Detail: specs/09-access-control.md.
• Compute/data collocation. Replicas live wherever the agent runs — laptop SSD, Cloudflare Worker DO, Fargate EFS, on-prem NAS. Sidecar indexes (HNSW, FTS) travel with the data. For tables too sensitive to replicate (replicate = false), a query proxy executes against the canonical store and returns only allowed results. Detail: specs/10-collocation.md.
• Compliance / data-sovereignty posture. The data plane never leaves the customer environment. Secrets never enter agent context. Every read is attributable. CF and AWS first-class deploys offer EU regions; on-prem topology covers air-gapped sites. The OSS data plane provides what an auditor needs to attest against — attestation itself is operator work (or the M3+ commercial layer). Detail: specs/11-compliance.md.
• AI-provider agnostic. MCP-only boundary + operator-chosen embedding model = context portable across model vendors. The CI gate that refuses any LLM-vendor SDK import in any Hakiri crate is the load-bearing discipline.
• Inference placement with Incognito mode. A typed inference_zone on the subject tuple (local:device / on-prem:* / private-cloud:* / public-cloud:*) plus an inference_zone_allowed tag on tables and columns declares where data is allowed to be processed once read. A client-side Incognito switch pins the zone to local:device / on-prem:* for a session; private-tagged context refuses to be served to a cloud zone regardless of which model the user has selected. PHI auto-narrows to the on-prem floor. Hakiri ships no LLM gateway and imports no model-vendor SDK. Detail: specs/15-inference-placement.md, ADR-0015.

Hard problems Hakiri solves

Five things that aren’t “an integration we wrote” — they are problems the architecture exists to answer.

1. Access control with the agent as a first-class principal

Every access control model in widespread use today assumes the principal is a human user (or a long-lived service account). Both abstractions fail for agents: they’re short-lived, derived, situated, and they read more than humans do by orders of magnitude.

Hakiri’s answer:

• Capability tokens, not roles. A token is {subject, actions, tables, constraints (row filter, redact list), ttl}, signed by the project’s sync key, verifiable offline.
• Composable subjects. Subject is a tuple (agent, host, on-behalf-of, task-id), not a flat string. Tokens are attenuated — an agent derives a more-restrictive child token for a sub-agent without going back to the issuer. Wire format: biscuits, because attenuation is in the design, not bolted on.
• Write-time and read-time enforcement. Sensitive columns are redacted at write time (a stolen R2 credential exposes only redacted Parquet); table-level access is enforced at the sync edge; query-time row filters apply at the CLI and at the query proxy.
• Provenance as audit, not as logs. Every read emits an OTel span tagged with the full subject tuple, the token id, and the rows touched. The span trail answers “did anyone look at this customer in the last 24h” in under a second.
• Policy as data. Policies live in the manifest — diffable, PR-reviewable. No console-only policy editor.

A compromised R2 credential exposes only redacted Parquet. An operator issues and revokes tokens in single commands — no Keycloak, no Auth0, no user database. Sub-agents receive attenuated child tokens from their parents without contacting the issuer.

Prior art studied: Macaroons (Birgisson et al., 2014), Biscuits, SPIFFE / SPIRE, Zanzibar (Pang et al., 2019).

Tracked in: specs/09-access-control.md.

2. Connectors authored and maintained by agents, not by people

Airbyte ships 300+ connectors. Fivetran ships 500+. Each took years and continues to consume substantial maintenance because every SaaS API drifts independently. The connector-count race against them is unwinnable on person-hours.

Hakiri inverts the problem: an LLM that reads an OpenAPI spec produces a working connector — given the right contract shape. The hard part is the contract, the test loop, the sandbox, and the maintenance story.

• WIT + WASM Component Model as the contract (ADR-0001). Typed interface, real sandbox, language-agnostic, hot-swappable.
• Scaffold from API descriptions. The MCP server exposes connector.scaffold-from-openapi, connector.scaffold-from-asyncapi, connector.scaffold-from-sql. Given a spec URL, the agent produces a working WASM component.
• Dry-run against samples. Every connector ships with a recorded fixture under fixtures/<connector>/sample.json. hakiri connector test replays without hitting the live API — the inner loop an authoring agent uses to iterate without burning rate limits.
• Drift detection. Catalog records the connector’s source spec version. When upstream changes, hakiri connector check-drift produces a structured diff an agent fixes in one MCP call. Maintenance is a recurring agent task, not a recurring human task.
• Capability-declared host access. Inherits the capability model from problem (1). An agent-authored connector cannot exfiltrate even under prompt injection.
• Connector provenance recorded. Author, model, prompt, source spec, fixture, sandbox capability list — all in the catalog. PR review is on the diff, not on imperative state-machine code.

An agent reads a public OpenAPI spec, produces a working connector with passing dry-runs in one MCP conversation, and submits a PR a human reviews as a diff. When the API drifts, drift detection produces a structured diff the agent fixes the same way. A regulated customer adds an internal-API connector themselves, sandboxed, without sending the spec to any third party.

Tracked in: specs/02-connectors.md, ADR-0001.

3. A semantic context layer agents can query in one call

Agents need both structured and semantic context, jointly. “Find recent customer support tickets from accounts at risk of churn, summarized” crosses structured (account.risk_score, ticket.created_at) and semantic (ticket text similarity) at the same time. Existing tooling forces the agent to do two queries against two surfaces and join in its head.

Hakiri exposes one unified surface:

• Indexes are first-class artifacts in the catalog. A table declares [indexes] blocks: vector { column = "body", model = "bge-large", dim = 1024 }, fts { columns = ["title", "body"], analyzer = "english" }. Sidecars next to Parquet, rebuildable from the canonical store.
• One MCP tool surface, joined under the hood. context.query takes a structured filter and a semantic query in the same call. The runtime decides plan order (filter then ANN, ANN then filter, hybrid) per declared statistics. Agents do not see “the vector store” and “the SQL store” as separate surfaces.
• Embedding model lives in the catalog. Switching models is hakiri index rebuild --model bge-large-en-v1.5, not a rewrite. The model identifier travels with each vector.
• Provenance through retrieval. Every retrieved passage carries its source row’s lineage edge — which run, which connector, which agent authored that connector.
• Policy through retrieval. Capability tokens from problem (1) filter retrieval results at the sync edge: an agent without read:customers.email does not see passages containing emails, regardless of semantic match.
• Schema for the agent, not the analyst. Tables expose an agent_description field — natural-language column descriptions and example queries the MCP tool surfaces at discovery time. The agent gets the same orientation a new hire would get from a README.
• Shard by access pattern. Manifests declare access_pattern = "recent_90d" / "by_account" / "by_repo"; the writer partitions Parquet so a retrieval scoped to a slice doesn’t drag the whole table.

An agent answers “what’s the renewal risk on account X” in one MCP call that crosses structured and semantic dimensions, with the join planned by the runtime. Cited passages carry provenance back to the originating row. An agent without read:customers.email capability does not see passages containing emails.

Tracked in: specs/12-semantic-context.md. Sidecar discussion: specs/04-context-store.md.

4. Context and memory independent of any LLM vendor

Every major model vendor and every popular agent framework ships a memory product that couples your data to their runtime: OpenAI Assistants memory, Anthropic projects, LangChain memory, LlamaIndex storage, Mastra workflows, CrewAI knowledge. Adopting any of them makes the context layer a function of that vendor’s API, roadmap, residency, and lifecycle.

Hakiri makes context and memory infrastructure — boring, inspectable, owned by the customer, replaceable below the agent.

• MCP is the only mandatory agent interface. No claude-only or openai-only code paths.
• Context store is the source of truth for memory. Whatever an agent “remembers” lives as Parquet rows — inspectable, queryable, exportable, deletable per-row. Not in a vendor’s opaque blob.
• Embeddings are operator’s choice. The embedding model identifier is stored next to each vector; swapping is a one-command rebuild. OpenAI, Voyage, Cohere, Jina, local bge-*, custom — all peers.
• No prompt construction inside Hakiri. Hakiri surfaces tools and data via MCP; the agent harness constructs prompts. There is no “Hakiri prompt format” to become a coupling surface.
• No vendor-specific tool shapes. MCP’s JSON Schema is the contract.
• Local inference is a first-class peer. llama.cpp, vllm, an internal gateway, an EU-resident provider — same Hakiri surface.
• CI-enforced. PRs that import an LLM-vendor SDK into any Hakiri crate get refused at the gate.

A team swaps Anthropic → OpenAI → local Llama over a weekend with zero schema migration, manifest rewrite, or data re-ingest. The memory and retrieved context are byte-identical; only the model endpoint and MCP client change. The same agent can be routed across providers per-task (triage on cheap, reasoning on expensive) by the client harness, without Hakiri knowing.

5. Genuinely small footprint, genuinely self-hostable, genuinely local-first

“Self-hostable” and “local-first” are claims almost every infrastructure project makes and almost none deliver. The usual failure modes — five binaries hiding behind one, 500 MB Docker images, a control plane that requires SaaS signup, sync that needs a stateful relay, deploys that degrade discontinuously past three nodes, memory creep — disqualify any tool a compliance-constrained or silo-unifying team can adopt.

Hakiri holds the line:

• Three build profiles, three budgets, hard-gated in CI. Lambda-cold-start budgets cannot coexist with the agent retrieval surface (HNSW + Tantivy + Wasmtime) in one binary — the honest split is hakiri-core (≤ 50 MB compressed), hakiri-full (≤ 150 MB), hakiri-coord (≤ 50 MB). See ADR-0012.
• No runtime dependencies. ldd $(which hakiri) on a fresh Alpine container shows only libc.musl-*.so.
• One process is the whole product per profile. hakiri serve is daemon, scheduler, WASM host, MCP server, sync engine, and (optionally) coordinator.
• Sync over plain object storage, no relay. Push and pull go directly to any S3-compatible bucket. The protocol is documented (Parquet + JSON manifests with a content-hash index). Two Hakiri instances never talk to each other.
• S3-compatible feature matrix is honest about capability variance. hakiri sync diagnose probes the bucket and refuses multi-writer mode where atomic conditional puts aren’t supported. R2 ✓, S3 ✓ (since Nov 2024), MinIO partial, B2 ✗, Garage recent, SeaweedFS spotty.
• Local-first means actually offline. No license check, no telemetry, no anonymous-usage ping, no auto-update probe, no “phone home for a token” step.
• Conflict resolution without a server. LWW with cursor-kind awareness (ADR-0005); single-writer leases stored in the bucket via atomic conditional puts.
• Scale-out without an orchestrator. Going from 1 node to 10 is scp + systemctl start, not a Helm chart.
• Cold-start budget. hakiri-core Lambda cold start ≤ 200 ms. CF Container (hakiri-full) cold start ≤ 8 s (runtime-set; Hakiri keep-warms). Laptop hakiri run start-to-first-record ≤ 1 s for a built-in connector against a sample.
• Memory budget per pipeline. A pipeline with 100k-row batches sustains ≤ 50 MB RSS added to the daemon’s idle. Streaming Parquet writes, bounded-arena Arrow batches, no “hold the table in memory” anti-patterns.

A developer runs curl ... | sh on a 2018 laptop and hakiri run github-issues produces Parquet in under a minute. A 3-node hakiri coord cluster on three $5/month Hetzner VMs runs continuously under 50 MB RSS each. Two laptops sync correctly through a shared MinIO bucket with no other process running. Memory at day 7 of soak = day 1 ± 5%.

Prior art studied: ClickHouse (single-binary OLAP with embedded Keeper), Caddy (single-binary HTTPS with zero config), SQLite, Litestream / rqlite, Ink & Switch’s local-first work, Automerge / yrs.

Tracked in: ADR-0005, ADR-0008, ADR-0012, specs/06-deployment.md.

How they compose

The five problems aren’t independent. Capability-declared host access for connectors (2) inherits the token model from (1). Retrieval results in (3) are filtered by capability tokens (1) before reaching the agent. Connectors (2) author the indexes the semantic layer (3) serves. The semantic layer (3) is the load-bearing answer to “what should memory look like if it’s provider-agnostic” (4). Capability tokens (1) verify offline against a manifest public key — the policy plane is independent of the model plane (4). And the footprint discipline (5) is the constraint that keeps the other four honest: any solution needing a JVM or a separate Postgres is not a solution.

What Hakiri isn’t

• Not a warehouse replacement. Snowflake, BigQuery, Redshift remain better destinations for BI/Looker/Sigma. Hakiri’s destination is the agent’s context, not the analyst’s dashboard.
• Not a managed cloud product. A hosted control plane is an optional M3 add-on. The data plane never leaves the customer’s environment.
• Not a streaming engine. Sub-second streaming joins belong to Materialize, RisingWave, Arroyo. Hakiri is record-batch oriented.
• Not a connector marketplace with billing. OSS only. Distribution via OCI registries or plain HTTPS.
• Not an LLM SDK or prompt framework. Hakiri does not call LLMs, construct prompts, pick models, or wrap a vendor SDK. The MCP boundary is the contract.
• Not a vendor-locked memory product. Memory is Parquet + a vector sidecar. Hakiri will not adopt OpenAI Assistants memory shapes, Anthropic project shapes, or any framework-specific store as its native format.

Who uses Hakiri

Primary: teams unifying many data silos for an internal agent

Modern operations run on dozens of SaaS systems — sales in Salesforce or HubSpot, support in Zendesk or Intercom, eng in GitHub + Linear + PagerDuty, knowledge in Notion or Confluence, finance in Stripe + NetSuite, comms in Slack, plus internal Postgres/MySQL and one or two industry-specific apps. Agents and analysts repeatedly hit the same wall: the answer requires three or four of these systems joined together, and there is no place that join exists.

These teams have technical buyers who self-serve OSS. Hakiri’s fit: agent-authored WASM connectors repaired in one conversation when an API drifts; MCP-native context store where joins across silos are SQL the agent already knows; local-first sync so the union of silos lives next to the agent; capability tokens that scope an agent to “the slices it should see across silos.”

Strategic: teams with data-compliance constraints

Healthcare networks, financial-services firms, public-sector and defense buyers, EU-resident SaaS vendors, legal and audit firms, biotech and pharma — anyone for whom the answer to “can we send this to a third-party SaaS to land it in a warehouse?” is no, by law, contract, or board policy.

The architecture choices that serve the primary audience — single binary, data plane in customer environment, capability-token ACLs, cat-able audit trail — happen to be the architecture this audience needs. The OSS data plane gives them what an auditor needs to attest against; the commercial layer (specs/internal/commercial-strategy.md) provides the entity, SLAs, and attestation pack on top.

Other shapes that fit

Persona	Primary need	Hakiri fit
Agent builder (solo or small team)	A queryable context layer for an LLM, authored by the LLM	MCP server + WASM connectors + DuckDB query face
Solo analyst	Offline-first ETL into Parquet on a laptop, optionally syncable	Single binary + local-first store + opt-in R2 sync
Small platform team (3–10 eng)	Multi-node sync layer without an SRE org	hakiri coord + Postgres catalog + systemd, no K8s
Sovereign / air-gapped deployment	Run on-prem with no outbound network	Static binary + local sync (MinIO/Garage) + offline-verifiable tokens
Cloud-native team on CF or AWS	Lift the same manifest onto managed primitives	hakiri deploy cloudflare / hakiri deploy aws
Multi-location agent operator	One agent running locally, at the edge, and in a VPC — same context	Indexed replicas collocated with each runtime; shard-by-access-pattern manifests
Multi-model platform team	Run Claude, Codex, and local Llama against the same context	MCP-only agent boundary, vendor-agnostic vector indexes

Hakiri is not for enterprise BI teams replacing Fivetran for Looker/Sigma/Hex pipelines.

How people use it

Eng-metrics agent across GitHub + Linear + PagerDuty

A platform team of 5–15 engineers runs an internal agent that answers questions like “which on-call incidents in the last 90 days map to which Linear issues, and which PRs eventually closed them?” — questions that used to take 45 minutes of human stitching across three tabs.

The engineer who owns it runs hakiri run on a laptop. The team promotes it to a single VM (hakiri serve on a $20/month Hetzner box) once they adopt it; a replica on a CF Worker drives a Slack-bot front-end so anyone in the org can /ask hakiri .... Sources: GitHub Issues + PRs + Reviews (OpenAPI), Linear (GraphQL with schema export), PagerDuty Incidents (OpenAPI). Three agent-authored connectors, each scaffolded from the public spec in a single MCP conversation. Drift detection picks up upstream API changes; the agent fixes the connector in another MCP conversation.

Incident-retro agent across Sentry + Datadog + GitHub + Slack

An SRE team runs postmortem assistance grounded in actual telemetry, code changes, and chat history. “What changed in the 30 minutes before the 14:02 spike on May 7th?” produces a sourced answer citing specific Sentry events, Datadog alerts, GitHub commits, and Slack messages by ID.

Shape: shared daemon (hakiri serve) inside the company’s VPC; replica on each SRE’s laptop for offline post-incident review. Sources: Sentry (REST), Datadog (REST with query DSL), GitHub Events, Slack Conversations API. Vector + FTS sidecar indexes on the unstructured fields (Sentry messages, Slack texts) drive semantic recall (“find similar past incidents”) with provenance back to the original record.

Other shapes deployed in the field

• Customer-360 across Salesforce + Stripe + Zendesk + product Postgres — B2B SaaS CS teams. Capability-token scoping per CS rep.
• Research / RFP agent over Notion + Confluence + Drive + a CRM — consulting firms. Local bge-large embeddings so client confidentiality is preserved before any data-residency conversation.
• EU-resident SaaS knowledge assistant — GDPR data-residency, regional cloud pinning, customer-held KMS keys.
• Clinical-trial agent over de-identified EHR — HIPAA, write-time redaction, scoped-per-study tokens.
• Public-sector / sovereign air-gapped deployment — MinIO, hakiri coord, on-prem llama.cpp, no public-internet path.
• Financial-services compliance reporting — PCI-scoped VPC, per-audit-engagement scoped tokens, OTel-audit trail as evidence.

The thread across all of these: the agent’s questions span silos and the answers must be attributable. Hakiri is the layer that makes both true without sending data to anyone the operator hasn’t approved.

How Hakiri compares

Hakiri sits at the intersection of three categories (ELT, agent memory, MCP integration) and no existing product covers all three. The competitors are the neighbors in each. The ETL-specific deep-dive (dlt, Airbyte, Meltano, Fivetran, Cloudflare Pipelines) lives in specs/07-comparisons.md; summary below.

Open-source ELT / ETL

• dlt (Apache-2.0, Python library) — closest spirit competitor: small, schema-inferring. Python-coupled, no MCP, no sandbox, no built-in context store. Picked when the team is Python-shop and the destination is an existing warehouse.
• Meltano + Singer (MIT) — declarative ELT over JSON-lines stdio. Inspires Hakiri’s WIT/WASM model but has throughput ceilings and no sandbox.
• Estuary Flow (BSL) — real-time CDC + ELT. Streaming-first (Hakiri is not), warehouse-shaped destinations.
• Apache NiFi / SeaTunnel (Apache-2.0, JVM) — heavy footprint, GUI-centric. Picked when an org already runs them.
• Talend / StreamSets / Hevo — older ELT platforms, mostly heavy GUI/SaaS.

SaaS ELT

• Fivetran, Stitch, Matillion, Hevo, Rivery, Airbyte Cloud — managed connector-as-a-service. All ship data to their infrastructure first. Disqualified for the compliance-constrained primary audience by construction.

Agent memory frameworks (OSS layer)

• LlamaIndex (MIT, Python) — RAG framework with storage abstraction. Library-shaped, no sandbox, no capability model, no sync layer.
• LangChain Memory / LangGraph state — memory inside an agent framework. Coupled to LangChain.
• Mem0 (Apache-2.0 + hosted) — vector + graph memory, REST API. Closest pure-memory competitor; hosted-first, memory-only (no source ingestion), no capability tokens, no local-first sync.
• SurrealDB Spectron — agent memory integrated into SurrealDB. Same critique of middleware Hakiri makes, different answer: Hakiri stores files on object storage; Spectron stores in a database. See ADR-0004.
• Zep (Apache-2.0 + hosted) — long-term memory with a knowledge-graph layer. Server-shaped (needs Postgres + a Zep service), no source ingestion.
• Letta (formerly MemGPT) — memory-architected agents. Agent runtime, not a context layer.
• Cognee (Apache-2.0) — knowledge-graph + vector memory pipeline. Closer to RAG-construction than silo-unifying ELT.

Vendor-locked agent memory

• OpenAI Assistants memory / Vector Stores / Files API — coupled to the OpenAI account.
• Anthropic Projects + Project Knowledge — coupled to Anthropic.
• Cohere RAG, AWS Bedrock Knowledge Bases, Vertex AI grounding — each ties data to a vendor’s inference plane.
• Cursor / Windsurf / Cline project context — IDE-resident, coupled to that editor.

These are the closest “you don’t need Hakiri, just use the platform’s memory” answers — and exactly the lock-in the MCP-only commitment refuses.

Per-tool MCP servers

The modelcontextprotocol/servers reference set + vendor-maintained servers (Block, GitHub, Cloudflare, AWS) cover one server per source. They work at the demo stage and degrade as soon as the agent needs to join across sources, audit a read, redact a field, or refresh incrementally. Hakiri is complementary at the MCP layer and a replacement for the data layer: a Hakiri pipeline ingests from the same sources those servers wrap, lands them in a unified context store, and exposes one MCP surface that joins across them.

Vector / context storage neighbors

• Turbopuffer (managed) — serverless vector storage on object storage. Closest in spirit to the Parquet + sidecars on R2 model. Storage-only.
• LanceDB (Apache-2.0, embedded) — embedded vector DB. Could back a Hakiri vector index sidecar.
• Pinecone, Weaviate, Qdrant, Chroma, Milvus — vector databases. Storage primitives. Hakiri uses a vector index (DuckDB vss or LanceDB sidecar); it does not aim to be one.
• Postgres + pgvector — cheap vector option. Operationally heavy compared to Parquet + sidecar.

Adjacent embedded / single-binary data stacks

• DuckDB / clickhouse-local / chDB — embedded analytical engines. Hakiri uses DuckDB (ADR-0004); these tools alone don’t solve ingestion, sync, policy, or agent boundary.
• MotherDuck (managed DuckDB) — different deploy model, disqualified for compliance-constrained audiences.
• Tinybird (managed ClickHouse) — different deploy model.
• Bauplan / DuckLake — Python-pipelines-on-object-storage. The data-layout thesis overlaps; Hakiri differentiates upstream of storage (agent-authored connectors, MCP retrieval surface, capability-token enforcement, single-binary deploy that includes ingestion). Pick DuckLake/Bauplan when you already have ingestion and a control plane; pick Hakiri when you don’t want to assemble ingestion + storage + agent surface + policy from four products.

Orchestrators with embedded ingestion

• Dagster Embedded ELT (Apache-2.0) — adds ingestion to Dagster. Reasonable in-place choice if Dagster is already the orchestrator.
• Prefect, Airflow, Temporal, Restate — workflow orchestrators. Hakiri runs inside a workflow step on these, or replaces them at small scale via the in-process scheduler.

What makes Hakiri non-substitutable

For any single category above, there is a better tool. The gap Hakiri fills is the intersection: ELT + agent memory + per-source ingestion + capability ACLs + single-binary + local-first sync + provider-agnostic. Buyers in the two primary audiences cannot assemble that intersection from existing parts without writing the glue themselves and accepting the operational cost of running 4–6 services. The assembly is the product.

Where Hakiri goes next

The phased milestone breakdown — M0 walking skeleton, M1 daemon + MCP + sync + team mode, M2 cloud parity + agent authoring loop, M3 hosted control plane + commercial layer — lives in specs/pm/roadmap.md with explicit success criteria that double as acceptance tests.

The internal notes on commercial sequencing live in specs/internal/commercial-strategy.md. Open architectural questions tracked separately in specs/internal/open-questions.md.