Back to overview System architecture

The complete architecture behind AMOS FLOW

A self-hosted, single-tenant, air-gap-first automation platform. A deterministic core you can audit, an opt-in AI layer you can govern, and a durable execution backbone that never loses a run. This page covers the whole system, end to end.

Single-tenant, on your infrastructure Air-gap-first, zero phone-home Durable execution on Temporal Hash-chained audit of every action
01 / Principles

Six architectural choices that fit together

Each principle solves a specific enterprise constraint. Together they let AMOS scale without per-bot fees, run fully offline, and stay auditable while still using AI where it adds value.

Control-plane / stateless-runner split

A stateful "brain" (orchestrator, durable store, vault, gateway) and stateless worker hosts that pull immutable, versioned workflow artifacts. Delete or recreate a runner without losing a run.

Durable-execution backbone

Every run is a Temporal workflow execution with an append-only event history, deterministic replay and idempotency caching. Partial-completion and runner crashes become recoverable events, never repeated side effects.

Hybrid deterministic core, opt-in agents

Orchestration is a declarative, replayable graph (closed-vocabulary JSON DSL). All non-determinism (LLM calls, agentic reasoning) is locked inside typed steps. AI is invited in only where it helps.

Event-driven backbone

An internal event bus (NATS JetStream, CloudEvents) decouples trigger ingestion, observability fan-out and outbound integration from the synchronous execution path.

Config-as-Code + GitOps

Workflows, agent definitions, config bundles, triggers and model definitions are versioned Git artifacts and the single source of truth. Every change, including a self-heal fix, is reviewable like a pull request.

Air-gap-first, zero phone-home

No outbound telemetry by default. Bring-your-own LLM only (no bundled weights). Install, upgrade and patch through signed offline bundles and a private registry mirror.

02 / System context

One trust boundary, three ways out

The whole system sits inside the customer's single-tenant perimeter. People act through a single surface, the Operator Console. Only three traffic types are allowed to cross the boundary.

CUSTOMER TRUST BOUNDARY · single-tenant · air-gap-capable

Actors (IdP-authed)

Business Analyst, authors workflows from SOPs and chat
Operator, runs and watches, handles exceptions
Administrator, install, RBAC, fleet, upgrades
Approver, signs off HITL and risk-tier gates
Auditor, reads the hash-chained trail
Service accounts, REST /api/v1 (OAuth2, mTLS)
AMOS FLOW AI-native automation platform
self-hosted · single-tenant

External systems

BYO LLM endpoint (vLLM, OpenAI-compatible)
Identity Provider (SAML/OIDC + SCIM)
Target apps (web, Windows desktop, REST)
Customer Vault / KMS / HSM
ITSM, egress-gated (off in air-gap)
SIEM export, egress-gated

Allowed egress: calling the BYO LLM, federating with the IdP, and acting on target apps. Webhook-OUT to ITSM and SIEM export are egress-gated and off by default in air-gap. The IdP can validate tokens offline through cached JWKS.

03 / Containers

The system, layer by layer

From authored source of truth, down through the control plane and durable backbone, out to a stateless runner fleet that touches your target systems outbound-only.

AuthoringConfig-as-Code + GitOps
Git source of truthDSL · config · agents · triggers · models
AI Authoring pipelinecomprehend, clarify, author, self-test, present
AI-Quality promotion gatev2 ≥ v1 else block, signed cert
Control planesingle-tenant · HA ≥3 node · air-gap-first
Platform API / OrchestratorREST /api/v1, RBAC+ABAC
Config servicelayered → Effective Config (hash-pinned)
Orchestrator Agentfleet supervision, self-heal
MCP / Skill Gatewayscope-RBAC, version-pin, DLP
Credential Vaultenvelope encryption, JIT
Governancefolder isolation, hash-chained audit
ObservabilityOTel, OpenSearch, VictoriaMetrics, Langfuse
Trigger subsystemcron, webhook, manual, deferrable
Event busNATS JetStream, CloudEvents
Durable execution backboneTemporal · single source of truth for state
Temporal clusterevent history, replay, idempotency
Workflow-Agent workersDSL interpreter, not codegen
Self-heal enginedetect → diagnose → recover → write-back
Stateless runner fleettwo-class · outbound-only · pull model
Windows desktop runnerpersistent interactive session, FlaUI
Headless runnerephemeral, gVisor sandbox, Playwright
Document-AI GPU runnerOCR, layout, table extraction
External, customer-ownedacross the trust boundary
BYO LLM (vLLM)no bundled weights, offline
Target systemsweb · Windows GUI · API · documents
Identity ProviderSAML/OIDC + SCIM
Customer Vault / KMSBYO-vault facade
04 / Components

Every logical component, grouped by layer

Each component has a clear responsibility, a typed interface, and an open-source technology behind it. Nothing is a black box.

Control plane

Platform API / Orchestrator

One unified REST API across every pillar, with RBAC + ABAC and folder isolation. Starts workflow executions, dispatches the fleet, runs the self-heal loop.

GoOpenAPI 3.1Keycloak

Config resolver (Effective Config)

At dispatch, deep-merges layered config (base → environment → trigger) into one immutable, content-hashed snapshot pinned to the run. Fail-closed on missing bindings.

GoGitCEL
Execution

Interpreter workflow

A deterministic Temporal workflow that reads the DSL as input and dispatches each node as an activity. Change the DSL and it runs at once, no rebuild.

TemporalDSL JSON

Workflow-Agent runtime

Per-run guardrails: iteration and token budgets, tool allowlists, mandatory human checkpoints, autonomy level. Verifies tool receipts so an agent cannot fake success.

Temporalvalidator

Activity workers

Run every side-effect step and every LLM call as a Temporal activity with its own retry policy and idempotency-keyed result caching.

Temporallease fencing

Self-heal engine

A closed loop: detect, diagnose, recover, validate, decide, write back. Multi-strategy recovery from cheap to costly, fixes written back as a reviewable DSL diff.

UI-TARS visionTemporal patching

Agentic-step harness

Opt-in agentic step: a Pydantic AI agent running inside an activity, returning a typed result. The rest of the run stays deterministic and replay-safe.

Pydantic AIMCP SDK

Work-queue manager

Durable transactional work queue with retry, dead-letter, priority and infinite pause-for-approval at zero idle runner cost.

TemporalNATS
Runner fleet

Runner agent (outbound-only)

Enrolls with a one-time token, holds an mTLS identity, heartbeats capabilities, pulls leased jobs. Opens no inbound port into the customer network.

mTLSWebSocket

Windows interactive session

Manages a real logged-in desktop session for legacy GUI automation, with auto-login from the vault and a pinned display resolution.

FlaUIUIAutomation

Headless gVisor runner

Browser, API, file and DB work in a clean sandbox per session, autoscaling to zero. AI-generated code runs inside gVisor or a Firecracker microVM.

PlaywrightgVisor

Capability x zone dispatcher

Matches required capability and identity to an eligible runner pool with load balance and failover. Fails fast and closed when no zone-matched runner exists.

pool targeting

Document-AI GPU runner

An IDP pipeline (OCR, layout, table, post-process) on dedicated GPU runners, with a pluggable engine and confidence-thresholded human review.

PaddleOCRDocling
AI layer (BYO, govern and gate, never host or train)

Authoring pipeline

A durable five-stage workflow: comprehend to an intermediate form, clarify, author DSL, self-test with a separate grader, present with a reliability report, then a mandatory human gate.

Pydantic AIXGrammar

AI-Quality gate-runner

Runs candidate v2 and incumbent v1 on the same golden dataset on your own LLM, with dual grounding. Verdict passes only if v2 is at least v1 on every gated metric, else it blocks.

promptfooHHEM-2.1cosign

LLM provider gateway

One abstraction so authoring, self-heal and agentic steps run against cloud or self-hosted vLLM unchanged. Air-gap loads models fully local with pinned versions.

vLLMXGrammar

Example library / retriever

A per-tenant, versioned library of SOP-to-IR-to-DSL triples, reused as golden datasets for the gate and regression fixtures for self-test. Strictly tenant-scoped.

embedding kNN
Human and UX

Operator Console

The single human surface: authoring chat, a deterministic read-only diagram, fleet and run monitoring, and the approval inbox. The only place approvals complete.

assistant-uiReact Flow

Human-task service

One typed node for approval, form and document-validation, all compiling to a Temporal suspend-and-signal pattern with SLA timers and a cross-workflow inbox.

TemporalRJSF

Channel adapters

Web Chat is the air-gap default and approval surface. Slack, Telegram and Teams are connected-only and notify-and-deep-link, never approve in-channel.

assistant-uiSlack Bolt
Cross-cutting

Vault

Envelope encryption with only ciphertext at rest, just-in-time in-memory decryption on the runner, never written to disk or logs. Pluggable BYO-vault facade.

Google TinkOpenBao

Audit hash-chain

An append-only, SHA-256 hash-chained trail of every human and agent action, including LLM prompts and responses. Exports to SIEM, never transits to the vendor.

SHA-256SIEM export

Observability pipeline

One OTel collector redacts sensitive fields before storage, then fans out to OpenSearch, VictoriaMetrics and Langfuse across six telemetry domains.

OpenTelemetryLangfuse

Identity / RBAC

Two-tier RBAC plus optional ABAC, with folder isolation as the hard multi-tenant boundary. IdP federation via SAML/OIDC and SCIM, offline JWKS validation.

KeycloakSPIFFE

MCP / Skill Gateway

Every tool call passes through a governed gateway: per-workflow scope allowlists, tool-description hashing and rug-pull guard, outbound DLP, cross-server chain detection.

MCP SDK

Licensing

Offline, signed, flat per-edition licensing locked to a cluster fingerprint. Distributes entitlements with no internet. Never bills per bot or per run.

Ed25519offline
05 / Runtime flows

How the core flows run

Three flows cover the lifecycle: authoring a workflow, executing it durably, and recovering when a UI changes. Every side effect is a retried activity, and durable state lives only in the event history.

AI authoring pipeline
ComprehendSOP or chat to IR
Clarifyresolve gaps
AuthorIR to valid DSL
Self-testgraded, pass^N
Presentdiagram + report
Approvemandatory human gate
Workflow execution
Triggercron, webhook, manual
Resolve confighash-pinned snapshot
Interpretdeterministic workflow
Activityside-effect or LLM
Validatepost-step gate
Durable historyreplay-safe state
Self-heal loop
Detectstep or post-cond fail
Diagnoseanchors + snapshot
Recoverfallback to vision
Validatepost-condition
Decidesuggest or auto
Write backreviewable DSL diff
06 / Data architecture

Polyglot persistence, each store for its job

Git is the source of truth for logic and config. The Temporal event history is the source of truth for execution state. Heavy blobs are passed by reference, and secrets never leave the vault as plaintext at rest.

StoreHoldsWhy
GitDSL, config bundles, agent and trigger definitions, model definitions, promptsSource of truth, reviewable like a PR, versioned, diff-able, replayable
PostgreSQLControl-plane catalog and metadata, Temporal persistence, hash-chained auditRelational, transactional, strong RBAC filtering, HA via replication
TemporalRun state, event history, signals and timers, durable agent stateDurable execution: deterministic replay, idempotency, safe self-heal patching
OpenSearchTraces, logs, full execution trace, cross-workflow task indexSearch and incident investigation, lifecycle-managed retention
VictoriaMetricsMetrics: cost per run, success rate, latency, tokens, ROI countersCompressed time series with configurable retention and downsampling
LangfuseLLM-observability traces, eval datasets, judge scores, prompt planeLLM-domain observability and evaluation, telemetry disabled for air-gap
Object storeFiles, screenshots, document-AI input and output, backups, offline bundlesPass-by-reference keeps the event history small and replay-safe
Vault storeSecret ciphertext and wrapped data keys, versioned master key-ringEnvelope encryption, only ciphertext at rest, KMS or HSM auto-unseal
NATS JetStreamCloudEvents, run events, approval and alert events, webhook-OUT layer 1Durable 1:N fan-out and dead-letter, always-on in-perimeter

Secret

AES-256-GCM envelope encryption, never plaintext at rest, just-in-time decrypt, wiped at teardown.

Audit

Append-only, tamper-evident hash-chain, 12 to 24 months plus legal hold, integrity preserved after restore.

PII

Redacted at capture and at the collector before storage, with DSAR export and erasure support.

Telemetry

Redacted and per-tenant scoped, lifecycle-managed with a disk-pressure guardrail so a full disk never crashes the platform.

07 / Deployment

One artifact, three topologies

The same offline-capable bundle and guided installer serve a single node, a high-availability cluster, or a fully air-gapped install. Moving between them never requires repackaging a workflow.

AspectSingle-nodeHA (multi-node)Air-gapped
ShapeAll control-plane on one node, co-located stores3+ nodes across 3+ zones behind a load balancer, replicated DBSingle-node or HA, but zero egress
NetworkInternet or localInternet or internalNo internet
AvailabilityNo HA commitment99.9% control-plane uptimePer the base tier
DRBackup and restoreActive/passive to active/activeBase tier plus offline restore
Best forPOC, pilot, dev and testProduction, regulated, high SLAFinance, healthcare, government

Outbound-only runners

Runners enroll, heartbeat and pull leased jobs over mTLS. The control plane never connects into a runner, so runners can live in isolated zones.

Signed offline bundle

Helm plus Zarf image, chart and license tooling, verified with cosign, mirrored into a private OCI registry. No image pulls reach the internet.

Zero-downtime upgrades

Durable execution plus Temporal worker versioning let in-flight runs finish on the old worker while new runs route to the new one. Rollback is a drain.

08 / Security

Deny by default, defense in depth, everything auditable

Least privilege at every layer, a semi-trusted AI wrapped in typed steps with guardrails, sandboxes and validators, and an immutable record of every action that never leaves your infrastructure.

Identity and access

Two-tier RBAC plus ABAC, folder isolation as a hard boundary, SAML/OIDC and SCIM federation, offline JWKS, service accounts via OAuth2 and SPIFFE.

Secrets

Envelope encryption with only ciphertext at rest, just-in-time in-memory decrypt on the runner, instant per-runner revocation, BYO-vault facade.

Sandbox isolation

AI-generated and semi-trusted code runs in gVisor or a Firecracker microVM, never a bare container, with CPU, memory, wall-clock and egress quotas.

Governed tool access

Every MCP tool call passes a gateway with scoped allowlists, version pinning, tool-description hashing, rug-pull guard and outbound DLP.

Immutable audit

SHA-256 hash-chained trail of every human and agent action, including LLM prompts and responses. Tampering breaks the chain and is detected on verify.

Air-gap and egress

No phone-home, BYO LLM only, egress-gated webhooks off by default, signed offline import for models, patches and licenses.

SOC 2 CC7/CC8EU AI ActNIST AI RMFISO 42001GDPR
09 / Quality attributes

Non-functional goals mapped to tactics

The quality attributes that drive enterprise adoption, each tied to the architectural tactic that delivers it.

Quality attributeArchitectural tactic
Reliability / durabilityTemporal event history and replay, idempotency caching, post-step validator, lease fencing tokens
SecurityEnvelope encryption with JIT decrypt, KMS/HSM auto-unseal, gVisor sandbox, MCP gateway with DLP, outbound-only runners
Air-gap / sovereigntyBYO LLM, offline Helm plus Zarf bundle, private OCI mirror, offline signing, egress-gated webhooks
Scalability without per-botControl-plane and stateless-runner split, two-class fleet, queue-depth autoscale to zero, capability and zone targeting
AuditabilityImmutable hash-chained audit of every action, SIEM export, full reproducible run trace, model version pinning, signed gate certificate
No lock-inPortable DSL in Git, BYO-vault facade, BYO model registry, pluggable Doc-AI engine, OSS-only stack, spec-first OpenAPI
ObservabilityOne OTel pipeline with redaction, fanning out to OpenSearch, VictoriaMetrics and Langfuse, with independent alerting and on-call
10 / Technology stack

Open source, self-hostable, by layer

Every layer is built on permissively licensed open source that runs fully on your infrastructure. No SaaS-only dependency anywhere in the critical path.

LayerPickLicenseRole
Durable executionTemporalMITThe durable backbone, single source of truth for run state and replay
DSL and validationXGrammar, Pydantic, AjvApache / MITClosed-vocabulary JSON DSL, constrained decoding and schema validation
Agentic runtimePydantic AIMITOpt-in agentic steps as typed agents inside a Temporal activity
LLM servingvLLMApacheReference BYO serving, OpenAI-compatible, runs air-gapped
Web automationPlaywright, Browser-UseApache / MITDeterministic browser driver plus agentic browser
Desktop automationFlaUI + UIAMITWindows desktop interactive automation
Document AIPaddleOCR, DoclingApache / MITOn-prem OCR, layout and table extraction
SandboxgVisor (Firecracker)ApacheSandbox for ephemeral runners and dry-run authoring
VaultGoogle Tink, OpenBaoApache / MPLEnvelope encryption and JIT decrypt, BYO-vault facade
Control-plane storePostgreSQLBSD-likeRelational store, Temporal persistence, audit host
Event busNATS JetStream, ValkeyApache / BSDInternal CloudEvents bus and cache
ObservabilityOpenTelemetry, OpenSearch, VictoriaMetrics, LangfuseApache / MITOne telemetry pipeline fanning out to three stores
IdentityKeycloak, SPIFFE/SPIREApacheSAML/OIDC plus SCIM and workload identity
API and gatewayoapi-codegen, APISIXApache / MITUnified OpenAPI 3.1 REST API
Installer / air-gapHelm + Zarf, cosignApacheGuided installer, signed offline bundle, private OCI mirror
11 / Decisions

Key architecture decisions

The defining decisions, all accepted. Each one trades a tempting shortcut for long-term reliability, auditability or no lock-in.

ADR-01

Interpreter, not codegen. The DSL is data for one deterministic Temporal workflow. An AI fix to the DSL runs at once, with no worker rebuild or redeploy.

ADR-02

Temporal is the only backbone. Adopt it, do not build, and keep no second checkpointer. Durability, replay and state are a single source of truth.

ADR-03

Agents inside activities. Agentic steps are opt-in Pydantic AI agents running inside an activity, composed on Temporal, not a heavy agent framework.

ADR-04

Stateless per run. A run's state is the DSL plus input plus event history. No free-form agent memory leaking across runs.

ADR-05

BYO LLM only. Pluggable cloud or local endpoint, with an eval harness and sizing guide shipped, never bundled weights.

ADR-06

Self-heal suggest-only by risk tier. Low can auto-apply if enabled, high always pauses for human approval. Fixes are reviewable diffs into versioned DSL.

ADR-13

Approval only in the Console. Channels notify and deep-link, never approve in-channel. Identity and channel are audited.

ADR-16

Flat per-edition licensing. Two editions by capability, never per bot or per run. Ed25519 offline, cluster-fingerprinted, no phone-home.

ADR-18

Air-gap-first. Control-plane and runner split, offline bundle and registry, offline signing, guided installer with a hard-block preflight.

ADR-20

Diagram strictly read-only. Render DSL to diagram one way only. No round-trip and no drag-and-drop editor that could drift from the source.

12 / Risks

Risks the architecture absorbs

The hardest failure modes in enterprise automation, and the structural decision that contains each one.

AI non-determinism and hallucination

An LLM could invent a parameter, fake a success, or mutate a running step.

Contained by: a deterministic recorded core, post-step validators and tool-receipt verification, risk-tiered human approval, suggest-only by default, and model version pinning for replay.

Windows interactive session

Desktop GUI cannot run in a container or Session 0, and breaks silently when a disconnect changes resolution.

Contained by: a dedicated interactive session manager with auto-login, a pinned display resolution, per-session slots, and a persistent runner class separate from the ephemeral fleet.

Exactly-once side effects

A re-dispatched job after a false heartbeat timeout could run a payment or email twice.

Contained by: lease fencing plus idempotency keys, Temporal result caching so completed steps never rerun, and outbound-only lease-pull runners.

Air-gap upgrade and migration

Install, upgrade, CVE patch, DR and model update are all harder offline.

Contained by: air-gap as a designed-for scenario, signed offline bundles and a registry mirror, documented offline upgrade and patch paths, and a hard-block preflight.

MCP / skill supply chain

Tool poisoning, rug-pull, cross-server exfiltration, confused-deputy OAuth.

Contained by: a governed gateway with no open marketplace, namespace verification, version pinning, tool-description hashing, scoped RBAC, outbound DLP, and signed import.

BYO-model variance

A customer model that is too weak degrades authoring and self-heal.

Contained by: an eval harness and sizing guide to certify a model before production, one code path against any model, and an AI-Quality gate that blocks a regression.

Want the deep-dive walkthrough?

We can take your architecture, security and procurement teams through any layer of this design, end to end, against your own requirements.

Book a demo