Runs on the engine

How a run goes from Run in Studio to node states on the canvas.

The life of a run

sequenceDiagram
  participant P as Studio page
  participant M as mooflair server
  participant E as Engine
  participant A as Agent (browser-<id>)
  P->>M: aStudioRun (flow, dry, range, resume)
  M->>E: POST /flow/:id/start { actor, mode, range, resume, draft } (secret)
  E->>M: POST /api/internal/run { app, run: { id }, start, hash }
  M-->>E: definition (when its hash changed) + what this run needs
  E->>E: create the Run (QUEUED)
  E-->>M: 201 { run, journal, callback }
  M-->>P: run + engine link
  P->>E: WebSocket /run/:id/socket
  E->>A: reconcile container, ping (up to 90 s)
  E->>A: NATS step.run (one node at a time)
  A-->>E: step.result, run.event (log, intervention, activity)
  E-->>P: events (saved first)
  P->>E: controls (pause, resume, cancel, skip, intervention.resolve)
  E->>A: NATS step.abort / step.control

How a run starts

Every run belongs to an app: a campaign, or a flow (its trigger runs and Studio's tests share its queue and browsers). mooflair starts one with POST /campaign/:id/start or POST /flow/:id/start (TAppStart: actor, input, upload, mode, range, resume, draft), and the engine's own scheduler starts campaigns in their windows and flows on their Schedule trigger (see Campaigns).

The engine then asks mooflair what the run needs, once (POST /api/internal/run). mooflair checks it there (maintenance, options, plan gates, the datasets the person may use, required fields) and answers two parts:

• definition (TAppDefinition): the published flow ready to run: graphs with their dots and browser names (edges in canvas order), the start node, the error flow, called flows (child), and every node type's compiled code. It has a hash; the engine keeps it per app and sends its hash back, so mooflair sends it again only when it changed (a published edit, a script edit, a called flow published). A draft's test always gets it and is never kept.
• run: what only this run gets: its campaign, options ({{ option.x }}), variables, limits (runMs, lanes, file size), secrets by node, the Browser for each browser name (made in mooflair when missing; the engine brings it over from GET /api/internal/browser/:id on first use), recording days, log streams, the run API token, the resume link and the journal id.

The engine puts both together into the run's context (TRunContext), kept in memory only, never saved or sent to a page. A run keeps the definition it started with: a change reaches the runs that start after it.

Queue and start

A run whose browser can't open because the engine is full (503) waits in the engine's queue (appWaiting), priority runs (limit.isPriority, Business and up) first, then the longest waiting, trying every 2 seconds for up to 10 minutes.

Server-only routes for mooflair's services: POST /workspace/:id/prune { before } (finished runs, logs and files before then: the plan's history), GET /workspace/:id/report?since= (real runs by status and their busiest flows: the weekly report).

Payload extras: limit { lane, fileMb, isPriority }, recording { days } (a JPEG after every browser step, swept after that many days), stream [{ url, secret }] (log lines and status changes batched every 2 s to the workspace's run.log webhooks, signed like mooflair's), file.upload (base64 files written as the run's files before it starts).

• A run takes its owner's turn (a campaign, or the flow in tests: one main run at a time, in memory in the engine's App). If another run holds it, the new run stays QUEUED in that queue; when the holder finishes, the next one takes over.
• A priority run (mode.priority: a campaign's row action such as Delete reply) takes no turn and starts at once, in a new tab of the browser its node names, beside the main run. The two never act on the outside world together: an effect step waits while the other is in one.
• A browser opens the first time a node of the run uses its name (App.open, then AppBrowser.start): it is marked running, its container reconciled, and the agent pinged every 2 s for up to 90 s. No answer: Browser didn't start in time. At capacity: The engine is at capacity as that node's error.

The walk

The engine walks the graph (AppRun, in memory) and sends each step to its browser's agent. A frame is one graph with its own outputs, node states, log path (Flow › Called) and secret scope: the flow, the error flow, or a flow a Run flow node called. A Run flow node walks its called flow right there, inside its own lane (its tab and browser); the called frame's end (its Return, its last lane, a failure) gives that Run flow node its items for the current input item: a failure gives it one item on its error dot, a cancel cancels the caller too. Only the main frame's nodes are reported in the run, can be skipped, jumped to or ranged, and move the cursor; the logs, waits for a human and the Now card of a called flow's nodes belong to the main flow's node it runs for (root), at any depth.

Items

What travels along edges is a list of items (like n8n): { json, file?, trail }. file holds the item's by key, as references (the bytes stay on the engine). trail links each item to its ancestors: the index of the item it came from in every earlier node's output (and its own index under its own node), so {{ nodes.<id>.output.x }} reads the item of that node linked to the current one (else its first). A trigger starts the walk with one empty item; a range or a resume starts at that node with what reached it last time.

A node's code returns { item: [{ dot, json, file?, pair? }] } (files go on only when the node returns them). json that isn't an object becomes { value }; an item for two dots is listed twice; an empty list ends that branch. Over 1 MB in all, the rest is cut (the node state's cut says how many) and doesn't go on.

Each lane works through its walk

walk = { lane: [{ id, from, queue, active, taken, origin }], parked }. A frame starts with one lane; every lane runs at the same time (Lanes), each in this order on every turn:

1. The active node runs its next step. run: "each" (most nodes): one step per input item, its fields resolved against that item ({{ item.x }}, {{ index }}); with batch.size the engine waits batch.waitMs after every that many items; with item.parallel it runs that many items at once (each in a tab of its own, results kept in item order; a pause reruns the whole group). run: "once" (Merge): one step with every item (ctx.items, and ctx.input by input dot), its results paired back by pair or in order when the counts match. A pin, a cached Run flow and a test run's effect node give their items without a step.
2. Else the last delivery queued (last in, first out: a branch finishes before the next starts). A node with one connected input runs once per delivery; one with several (a join, e.g. Merge's a and b) parks the items until every connected input has delivered.
3. Else a parked join that no other lane can still reach runs in this lane with what it has (a missing input counts as no items).
4. Else the lane ends. The frame ends when every lane did, or when one fails or returns (the others stop).

When a node is done its items leave along every edge of their dot, queued in canvas order (mooflair sorts edges by their target's position, top to bottom then left to right), so the upper branch runs first. Before it starts: disabled or skipped, its items pass through to its first dot; no items runs it only with isAlways (one empty item); isOnce keeps the first. It stops at range.toId (that branch only), on Return (the frame ends with its items), on failure, or on cancel.

For each step: cursor and the node's RUNNING state, fields resolved () and secrets merged, the step sent to the agent (step.run, with item, items, input, index) which runs the compiled code with the and a step signal (run controller ∪ node timeout), then retries and On error (): continue and output add { error } as that item's result (first dot, or the error dot).

The main frame's walk is saved on the run (walk) after every item: a resume goes on from each lane, running its active node's current item again.

Lanes

A lane is a branch of the walk running at the same time as the others, with its own queue and its own tab in the same named browser (opened at the page of the lane it came from). Lanes start from:

• A node with branch.mode: "parallel": each outgoing edge goes on in its own lane (the first stays in the current one). Past branch.limit, or the run's cap ENGINE_MAX_LANES (default 8), the rest go on one after another in the current lane.
• Lane items: an output item marked lane (a 's items) goes on in its own lane, with the stack item it took. With failure: "skip", a failure in that lane ends its turn and the lane goes back to the node that started it.

What they share and how they meet:

• Joins wait for every lane that can still reach them (the graph from each lane's node and queue), then run in the last lane to arrive.
• Node state is shared per node across lanes: running counts the copies running now (lanes and parallel items; the canvas shows ∥N), count every execution.
• Effect steps still go one at a time per browser (the agent serializes them), and stop waits for every lane's effect step.
• Waits for a human queue up: one shows at a time per run (the first), its answer goes to that step (browser.<id>.step.control with its stepId), and the other lanes wait their turn. The run is INTERVENTION while any waits.
• Pause, stop and cancel apply to every lane; a resume restores each lane's queue. A stack's jump (stack.done on a current item) goes to the lane that took that item.
• The cursor follows the last lane that moved.
• Lanes need a plan with the lane feature (Team); mooflair refuses a run that uses them otherwise.

Stacks

A node type with a stack (e.g. Collect posts' items) turns that list in its first output item into a live stack, keyed <scope><node id> (scope: "", or <flow id>/ inside a called flow; only the run's own go to viewers). Each item is keyed by its id and starts waiting. The same keys again keep the stack (a loop on its next item); new ones replace it. The engine keeps every state:

• a node whose fields read nodes.<id>. locks that stack (edits would change nothing);
• with take, the output's item becomes current; when the node runs again it becomes done;
• stack.done on the current item sets a jump: the step it's on is aborted (unless it's an effect step or inside a called flow; then right after it) and the walk delivers back into that node instead of going on.

limit.runMs fails the run once it has taken longer (skipped when 0).

Events (agent → engine → viewers)

type TRunEvent =
	| { type: "status"; status: TRunStatus; error: string | null }
	| { type: "node"; nodeId: string; node: TRunNode }
	| { type: "cursor"; nodeId: string | null; browserId: string | null }
	| { type: "browser"; id: string; name: string }                       // the run opened a browser of this name
	| { type: "log"; log: { id?; nodeId; level; type; message; path: string[]; context; browserId?; accountId? } }  // the engine fills browserId/accountId from the run (a called flow's own run relays its own)
	| { type: "intervention"; intervention: TRunIntervention | null }
	| { type: "stack"; nodeId: string; stack: { version: number; edit: boolean; isLocked: boolean; item: { key; state: "waiting" | "current" | "done" | "skipped" | "removed"; data }[] } }
	| { type: "activity"; activity: { nodeId; title; card: "post" | "product" | "list" | "text" | "image"; data; at } };

type TRunIntervention = { nodeId; title; message; detail: Record<string, string> | null; step: string[]; button: { key; label; tone; check: boolean }[]; data; auto: { key; at } | null; check: { isAuto; at; reason; isFailed } | null; since: number };

Stacks and activity are saved on the run (stack, activity), so a viewer joining later sees them. Activity is sent a few times a second at most.

Viewer messages

Events are the agent's and the walker's; viewers never get them. AppRun.save is the run's one reducer: each event becomes one $set (dotted paths of TRun), saved to MongoDB, then sent as it is:

A viewer writes each path on its copy of the run, so the page and the saved run can't differ, and a new field needs no viewer change. A log is its own message with its saved id (the page never shows one twice). AI log lines (type: "ai", context.usage, context.cost) add to FlowRun.ai with $inc, and viewers get the new totals as { ai }. A campaign's viewers also get each new run whole (type: "run") before any change to it.

The check of an intervention (check, null when the step has none) is polled every 3 seconds while isAuto is on, and the step goes on by itself once it passes. Off (the dialog's Auto check when it's done switch, intervention.check), only Done checks: change settings or the account first. The switch shows only for a step with a real check (a sign-in, a captcha, a selector).

auto ({ key, at }) is the answer picked by itself at at unless someone answers first: the agent fires it with a timer (Approval's auto approve or reject, a Form's or a Wait's or a chat reply's limit as timeout, a sign-in's 6 hours as timeout), and the log says it was picked by itself. A step never fakes a check to get there.

Node state:

{ "status": "DONE", "item": [{ "dot": "next", "json": { "items": [] }, "trail": { "collect": 0 } }], "cut": 0, "error": null, "count": 2, "browserId": "6ae18cd53617f203545040bd", "startedAt": "…", "finishedAt": "…" }

Controls (viewer → engine → agent)

Sent on the run socket as JSON:

{ "type": "pause" }
{ "type": "resume" }
{ "type": "cancel" }
{ "type": "skip", "nodeId": "reply" }
{ "type": "stop", "reason": "Stopped by schedule" }
{ "type": "intervention.resolve", "result": { "key": "approve", "data": "edited reply text" } }
{ "type": "intervention.check", "isAuto": false }
{ "type": "stack.remove", "nodeId": "collect", "key": ["2115…"], "version": 3 }
{ "type": "stack.move", "nodeId": "collect", "key": "2116…", "index": 0, "version": 4 }
{ "type": "stack.done", "nodeId": "loop", "key": "2117…", "version": 7 }

• stop ends the run at its next safe point: right away, unless it's in an effect step, which finishes first. It ends CANCELLED with the reason as its error.
• intervention.resolve answers with a button's key. A button with check runs the intervention's check first: not done, the intervention comes back with check.isFailed and the reason instead of going on. force goes on regardless.
• stack.* must carry the stack's current version and only change an editable, unlocked stack; a refused one gets the current stack sent back. stack.done on a waiting item means it's never taken; on the current one the run goes back to that node.

Cancelling or stopping a run still in the queue just takes it out. Pause aborts the current step's signal; the node goes back to PENDING and runs again on resume.

Called flows

A Run flow node walks the called flow's graph inside the same run, in the same browsers: each node acts in the browser its browser setting names (see ). The run's cursor follows the node, the Run flow node remembers the browser it acted in (node.<id>.browserId), and the called flow's logs and waits for a human belong to the run, nested under the Run flow node by flow name (path). When a called flow ends, the run goes on with the browser the caller used last.

Error flow

When a run fails and its flow has an error flow, the engine walks the error flow inline with { flow, run, error } as its Flow input. Its outcome doesn't change the run's status. A run a break ended doesn't run it.

AI agent

ai/agent runs in the engine (lib/app/run/agent.ts), like Run flow. Its AI key (and fallback) and MCP credential come resolved in the node's secrets; the engine calls the OpenAI-compatible chat API itself (lib/ai), with the node's tools as functions:

• flow tools are walked as frames of the run, in the node's lane, with the model's arguments as their input (their schema from the called flow's Flow input sample). Their Return answers the model.
• node and code tools are steps (execute), a node tool with its own type and fields ({{ ai.<name> }} filled from the arguments), a code tool as the node type tool/<scope><node id>/<index> that mooflair compiled into the definition.
• http, calculator, think and the MCP server's tools (lib/mcp, Streamable HTTP) run in the engine.

Each model call logs as type ai (so the run's AI totals count it), each tool call as a step log. Memory goes through mooflair's /api/run/memory.

Outside waits

A node can wait for the outside world instead of a viewer: ctx.intervention({ …, isOutside: true }) (Wait for a call or a form, Form at a public link). Such a wait shows like any other (status INTERVENTION, no live view, no buttons), but a viewer's intervention.resolve is ignored. It's answered by a call to the run's resume URL, which mooflair relays to POST /run/:id/resume { data } (the URL's token names the run): the engine answers its first outside wait with { key: "resume", data }. The run's context carries the URL as callback.link ({{ run.resume }}, ctx.run.resume).

Error directives

A node's setting.error.mode of commit, rollback or break () ends the whole run from any frame or lane. The first one wins (walker.directive), and the frame outcome stops the other lanes like any end:

Resuming an INCOMPLETE run (resume.runId) goes on from its saved walk, so the node that broke runs its item again. The old run becomes CANCELLED with "Resumed as run ", and the new one counts attempt + 1. POST /run/:id/control { "type": "discard" } ends an INCOMPLETE run as CANCELLED ("Discarded"); any other status answers 409.

Usage and plan limits

The run's context carries limit.runMs (the flow's time limit, capped by the plan's longest run; time spent waiting for room for a browser is added back). Browsers at once (the plan's) and the workspace's RAM and CPU limit are checked by mooflair's lease when a container starts, not in the payload: a run that finds no room waits ("Your plan allows N browsers at once: waiting for one to stop"). See .

The engine counts node steps per workspace (each step a browser ran; pins, cache hits and simulated effects don't count). Every minute it posts them to mooflair's POST /api/internal/usage (with the engine secret), with browser time: the seconds since the last report for each running browser container, and worker time apart (mooflair counts it toward browser hours at billing.meter.worker.weight, 0.25 by default; ), and the containers running now (lease). Mooflair adds them to the month's counters and answers the workspaces that are over a plan limit, the container sizes (resource) and what each workspace over its admin-set limit frees (free). The engine also reports once at boot (before any run starts a container) and whenever mooflair asks (POST /usage).

A run of an over-limit workspace pauses at its next step (no step is cut in half) with the log "Paused: this workspace reached its plan limit". When a later report no longer lists the workspace (an upgrade, or a new month), the engine resumes those runs by itself. If mooflair doesn't answer, the counts are kept for the next report. Needs MOOFLAIR_URL on the engine.

NATS subjects

FAQ

Where do I see a called flow's run?

Its logs are in the caller's run under the Run flow node. A called flow no longer has a run of its own.

Engine restarts

A run lives in the engine's memory. On a stop (SIGTERM/SIGINT: a deploy, a dev reload) every run pauses at a safe point first: a post, reply or other effect step in flight finishes (up to 30 seconds; compose gives the engine stop_grace_period: 40s), the rest stop where they are, and the saved state is flushed. On boot the engine marks them FAILED (The engine restarted), tells every browser still up to drop the runs it held (browser.<id>.run.reset: their steps stop, their windows close, the home window stays), and, once it has its apps' config from mooflair (GET /api/internal/app, again every 30 seconds until mooflair answers, for up to 30 minutes), resumes every campaign run it cut off from where it was. A run cut in the middle of an effect step anyway (a crash) isn't resumed (the run records its effect nodes): you get a notice to check it.