Server runner
The zekno-runner program on your server. Its commands, its session, where it keeps its files, what it refuses, and how to install it.
Pre-alpha and unaudited
The runner, the command-line tool and the core library have had internal reviews only. Do not use them for anything you could not afford to lose, and do not rely on them in production before an external audit.
The runner, zekno-runner, is the program that runs on your server. It is an
ordinary command-line program, not a service: it opens no network port, never
asks a question, and runs only when it is started. Each run reads its standard
input, answers on standard output, and exits.
The command-line tool starts it over SSH. The runner checks that a deploy was signed by your team, is addressed to this server and is newer than the last one. Only then does it decrypt the secrets and start Docker Compose. Having SSH access is not enough to make it run something: the signature is.
When an app has a domain, the runner starts a proxy on the server, a container that listens on ports 80 and 443 (see Domains and the proxy). The runner itself still opens no port.
Commands
| Command | Reads | Prints |
|---|---|---|
zekno-runner setup | the founding team list and the team code | this server's public key, its id and its code |
zekno-runner sync | newer team lists | ok or a refusal |
zekno-runner apply | newer team lists and one deploy | started, start_failed or a refusal |
zekno-runner status | nothing | versions and deploy numbers, never values |
zekno-runner version | nothing | its version and protocol, such as {"version":"0.1.0","protocol":2} |
zekno-runner session | one request per line, until its input closes | one frame per line |
Each command but session prints one JSON line on standard output. setup,
sync and apply read one JSON message from standard input, of at most
32 MiB. During a one-shot apply, Docker's own output goes to standard error.
Any other word, or more than one, prints the usage on standard error.
deploy, records and proxy, like state and logs, are requests of a
session only: zekno-runner deploy, zekno-runner records and
zekno-runner proxy print the usage, with exit status 2.
Protocol 3
Runner 0.1.0 speaks protocol 2. A runner built from the repository speaks
protocol 4, and answers version with {"version":"0.1.0","protocol":4}. No
runner release speaks protocol 3 or 4 yet: runner 0.1.1, the first that will,
is not published.
Protocol 3 means the runner carries out deploy format 2, zekno.deploy.v2:
each deploy names the deploy of the app it was made after (previous), may
carry an expiry (expiresAt), and lists each secret by its version. The
runner keeps each secret's latest record, so a deploy sends only the records
it lacks. It answers two more requests in a session, deploy and records.
It still opens no network port and makes no connection of its own.
Deploy format 2 also has fields for builds from source and a command to run
before a deploy. The runner carries out neither, and refuses a deploy that
uses one. It takes a deploy in deploy format 1, through apply, only for the
apps it held before it was updated that have had no deploy in format 2 since
(see What it refuses).
Protocol 4
Protocol 4 means the runner also carries out the two other fields of deploy format 2: its routes, each a domain, a service and a port, and its health check (see Domains and the proxy). A deploy can end with two more outcomes, both after its number is used:
routes_failed: the app started, and its routes could not be put in force, with the reason,port_taken,no_container,host_networkornot_loaded, and the port or the service it concerns.unhealthy: the app started, and a service was not healthy in time, with that service. Nothing is rolled back.
In a session, deploy sends a health event when it begins to wait for a
service. status with the input {"routes":true} also answers the routes in
force of each app that has any, and the proxy's state: none, running,
stopped, missing or other_image. Without that input it answers as
before, so every older zekno reads it. proxy, a new request, makes the
proxy what the routes in force need, from the state alone: it starts the
proxy, makes it again from the pinned image, loads its configuration when it
differs, or removes it when no app has a route. It needs no signature, since
it puts in force no route a signed deploy did not. zekno server update
sends it after an update.
The runner itself still makes no connection of its own. The proxy connects to Let's Encrypt for certificates, and Docker fetches the proxy's pinned image the first time it is needed.
The session
zekno-runner session lets the command-line tool make several requests over
one connection, so one SSH login serves a whole command. The tool writes one
request per line; the runner writes one frame per line. Each line is JSON, at
most 32 MiB.
CLI → {"command":"status"}
runner ← {"result":{ …status… }}
CLI → {"command":"apply","input":{ …deploy… }}
runner ← {"event":"progress","line":" Container zekno-demo-web-1 Recreate"}
runner ← {"event":"progress","line":" Container zekno-demo-web-1 Started"}
runner ← {"result":{"outcome":"started","app":"demo","sequence":5}}
CLI closes its side| Request | Events | Result |
|---|---|---|
version | none | its version and protocol |
status | none | as the status command, or a refusal; with {"routes":true}, also the routes in force and the proxy's state |
setup | none | as the setup command, or a refusal |
sync | none | as the sync command |
apply | progress: Docker's output, one line each | as the apply command |
state | none | what is running for each app, or a refusal |
logs | log: one line of an app's logs | {"outcome":"ok"} once Docker's log command ends |
deploy | progress: Docker's output, one line each; health: the service it waits for | started, start_failed, routes_failed, unhealthy or a refusal |
records | none | the statements of the secret records it keeps for an app, or a refusal |
proxy | none | the proxy's state, none or running; routes_failed with port_taken or not_loaded; or a refusal |
state, logs, deploy, records and proxy exist only inside a session.
deploy and records need protocol 3, and proxy protocol 4: a runner from
before it answers with invalid_request before it runs anything.
- Requests are handled one at a time, in order. Each is answered by zero or more events, then exactly one result.
- Every request goes through the same code as its one-shot command: the same lock, the same checks, the same order of work. A session is a way of carrying requests, not a second way in.
deploytakes newer team lists, one deploy in deploy format 2 and the records of the listed secrets the runner lacks ({"rosters":[…],"envelope":{…},"records":[…]}). It takes the same lock asapply, checks the deploy in a fixed order, and only then uses its number and starts Docker Compose.recordstakes an app ({"app":"<app>"}) and answers{"statements":[…]}: for each secret of the app it keeps a record of, the record's signed statement, sorted by name. A statement longer than 4,096 characters is left out. A statement says the secret's name, its version, the hash of its sealed value and who set it, never a value, sealed or not. An app it does not hold, or one it takes format 1 for, answers{"statements":[]}. Likestatus, it needs the runner to be set up and its folder to be private, takes no lock and changes nothing.stateandlogsread from Docker and change nothing. They need the runner to be set up and its folder to be private, likestatus, and take no lock.staterunsdocker compose -p zekno-<app> ps --all --format jsonfor each app the runner has accepted a deploy of, and keeps only each service's name, state, exit code and health. Labels, commands, container names and host paths are dropped on the server. Docker gets 30 seconds per app; an app it could not be asked about is left out, which means "not known", not "stopped". An answer holds at most 1,024 apps and 4,096 services.logstakes an app the runner has accepted a deploy of,tailfrom 0 to 1000, andfollow, true unless it isfalse. It runsdocker compose -p zekno-<app> logs --follow --timestamps --no-color --tail <n>(without--followwhenfollowis false) and sends each line as a service, a time and its text, at most 16384 characters.- Docker's commands for
stateandlogsrun in the runner's own folder, and get onlyPATH,HOME,DOCKER_HOST,DOCKER_CONFIG,DOCKER_CONTEXTandXDG_RUNTIME_DIRfrom its environment. No secret.docker compose up, forapplyanddeploy, runs in that folder too, with the same variables and the app's secrets. - A line that is not a request is answered with the refusal
invalid_request. After three in a row, the session ends. - A client that gets more than 1,024 requests or 32 MiB ahead of the answers has its session ended. Requests still waiting are not run.
- After each result the runner waits until the frame has been read, the reader has gone or the input has closed, so a client that does not read cannot make it hold answers in memory. Progress frames do not wait: once 8 MiB of them wait to leave, further ones are dropped. A result is never dropped.
- When the input closes during
applyordeploy, once the deploy has been accepted, it carries on to its end: its number is used. When the input closes duringlogs, or nobody reads any more, the runner stops Docker's log command and exits.
Exit status
| Command | 0 | 1 | 2 |
|---|---|---|---|
setup, sync, apply, status, version | done | refused, or the app did not start | wrong usage |
session | its input closed, or its reader went away | three unreadable lines in a row, a client far ahead, or an unexpected error | wrong usage |
A command-line tool that gets status 2 for session knows it reached a runner
from before sessions, one that does not report its version, and uses the
one-shot commands instead.
Its folder
ZEKNO_RUNNER_HOME when it is set. Otherwise root always uses
/var/lib/zekno. Any other user uses ~/.local/state/zekno if a runner is
already set up there; else /var/lib/zekno only if that folder exists and
belongs to them (a server set up by an earlier version); else
~/.local/state/zekno. Mode 0700.
| File | What |
|---|---|
server.key | This server's secret key. Mode 0600. It never leaves the server. |
server.pub | This server's public key. |
state.json | The team list it trusts, the last deploy number per app, the latest version of each secret, and a version mark: "v": 3, each app's deploy format under formats, and under routes the routes in force of each app that has a domain or a health check, with the names it gave their containers. |
records/<app>/<hash>.json | The latest record of each secret of an app deployed in format 2, named by the hash of its sealed value. Mode 0600. |
apps/<app>/compose.yaml | The Compose file of the last accepted deploy. |
proxy/etc/caddy.json | The proxy's configuration, written whole by the runner. The proxy reads the folder read-only. |
proxy/etc/caddy.next.json | A new configuration while Caddy checks it. It replaces caddy.json only when it is valid, and is removed when it is not. |
lock | Present while a setup, sync, apply, deploy or proxy is at work, so two runs never overlap. |
Decrypted secrets are never written here. They are handed to Docker Compose through its environment. Docker itself keeps a container's environment in its own configuration. App state and logs are read from Docker when asked and never stored.
A record is the secret's signed statement and its value sealed for this server's key, as it arrived. The runner keeps one for each live secret of an app in format 2, and deletes the one it replaces, and the one of a removed secret, once the state that no longer names it is saved. It opens a record only after every check of a deploy that lists it, and checks the record's signature again at each deploy. A server keeps at most 8,192 records, and at most 64 MiB of them. Deploys in format 1 leave no record.
Runner 0.1.0 writes state.json without the mark. A runner that speaks
protocol 4 reads such a state and writes nothing until its first change
(sync, apply or deploy), which writes it whole with the mark; setup
writes the mark from the start. A state marked "v": 2, which a runner of
protocol 3 writes, is read as holding no routes. Each app it held when it first read the state
is at format 1 until its first deploy in format 2, and stays at format 2 from
then on. An app it never held is at format 2 from its first deploy. Once the
state has the mark, runner 0.1.0 cannot read it (see
What to know before you rely on it).
What it refuses
A refusal is one JSON line, {"outcome":"refused","reason":"<word>"}, with
exit status 1, or a result frame in a session. The reason is one of a fixed
list of words; no answer carries a path or other detail of the server.
- Not set up:
status,sync,apply,stateandlogsrefuse withnot_set_up. - Already set up:
setupon a server that has a key refuses withalready_set_up. The key is never overwritten. - A folder it cannot trust: a
state.jsonthat is damaged or incomplete, or a folder or key file that belongs to another user or that anyone else can read or write, is refused withinvalid_state. A broken file is never treated as "nothing accepted yet". Every command refuses to act in a folder other users could write to, or with astate.jsonthey could change, and leaves both as they are. - Another run at work:
busy. - A deploy it cannot verify: a deploy not signed by a current member of the team, one meant for another server, one whose number is not newer than the last it accepted, or one whose secrets do not match its list, is refused before anything is decrypted. The reason says which.
- A deploy out of order: a deploy in format 2 names the app's last deploy
it was made after. When that is not the runner's last number for the app,
it is refused:
replayedwhen it is lower, because another deploy got there first, andsequence_jumpwhen it is higher. - A secret not accounted for: a deploy in format 2 must list every secret
of the app the runner holds a live version of, as current or as removed, and
may carry records only for secrets it lists; otherwise
secrets_mismatch. A listed secret the runner keeps no matching record of, and that was not sent with the deploy, issecret_missing. A secret whose record was set by someone the team list it holds no longer includes, sent or kept, isuntrusted_secret. - An expired deploy:
expired, when the deploy's expiry has passed by the server's clock, or is more than 7 days after it.zeknosigns no expiry over SSH. The order of deploys never depends on the clock. - A domain another app holds: a deploy in format 2 with a route for a
domain that another app on the server holds is refused with
domain_taken, before anything runs. - What it does not carry out yet: a deploy in format 2 that asks for a
build or a command before the deploy is refused with
unsupported_version, once its format has been checked. One whose field breaks its own rule, with a field the format does not have, with avother than 2, or written in any other spelling than the onezeknosigns, ismalformed. - Format 1, for an app that has moved on: a deploy in format 1 (
apply) is taken only for an app the runner held when it first read a state without the mark, and that has had no deploy in format 2 since. Any other is refused withunsupported_version, a word every olderzeknocan read. An app that holds a domain has had one, so a deploy in format 1 never releases it. - Too much: a deploy that would make it hold more than 1,024 apps, 8,192
secret versions or 1,024 domains, or keep more than 8,192 records or 64 MiB
of them, is refused with
too_large, before anything is opened. So is a deploy that sends a record whose statement is longer than 4,096 characters. - A request it cannot read: one that does not match its message, is larger
than 32 MiB, or asks for the logs of an app it never accepted a deploy of, is
refused with
invalid_request. The app name is checked against the app-name pattern before Docker sees it.
Every refusal of a deploy in format 2 comes before its number is used, and
changes nothing but the team lists sent with it: state.json, records/,
the Compose file and Docker are as they were.
Domains and the proxy
A deploy with routes or a health check does more once docker compose up
has returned, under the same lock. The runner finds each routed or checked
service's containers by their Compose labels, and gives each a random name on
the app's route network, zekno-route-<app>, which it makes. It writes the
proxy's configuration whole, from the routes in force of every app and the
names it made, to a second file, proxy/etc/caddy.next.json. Caddy checks
that file, and only a valid one replaces proxy/etc/caddy.json. When a route
needs the proxy and it is not running, the runner then starts it,
zekno-proxy, from the Caddy image it pins by digest; a running proxy is
told to load the new file. The load counts only when what the proxy saved
holds the new file's nonce, a random mark the runner puts in each file. Only
then does it save the new routes. Then it deletes the certificate and key of
each released domain, also when the proxy is stopped or gone, and removes the
proxy when no app has a route. Then it waits for the health check. What each
step means for your apps is on Domains and HTTPS
and Health checks.
A deploy whose app has no route and no health check, and held none, makes no
Docker call it did not make before. A deploy that stops before its start
leaves the app's routes as they were. When docker compose up fails, the
outcome is start_failed, as before, and the routes are still carried out
as far as they go, so a domain the deploy leaves out is released.
Install on a server
The server needs Docker with Compose, and Linux on x86-64 (a processor with
SSE4.2: Intel Nehalem, AMD Bulldozer or newer) or on ARM64, with the GNU C
library (glibc) 2.17 or newer. Alpine and other systems with musl are not
supported yet. The rest of what a server needs, and what was tried, is on the
Your key and servers page. ZEKNO_DOCKER names
the Docker command if it is not docker.
With zekno server add
zekno server add <name> --ssh <user@host> installs the runner by itself: the
server downloads the runner release that build of the command-line tool
carries, checks its SHA-256 against the value the tool holds, checks that it
starts, and only then puts it in place. A tool built from main installs a
prod release, such as 0.1.0; one built from dev installs a development
release, such as 0.1.0-dev.1. The steps and their limits are on the
Your key and servers page.
| Who the tool logs in as | Runner | Its folder |
|---|---|---|
| root | /usr/local/bin/zekno-runner | /var/lib/zekno |
| any other user | ~/.local/bin/zekno-runner | ~/.local/state/zekno |
The first runner release is 0.1.0. A command-line tool built from the
repository holds its checksums and installs it by itself. Installing by hand,
below, and --runner-file remain for development and for servers that cannot
reach GitHub.
By hand
# In apps/runner of the repository. Build for the server's system, not your own
# (check with `ssh you@server uname -m`: x86_64 → compile:linux-x64, aarch64 → compile:linux-arm64).
bun run compile:linux-x64 # dist/zekno-runner-linux-x64
# As root, at the place the installer uses:
xz -T0 -c dist/zekno-runner-linux-x64 | ssh root@server 'xz -d > /usr/local/bin/zekno-runner && chmod 755 /usr/local/bin/zekno-runner'
# As any other user, at the place the installer uses:
xz -T0 -c dist/zekno-runner-linux-x64 | ssh deploy@server 'mkdir -p ~/.local/bin && xz -d > ~/.local/bin/zekno-runner && chmod 755 ~/.local/bin/zekno-runner'
ssh deploy@server '~/.local/bin/zekno-runner status'compile:linux-x64 and compile:linux-arm64 are the scripts a release is
built with. The file is about 110 MB; compressed with xz it is about 40 MB.
The last command prints {"outcome":"refused","reason":"not_set_up"} on a
fresh install. Then run zekno server add from your machine: a runner at the
place the installer uses, or on the PATH, is found and used as it is. A
runner anywhere else is added with --runner-path <path>. The folder must
belong to the user that runs the runner.
bun run compile builds dist/zekno-runner for the machine you are on, which
is only what you want when that machine is the server.
For development: --runner-file
zekno server add <name> --ssh <user@host> --runner-file dist/zekno-runner-linux-x64
sends a runner you built over SSH. Its SHA-256 is computed on your machine and
checked on the server when it arrives; it must answer with the version the
command-line tool carries; then it is put in place as the installer would. A
runner already on the server is used as it is.
How releases are made
A runner release is a GitHub Release of the
repository, started by pushing a tag
runner-v<version>. There are two kinds, called environments:
| dev | prod | |
|---|---|---|
| Tag | runner-v<x.y.z>-dev.<n>, such as runner-v0.1.0-dev.1 | runner-v<x.y.z>, such as runner-v0.1.0 |
| Cut from | a commit on dev | a commit on main |
| On GitHub | a pre-release, never shown as the latest | a release |
| Approval | none | waits for the maintainer in the GitHub Environment prod before anything is published |
| Installed by | a command-line tool built from dev | a command-line tool built from main |
A workflow in the repository runs on such a tag. It first stops the release
when the tag is not a version, when the tagged commit is not on the branch for
its kind, when the runner's own version is another, when the repository pins
no Bun version, or, for prod, when the command-line tool on that commit
carries a development version. Then it builds the runner for Linux on x86-64
and ARM64 with the pinned Bun, checks that the x86-64 runner answers with the
tag's version, and publishes four files: the two runners compressed with gzip,
SHA256SUMS with the SHA-256 of each unpacked runner, and RELEASE-NOTES.md,
which is also the release's text. A tag that already has a release gets no
second one. Its actions are pinned to full commits. Every install skips the
packages' own install scripts, and each job after the first checks the tag
itself before it does anything else. It uses no secret beyond the token GitHub
gives every workflow, and that token can write only in the job that publishes.
The runner is built in a job that can read and not write; the packages and the
built runner run only there. It hands the four files on to the publishing job,
which checks out nothing, installs nothing and runs nothing it was handed: it
checks that exactly the four files are there, within their sizes, and that the
two runners unpack to what SHA256SUMS says, then publishes them. For a prod
release the build runs at once, and publishing waits for the maintainer's
approval. A build can still hand on files other than its source makes: that is
what the second build catches, and no command-line tool installs a release
before it.
The workflow does not record checksums in the command-line tool. Only the
maintainer's second build does, once a build of the same commit on their own
machine gave the same files (see
How a release is trusted). The release
procedure, from start to end, and how to withdraw a release, is
apps/runner/RELEASING.md in the repository.
What to know before you rely on it
- The first setup wins. Until
zekno server addhas run, whoever can start the runner decides which team the server obeys. Install it and add the server in one sitting. If someone got there first,zekno server addsays the runner already belongs to a team. - Starting the runner means reading every app's state and logs.
stateandlogsneed no signature. Anyone who can start the runner on the server, for example through SSH access meant for something else, can read what is running and every app's logs. They name no secret, but logs can hold anything an app prints. This is an open finding of the internal review that waits for the maintainer's decision.recordsneeds no signature either: it gives the statements of each app's secrets, their names, versions and who set them, never a value. The design accepts that. - Updating it is one way. Once a runner that speaks protocol 3 or 4 has
written its state, runner 0.1.0 cannot read it. Put back by hand, it refuses
status,sync,apply,stateandlogswithinvalid_state, andsetupwithalready_set_up, and never reads the state as empty. Azeknofrom before deploy format 2 can then deploy only the apps that have had no deploy in format 2 (see Updating is one way). - Restoring its folder from a backup is not supported. A folder put back from a backup holds lower deploy numbers and the team list of that time, so deploys the runner accepted since could be delivered to it again, in order, and run again. Set the server up again instead: its new key gives it a new server id, and no earlier deploy is addressed to it.
- The runner believes its environment.
ZEKNO_RUNNER_HOME,ZEKNO_DOCKERandPATHdecide where its state lives and which program receives the secrets. Do not let a less-trusted user start it with their own environment: noPermitUserEnvironment, noAcceptEnv ZEKNO_*, no sudo rule that keeps those variables. The compiled runner does not load.envorbunfig.tomlfrom the folder it is started in, which over SSH is the login user's home folder: it is built with both turned off. Like any program built with Bun, it still obeys three of Bun's own variables:BUN_OPTIONS(for example--preload <file>, which runs that file first),BUN_BE_BUN=1(it then acts as Bun itself) andBUN_INSPECT(it opens a debugger). Whoever can set them already controls that account. The server's SSH must not acceptBUN_*from clients: noAcceptEnv BUN_*.ZEKNO_PROXY_TEST, a switch for the tests, makes the proxy take certificates from its own test authority and listen on 127.0.0.1 only; a deploy cannot set it. - A domain opens ports 80 and 443 to everyone. The proxy holds the TLS keys of every domain on the server, and ends TLS for every app with a domain. Its configuration is reloaded only when it changes, and each reload can drop a few new connections.
- A health check holds the lock. While a deploy waits for one, up to 30
minutes, and up to 5 minutes more when the runner first fetches the image
a check runs, every other request that takes the lock is refused with
busy. - Signing a deploy is equal to root on the server. A Compose file can mount anything Docker can reach. The signature decides who may do that; the runner does not limit what they deploy.
- Docker's messages are passed back as they are. An error from Docker can repeat a value that was filled in from a secret, such as a malformed port.
- A release is trusted through its checksums, not a signature. The command-line tool installs a runner only when its SHA-256 equals the value the tool holds, and that value is written only by the maintainer's second build, once a build of the same commit on their own machine gave the same files as the release. That proves the published files are what this source and this build tool produce. It does not prove that Bun or the npm packages are good: both builds use the same ones. Releases are not signed, and no build provenance is published yet.
- Who may start a release. Whoever can push a
runner-v*tag starts a release, and chooses the workflow that runs for it. The release procedure sets a tag ruleset on GitHub before the first release, so that only the repository's admins can push such tags. A release started by anyone else is still installed by no command-line tool until the second build records its checksums.
The internal security reviews these come from are in the specs folder of
the repository.