Skip to content
{yen}

Guardrails for agents

Fleets do not fail in exotic ways. People who run Proxmox clusters, Kubernetes clusters and homelabs hit the same mistakes again and again, on machines they do not look at that day:

An agent makes these mistakes faster than a person, and a rule in a prompt (“do not touch production”) does not stop it. A rule in the fleet does. This fleet has three machines in two zones and the services on them:

modules/fleet.nix
# A small homelab: three machines in two zones, and the services on them. A service points at its machine.
{ yen, ... }:
{
  yen.kinds.fleet = {
    spawns.host = _: yen.host;
    sinks.out.reduce = yen.lib.reduce.merge;
  };
  yen.kinds.host = {
    fields = {
      addr = yen.lib.str;
      zone = yen.lib.enum [
        "prod"
        "dev"
      ];
      backup = yen.lib.opt (yen.lib.enum [
        "nightly"
        "none"
      ]) "nightly";
    };
    spawns.service = yen.lib.pointing "on" yen.service;
  };
  yen.kinds.service.fields = {
    on = yen.kinds.host;
    port = yen.lib.int;
    reach = yen.lib.opt (yen.lib.enum [
      "lan"
      "internet"
    ]) "lan";
  };


  yen.fleet.main = { };
  yen.host.edge = {
    addr = "10.0.0.1";
    zone = "prod";
  };
  yen.host.db1 = {
    addr = "10.0.0.2";
    zone = "prod";
  };
  yen.host.lab = {
    addr = "10.0.1.1";
    zone = "dev";
  };


  yen.service.proxy = {
    on = yen.host.edge;
    port = 443;
    reach = "internet";
  };
  yen.service.postgres = {
    on = yen.host.db1;
    port = 5432;
  };
  yen.service.metrics = {
    on = yen.host.db1;
    port = 9100;
  };


  yen.aspects.database.at = yen.service.postgres;
  yen.aspects.prod-credentials.at = yen.host.edge;
  yen.flake = { out, ... }: out;
}

Each rule below is one of those mistakes, written once. A rule holds across every machine, and a fleet that breaks one is not built.

modules/invariants.nix
# Rules every change must keep, each one a mistake that fleets keep making. A rule with `bad = X` that holds is a violation.
{ yen, ... }:
let
  inherit (yen.facts)
    host
    service
    parent
    has
    lt
    ;
in
{
  # a copied address: a cloned machine comes up with the IP of its original
  yen.invariants.unique-addr =
    {
      A,
      B,
      X,
    }:
    {
      bad = [
        A
        B
      ];
      pos = [
        (host.addr A X)
        (host.addr B X)
        (lt A B)
      ];
    };


  # a port taken twice: the second service on a machine fails to start, or answers for the first
  yen.invariants.unique-port =
    {
      A,
      B,
      H,
      P,
    }:
    {
      bad = [
        A
        B
      ];
      pos = [
        (parent A H)
        (parent B H)
        (service.port A P)
        (service.port B P)
        (lt A B)
      ];
    };


  # a database on the internet
  yen.invariants.database-not-public =
    { S }:
    {
      bad = S;
      pos = [
        (service.reach S "internet")
        (has yen.aspects.database S)
      ];
    };


  # a database on a machine that is not backed up
  yen.invariants.database-has-backup =
    { S, H }:
    {
      bad = H;
      pos = [
        (parent S H)
        (has yen.aspects.database S)
        (host.backup H "none")
      ];
    };


  # production credentials on a machine of the dev zone
  yen.invariants.no-prod-credentials-in-dev =
    { H }:
    {
      bad = H;
      pos = [
        (host.zone H "dev")
        (has yen.aspects.prod-credentials H)
      ];
    };
}

An agent proposes a change

Section titled “An agent proposes a change”

A change is data. Stage it and the fleet is evaluated with the change, and nothing is written. The agent gets the answer that a person would get from an outage, before the outage. Each change below is a plausible edit.

A second service on a taken port

Section titled “A second service on a taken port”

An agent adds Grafana to the database machine and gives it the port it knows, 5432.

changes/second-service-on-5432.json
{ "yen": { "service": { "grafana": { "on": { "kind": "host", "name": "db1" }, "port": 5432, "reach": "lan" } } } }
result
{
  errors = [ ];
  ok = false;
  violations = [
    [
      "bad"
      "unique-port"
      "fleet:main/host:db1/service:grafana"
      "fleet:main/host:db1/service:postgres"
    ]
  ];
}

The database goes on the internet

Section titled “The database goes on the internet”

To reach it from outside, the agent changes reach to internet.

changes/open-the-database.json
{ "yen": { "service": { "postgres": { "reach": "internet" } } } }
result
{
  errors = [ ];
  ok = false;
  violations = [
    [
      "bad"
      "database-not-public"
      "fleet:main/host:db1/service:postgres"
    ]
  ];
}

The backup is switched off

Section titled “The backup is switched off”

To save disk, the agent sets backup to none on the database machine.

changes/stop-the-backup.json
{ "yen": { "host": { "db1": { "backup": "none" } } } }
result
{
  errors = [ ];
  ok = false;
  violations = [
    [
      "bad"
      "database-has-backup"
      "fleet:main/host:db1"
    ]
  ];
}

A production credential in the dev zone

Section titled “A production credential in the dev zone”

To test a deploy, the agent puts prod-credentials on the lab machine.

changes/prod-token-to-lab.json
{ "yen": { "aspects": { "prod-credentials": { "at": [ { "kind": "host", "name": "lab" } ] } } } }
result
{
  errors = [ ];
  ok = false;
  violations = [
    [
      "bad"
      "no-prod-credentials-in-dev"
      "fleet:main/host:lab"
    ]
  ];
}

A clone with the original’s address

Section titled “A clone with the original’s address”
changes/clone-with-same-addr.json
{ "yen": { "host": { "lab2": { "addr": "10.0.0.2", "zone": "dev" } } } }
result
{
  errors = [ ];
  ok = false;
  violations = [
    [
      "bad"
      "unique-addr"
      "fleet:main/host:db1"
      "fleet:main/host:lab2"
    ]
  ];
}

A change that is fine

Section titled “A change that is fine”

The agent adds a wiki on the edge machine, on a free port. No rule breaks, so the change may land:

changes/add-a-wiki.json
{ "yen": { "service": { "wiki": { "on": { "kind": "host", "name": "edge" }, "port": 8080 } } } }
result
{
  errors = [ ];
  ok = true;
  violations = [ ];
}
ContributeCommunitySponsor