Diagram and docs batteries

Docs drift when they are written by hand. Batteries make diagrams, tables and links from the fleet itself: a renamed kind, a new host or a moved line shows up in the next nix run .#write-files, or fails the eval. This example is a small fleet to draw: environments hold hosts, and a host may refer to its database host.

# A small fleet to draw: environments hold hosts; a host may refer to its database host.
{ yen, ... }:
let
  inherit (yen.lib) all not maybe;
in
{
  yen.kinds.fleet = {
    spawns.environment = _: yen.environment;
    sinks.out.reduce = yen.lib.reduce.merge;
  };
  yen.kinds.environment = {
    spawns.host = yen.lib.pointing "environment" yen.host;
    # hosts report to their environment
    sinks.inventory.out =
      { scope, sinks, ... }:
      {
        inventory.${scope.name} = sinks.inventory;
      };
  };
  yen.kinds.host = {
    fields = {
      environment = yen.kinds.environment;
      db = maybe yen.kinds.host;
    };
    sinks.nixos = { };
  };
  yen.fleet.main = { };
  yen.environment = {
    prod = { };
    dev = { };
  };
  yen.host = {
    web.environment = yen.environment.prod;
    pg.environment = yen.environment.prod;
    lab = {
      environment = yen.environment.dev;
      db = yen.host.pg;
    };
  };
  yen.aspects = {
    server = {
      at = yen.kinds.host;
      inventory = { host, ... }: host.name;
    };
    legacy.at = yen.host.lab;
    monitored = {
      at = all [
        yen.aspects.server
        (not yen.aspects.legacy)
      ];
      nixos.services.prometheus.exporters.node.enable = true;
    };
    alerts.at = yen.aspects.monitored; # names monitored: monitored is a gate, and the diagram shows what it reads
  };
  yen.flake = { out, ... }: out;
}

The diagrams are values of the fleet. The aspect diagrams asks yen for the whole fleet with a tree request and draws each view from it.

# Diagrams of the fleet: every view is data ({ nodes; edges; }), flowchart renders it.
{ yen, ... }:
let
  whole.graph = true; # a tree request: the whole fleet
  inherit (yen.lib.batteries) mermaid;
  draw = direction: view: mermaid.flowchart { inherit direction; } view;
  # the diagrams of a fleet: `fleet` is the whole fleet, with its data, that the tree request selects
  diagramsOf = fleet: {
    # each environment as a subgraph around its hosts
    scopes = draw "TD" (
      mermaid.scopes {
        keep = s: !(fleet.lib.holds fleet.kinds.fleet s);
        label = s: s.name;
        group = s: if fleet.lib.holds fleet.kinds.host s then dirOf s.id else null; # (1) hosts inside their environment (the parent id)
      } fleet.lib.graph
    );
    kinds = draw "LR" (mermaid.kinds fleet.kinds); # (2)
    gates = draw "LR" (mermaid.gates fleet.aspects); # (3)
    routes = draw "BT" (mermaid.routes fleet.lib.graph); # (4)
    # (5) kind level: kind -> sink -> kind, counted; sinks as rounded nodes, outputs as cylinders
    routing = mermaid.flowchart {
      shapes = {
        sink = mermaid.shapes.round;
      };
    } (mermaid.routing fleet.lib.graph);
    sinks = mermaid.flowchart {
      shapes = {
        sink = mermaid.shapes.round;
        output = mermaid.shapes.cylinder;
      };
    } (mermaid.sinks fleet.lib);
  };
in
{
  yen.aspects.diagrams = {
    at = yen.fleet.main;
    cycle =
      { tree, ... }:
      {
        tree = [ whole ]; # (6)
        out = [ { diagrams = diagramsOf (tree.select whole); } ];
      };
  };
}

A view is plain data, { nodes; edges; }, and flowchart renders it as text. At (1), group puts each host in a subgraph named by its environment. (2) draws the kinds, (3) the aspects that other aspects name in at, and (4) who gives what to whom. At (5), routing counts at kind level and sinks shows the sinks that give to sinks and to flake outputs. At (6), tree = [ whole ] is the tree request.

Scopes, grouped by environment

A scope is drawn under its parent. The environments are subgraphs:

flowchart TD
  subgraph n879a4d9e78 ["dev"]
    nf470a97d16["lab"]
  end
  subgraph n4a308930d1 ["prod"]
    n6972941d96["pg"]
    n4019172c9d["web"]
  end

Kinds

A dotted edge is a field that refers to a kind.

flowchart LR
  nba34edff52["environment"]
  n0d07ea97e1["fleet"]
  n91a43a88c2["host"]
  nba34edff52 --> n91a43a88c2
  n0d07ea97e1 --> nba34edff52
  n91a43a88c2 -.->|"db"| n91a43a88c2
  n91a43a88c2 -.->|"environment"| nba34edff52

Gates

monitored reads server, and legacy under a not. A dashed edge is a read under not.

flowchart LR
  n8ad99c472c["legacy · 1 gated"]
  n80203bcb9f["monitored · 1 gated"]
  ncefa04ffa8["server · 1 gated"]
  nb2fb85131e["host:lab"]
  n8605f65baa["kind host"]
  nb2fb85131e --> n8ad99c472c
  ncefa04ffa8 --> n80203bcb9f
  n8ad99c472c -.->|"not"| n80203bcb9f
  n8605f65baa --> ncefa04ffa8

Routes

Who gives what to whom:

flowchart BT
  n6c4c773b6d["fleet main"]
  n879a4d9e78["environment dev"]
  nf470a97d16["host lab"]
  n4a308930d1["environment prod"]
  n6972941d96["host pg"]
  n4019172c9d["host web"]
  nf470a97d16 -->|"inventory"| n879a4d9e78
  n6972941d96 -->|"inventory"| n4a308930d1
  n4019172c9d -->|"inventory"| n4a308930d1
  n879a4d9e78 -->|"out"| n6c4c773b6d
  n4a308930d1 -->|"out"| n6c4c773b6d

Routing by kind, and sinks

flowchart LR
  nba34edff52["environment"]
  n0d07ea97e1["fleet"]
  n91a43a88c2["host"]
  n890cfd7ebf("inventory")
  ncf7fef0f82("links")
  n011dfa1a2c("nixos")
  ndc260b07ca("out")
  n0d07ea97e1 -->|"1"| ncf7fef0f82
  n0d07ea97e1 -->|"3"| ndc260b07ca
  n91a43a88c2 -->|"3"| n890cfd7ebf
  n91a43a88c2 -->|"2"| n011dfa1a2c
  n890cfd7ebf -->|"3"| nba34edff52
  ncf7fef0f82 -->|"1"| n0d07ea97e1
  n011dfa1a2c -->|"2"| n91a43a88c2
  ndc260b07ca -->|"3"| n0d07ea97e1

flowchart LR
  na06361dc4d("inventory")
  n7593b22b31("links")
  na788f0f454("nixos")
  nc03228363e("out")
  n4218c12206[("flake outputs")]
  na06361dc4d -->|"dev, prod"| n4218c12206
  n7593b22b31 -->|"main"| n4218c12206

Tables, links and a vocabulary

Tables, links that follow edits, and doc lines checked against the live names come from the same fleet:

# Docs that follow the fleet: a kinds table, links to the line that defines a thing, a doc line for every kind.
{ yen, ... }:
let
  inherit (yen.lib.batteries) docs;
  whole.graph = true; # a tree request: the whole fleet
  # the tables of a fleet: `fleet` is the whole fleet, with its data, that the tree request selects
  tablesOf = fleet: {
    kinds = docs.kinds fleet.kinds fleet.lib.graph; # (1)
    # any table: the hosts and their environment
    hosts = docs.table [ "Host" "Environment" ] (
      map (h: [
        h
        fleet.host.${h}.environment.name
      ]) (builtins.attrNames fleet.host)
    );
  };
in
{
  yen.kinds.fleet.sinks.links = {
    reduce = docs.links { root = ./.; }; # (2)
    out =
      { sinks, ... }:
      {
        links = sinks.links;
      };
  };
  yen.aspects.docs = {
    at = yen.fleet.main;
    links = {
      name = "db-field";
      file = "fleet.nix";
      needle = "db = maybe"; # (3)
      text = "db";
    };
    cycle =
      { tree, ... }:
      {
        tree = [ whole ];
        out = [ (tablesOf (tree.select whole)) ];
      };
    # flake templates from the directories under templates/ that have a flake.nix
    out.templates = builtins.mapAttrs (_: t: t.description) (
      docs.templates {
        root = ../templates;
        description = d: (import (d + "/flake.nix")).description;
      }
    );
    # a kind with no line here, or a line for a kind that is gone, is an error
    errors = docs.vocabulary {
      # (4)
      what = "kinds";
      names = builtins.attrNames yen.kinds;
      lines = {
        fleet = "the root";
        environment = "where hosts run";
        host = "a NixOS machine";
      };
    };
  };
}

(1) makes the table of kinds, with their spawns, sinks and counts:

Kind	Fields	Spawns	Sinks	Scopes
`environment`		`host`	`inventory`	2
`fleet`		`environment`	`links`, `out`	1
`host`	`db`, `environment`		`nixos`	3

Any table works the same way: the hosts and their environment.

Host	Environment
lab	dev
pg	prod
web	prod

(2) and (3) make a link to the first line of fleet.nix that holds the needle db = maybe. The reduce of the sink gives the links by name. A flake template for each directory that has a flake.nix:

{
  hello = "hello: the smallest flake";
}

The link follows the line: [db](fleet.nix#L23). At (4), a kind with no doc line is an error:

[
  {
    code = "doc-missing";
    error = "kinds `cluster`: a live name with no doc line";
  }
]

The batteries

Battery	Gives
`mermaid.flowchart { direction; shapes; } view`	the text of a flowchart; a node that others name as `group` is a subgraph
`mermaid.scopes { keep; label; group; edge; } graph`	a view of the scopes, each under its parent
`mermaid.routes graph`, `mermaid.routing graph`	who gives what to whom; at kind level with counts
`mermaid.kinds kinds`, `mermaid.sinks lib`	spawns and ref fields; sinks that give to sinks and to flake outputs
`mermaid.gates aspects`	the aspects other aspects name in `at`, and what each reads (dashed under `not`)
`docs.table head rows`, `docs.kinds kinds graph`	markdown tables
`docs.links { root; base; }`	a reduce: links to the first line that contains a needle; a gone needle is an error
`docs.vocabulary { what; names; lines; }`	errors: a live name with no doc line, a doc line with no live name
`docs.templates { root; description; }`	flake templates from the directories that have a flake.nix (these docs use it)

Every block on this page is a file of the template batteries-diagrams. Copy it and run it:

nix flake init -t github:vic/yen#batteries-diagrams

flake.nix
Directorymodules/
- diagrams.nix
- docs.nix
- fleet.nix
Directorytemplates/
- Directoryhello/
  - flake.nix
Directoryvariants/
- new-kind.nix

Contribute Community Sponsor