.github/copilot-instructions.md

@@ -1,91 +0,0 @@
-# Copilot Instructions — nixos-config
-
-## Architecture
-
-This is a **NixOS flake** managing multiple hosts using [flake-parts](https://flake.parts). The flake output is composed entirely from `flake-parts/*.nix` — each file is auto-imported via `builtins.readDir`.
-
-### Layers (top → bottom)
-
-1. **Hosts** (`hosts/`) — minimal per-machine config: profile import, overlay IP, underlay interface, enabled services. All `.nix` files in a host directory are auto-imported recursively by `flake-parts/hosts.nix`.
-   - **External hosts** (`external-hosts/`) — non-NixOS devices (e.g., a phone) that participate in the overlay network and syncthing cluster but aren't managed by NixOS. They import `nixosModules.default` directly (no profile) and only declare `custom.networking` and `custom.services` options so their config values are discoverable by other hosts. This enables auto-generating nebula certs, DNS records, and syncthing device lists for them.
-   - `allHosts` = `nixosConfigurations // externalConfigurations` — passed to every module via `specialArgs`, so any module can query the full fleet including external devices.
-2. **Profiles** (`profiles/`) — role presets. `core.nix` is the base for all hosts; `server.nix` and `workstation.nix` extend it. Profile names become `nixosModules.<name>-profile`.
-3. **Modules** (`modules/system/`, `modules/home/`) — reusable NixOS/Home Manager modules auto-imported as `nixosModules.default` / `homeModules.default`. Every module is always imported; activation is gated by `lib.mkEnableOption` + `lib.mkIf`.
-4. **Users** (`users/seb/`) — Home Manager config. Per-host overrides live in `@<hostname>/` subdirectories (e.g., `users/seb/@desktop/home.nix` imports `../home.nix` and adds host-specific settings).
-
-### Networking model
-
-- **Underlay** (`modules/system/networking/underlay.nix`) — physical network via systemd-networkd.
-- **Overlay** (`modules/system/networking/overlay.nix`, `modules/system/services/nebula/`) — Nebula mesh VPN (`10.254.250.0/24`, domain `splitleaf.de`). All inter-host communication (DNS, Caddy, SSH) routes over the overlay.
-- **DNS** (`modules/system/services/dns.nix`) — Unbound on overlay, auto-generates records from `allHosts`.
-
-### Web services pattern
-
-Each web service module (`modules/system/web-services/*.nix`) follows a consistent structure:
-
-```nix
-options.custom.web-services.<name> = { enable; domain; port; doBackups; };
-config = lib.mkIf cfg.enable {
-  # upstream NixOS service config
-  custom.services.caddy.virtualHosts.${cfg.domain}.port = cfg.port;  # reverse proxy
-  custom.services.restic.backups.<name> = lib.mkIf cfg.doBackups { ... };  # backup
-  custom.persistence.directories = [ ... ];  # impermanence
-};
-```
-
-Hosts enable services declaratively: `custom.web-services.forgejo = { enable = true; domain = "git.example.com"; doBackups = true; };`
-
-## Conventions
-
-- **All custom options** live under `custom.*` — never pollute the top-level NixOS namespace.
-- **`cfg` binding**: always `let cfg = config.custom.<path>;` at module top.
-- **Pipe operator** (`|>`): used pervasively instead of nested function calls.
-- **No repeated attrpaths** (per `statix`): group assignments into a single attrset instead of repeating the path. E.g. `custom.networking.overlay = { address = "..."; role = "server"; };` — not `custom.networking.overlay.address = "..."; custom.networking.overlay.role = "server";`. Setting a single attribute with the full path is fine. Conversely, don't nest single-key attrsets unnecessarily — use `custom.networking.overlay.address = "...";` not `custom = { networking = { overlay = { address = "..."; }; }; };`.
-- **`lib.singleton`** instead of `[ x ]` for single-element lists.
-- **`lib.mkEnableOption ""`**: empty string is intentional — descriptions come from the option path.
-- **Secrets**: [sops-nix](https://github.com/Mic92/sops-nix) with age keys. Each host/user has `secrets.json` + `keys/age.pub`. The `.sops.yaml` at repo root is a placeholder — the real config is generated via `nix build .#sops-config` (see `flake-parts/sops-config.nix`).
-- **Impermanence**: servers use `custom.persistence.enable = true` with an explicit `/persist` mount. Modules add their state directories via `custom.persistence.directories`.
-- **Formatting**: `nix fmt` runs `nixfmt` + `prettier` + `just --fmt` via treefmt.
-- **Path references**: use `./` for files in the same directory or a subdirectory. Use `${self}/...` when the path would require going up a directory (`../`). Never use `../`.
-- **Cross-host data**: modules receive `allHosts` via `specialArgs` (see Hosts layer above). Used by DNS, nebula static host maps, syncthing device lists, and caddy service records.
-
-## Developer Workflows
-
-| Task | Command |
-|---|---|
-| Rebuild & switch locally | `just switch` |
-| Test config without switching | `just test` |
-| Deploy to remote host(s) | `just deploy hostname1 hostname2` |
-| Format all files | `just fmt` or `nix fmt` |
-| Run flake checks + tests | `just check` |
-| Check without building | `just check-lite` |
-| Update flake inputs | `just update` |
-| Edit SOPS secrets | `just sops-edit hosts/<host>/secrets.json` |
-| Rotate all secrets | `just sops-rotate-all` |
-| Install a new host | `just install <host> root@<ip>` |
-| Open nix repl for a host | `just repl <hostname>` |
-
-SOPS commands auto-enter a `nix develop .#sops` shell if `sops` isn't available, which handles Bitwarden login and age key retrieval.
-
-## Adding a New Module
-
-1. Create `modules/system/services/<name>.nix` (or `web-services/`, `programs/`, etc.).
-2. Define options under `options.custom.<category>.<name>` with `lib.mkEnableOption ""`.
-3. Guard all config with `lib.mkIf cfg.enable { ... }`.
-4. For web services: set `custom.services.caddy.virtualHosts`, optionally `custom.services.restic.backups`, and `custom.persistence.directories`.
-5. No imports needed — the file is auto-discovered by `flake-parts/modules.nix`.
-
-## Adding a New Host
-
-1. Create `hosts/<hostname>/` with `default.nix`, `disko.nix`, `hardware.nix`, `secrets.json`, and `keys/` (containing `age.pub`, `nebula.pub`).
-2. In `default.nix`, import the appropriate profile (`self.nixosModules.server-profile` or `self.nixosModules.workstation-profile`) and set `custom.networking.overlay.address` + `custom.networking.underlay.*`.
-3. The host is auto-discovered by `flake-parts/hosts.nix` — no registration needed.
-
-## Tests
-
-Integration tests live in `tests/` and use NixOS VM testing (`pkgs.testers.runNixOSTest`). Run via `just check`. Key details:
-
-- Each test directory contains a `default.nix` that returns a test attrset (with `defaults`, `nodes`, `testScript`, etc.).
-- The `defaults` block imports `self.nixosModules.default` and **overrides `allHosts`** with the test's own `nodes` variable: `_module.args.allHosts = nodes |> lib.mapAttrs (_: node: { config = node; });`. This scopes cross-host lookups (DNS records, nebula static maps, etc.) to only the test's VMs, preventing evaluation of real host configs.
-- Test nodes define their own overlay addresses, underlay interfaces, and use pre-generated nebula keys from `tests/*/keys/`.
-- The `testScript` is written in Python, using helpers like `wait_for_unit`, `succeed`, and `fail` to assert behavior.

.sops.yaml

@@ -1,5 +1,5 @@
 # This is a placeholder.
-# The real .sops.yaml is generated via `nix build .#sops-config`.
-# See flake-parts/sops-config.nix for details.
+# The real .sops.yaml is generated via `nix build .#sops-config`
+# See flake-parts/sops.nix for details.
 
 creation_rules: []

flake.lock

@@ -37,6 +37,27 @@
         "type": "github"
       }
     },
+    "crowdsec": {
+      "inputs": {
+        "flake-utils": "flake-utils",
+        "nixpkgs": [
+          "nixpkgs"
+        ]
+      },
+      "locked": {
+        "lastModified": 1752497357,
+        "narHash": "sha256-9epXn1+T6U4Kfyw8B9zMzbERxDB3VfaPXhVebtai6CE=",
+        "ref": "refs/heads/main",
+        "rev": "84db7dcea77f7f477d79e69e35fb0bb560232667",
+        "revCount": 42,
+        "type": "git",
+        "url": "https://codeberg.org/kampka/nix-flake-crowdsec.git"
+      },
+      "original": {
+        "type": "git",
+        "url": "https://codeberg.org/kampka/nix-flake-crowdsec.git"
+      }
+    },
     "disko": {
       "inputs": {
         "nixpkgs": [
@@ -114,6 +135,23 @@
         "type": "github"
       }
     },
+    "flake-utils": {
+      "inputs": {
+        "systems": "systems"
+      },
+      "locked": {
+        "lastModified": 1731533236,
+        "narHash": "sha256-l0KFg5HjrsfsO/JpG+r7fRrqm12kzFHyUHqHCVpMMbI=",
+        "owner": "numtide",
+        "repo": "flake-utils",
+        "rev": "11707dc2f618dd54ca8739b309ec4fc024de578b",
+        "type": "github"
+      },
+      "original": {
+        "id": "flake-utils",
+        "type": "indirect"
+      }
+    },
     "home-manager": {
       "inputs": {
         "nixpkgs": [
@@ -274,6 +312,7 @@
       "inputs": {
         "betterfox": "betterfox",
         "comin": "comin",
+        "crowdsec": "crowdsec",
         "disko": "disko",
         "firefox-addons": "firefox-addons",
         "flake-parts": "flake-parts",
@@ -308,6 +347,21 @@
         "type": "github"
       }
     },
+    "systems": {
+      "locked": {
+        "lastModified": 1681028828,
+        "narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
+        "owner": "nix-systems",
+        "repo": "default",
+        "rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nix-systems",
+        "repo": "default",
+        "type": "github"
+      }
+    },
     "treefmt": {
       "inputs": {
         "nixpkgs": [

flake.nix

@@ -35,6 +35,11 @@
       inputs.nixpkgs.follows = "nixpkgs";
     };
 
+    crowdsec = {
+      url = "git+https://codeberg.org/kampka/nix-flake-crowdsec.git";
+      inputs.nixpkgs.follows = "nixpkgs";
+    };
+
     vscode-extensions = {
       url = "github:nix-community/nix-vscode-extensions";
       inputs.nixpkgs.follows = "nixpkgs";

modules/system/services/alloy.nix

@@ -32,6 +32,9 @@ in
         caddy = lib.mkEnableOption "" // {
           default = config.services.caddy.enable;
         };
+        crowdsec = lib.mkEnableOption "" // {
+          default = config.services.crowdsec.enable;
+        };
       };
       logs.openssh = lib.mkEnableOption "" // {
         default = config.services.openssh.enable;
@@ -136,6 +139,20 @@ in
             }
           '';
         };
+        "alloy/crowdsec-metrics.alloy" = {
+          enable = cfg.collect.metrics.crowdsec;
+          text = ''
+            prometheus.scrape "crowdsec" {
+              targets = [{
+                __address__ = "localhost:${toString config.custom.services.crowdsec.prometheusPort}",
+                job         = "crowdsec",
+                instance    = constants.hostname,
+              }]
+              forward_to      = [prometheus.remote_write.default.receiver]
+              scrape_interval = "15s"
+            }
+          '';
+        };
         "alloy/sshd-logs.alloy" = {
           enable = cfg.collect.logs.openssh;
           text = ''

modules/system/services/crowdsec/bouncers.nix

@@ -0,0 +1,40 @@
+{
+  config,
+  inputs,
+  pkgs,
+  lib,
+  ...
+}:
+let
+  cfg = config.custom.services.crowdsec;
+in
+{
+  imports = [ inputs.crowdsec.nixosModules.crowdsec-firewall-bouncer ];
+  disabledModules = [ "services/security/crowdsec-firewall-bouncer.nix" ];
+
+  options.custom.services.crowdsec.bouncers.firewall = lib.mkEnableOption "";
+
+  config = lib.mkIf cfg.bouncers.firewall {
+    services.crowdsec-firewall-bouncer = {
+      enable = true;
+      package = inputs.crowdsec.packages.${pkgs.stdenv.hostPlatform.system}.crowdsec-firewall-bouncer;
+      settings = {
+        api_key = "cs-firewall-bouncer";
+        api_url = "http://localhost:${toString cfg.apiPort}";
+      };
+    };
+
+    systemd.services.crowdsec.serviceConfig.ExecStartPre = lib.mkAfter (
+      lib.getExe (
+        pkgs.writeShellApplication {
+          name = "crowdsec-add-bouncer";
+          text = ''
+            if ! cscli bouncers list | grep -q "firewall"; then
+              cscli bouncers add "firewall" --key "cs-firewall-bouncer"
+            fi
+          '';
+        }
+      )
+    );
+  };
+}

modules/system/services/crowdsec/default.nix

@@ -0,0 +1,115 @@
+{
+  config,
+  inputs,
+  pkgs,
+  lib,
+  ...
+}:
+let
+  cfg = config.custom.services.crowdsec;
+
+  user = config.users.users.crowdsec.name;
+in
+{
+  disabledModules = [ "services/security/crowdsec.nix" ];
+  imports = [ inputs.crowdsec.nixosModules.crowdsec ];
+
+  options.custom.services.crowdsec = {
+    enable = lib.mkEnableOption "";
+    apiPort = lib.mkOption {
+      type = lib.types.port;
+      default = 8080;
+    };
+    prometheusPort = lib.mkOption {
+      type = lib.types.port;
+      default = 6060;
+    };
+    sources = {
+      iptables = lib.mkEnableOption "" // {
+        default = true;
+      };
+      caddy = lib.mkEnableOption "" // {
+        default = config.services.caddy.enable;
+      };
+      sshd = lib.mkEnableOption "" // {
+        default = config.services.openssh.enable;
+      };
+    };
+  };
+
+  config = lib.mkIf cfg.enable {
+    sops.secrets."crowdsec/enrollment-key" = {
+      owner = user;
+      restartUnits = [ "crowdsec.service" ];
+    };
+
+    users.groups.caddy.members = lib.mkIf cfg.sources.caddy [ user ];
+
+    services.crowdsec = {
+      enable = true;
+      package = inputs.crowdsec.packages.${pkgs.stdenv.hostPlatform.system}.crowdsec;
+      enrollKeyFile = config.sops.secrets."crowdsec/enrollment-key".path;
+      settings = {
+        api.server.listen_uri = "localhost:${toString cfg.apiPort}";
+        cscli.prometheus_uri = "http://localhost:${toString cfg.prometheusPort}";
+        prometheus = {
+          listen_addr = "localhost";
+          listen_port = cfg.prometheusPort;
+        };
+      };
+
+      allowLocalJournalAccess = true;
+      acquisitions = [
+        (lib.mkIf cfg.sources.iptables {
+          source = "journalctl";
+          journalctl_filter = [ "-k" ];
+          labels.type = "syslog";
+        })
+        (lib.mkIf cfg.sources.caddy {
+          filenames = [ "${config.services.caddy.logDir}/*.log" ];
+          labels.type = "caddy";
+        })
+        (lib.mkIf cfg.sources.sshd {
+          source = "journalctl";
+          journalctl_filter = [ "_SYSTEMD_UNIT=sshd.service" ];
+          labels.type = "syslog";
+        })
+      ];
+    };
+
+    systemd.services.crowdsec.serviceConfig = {
+      # Fix journalctl acquisitions
+      PrivateUsers = false;
+
+      ExecStartPre =
+        let
+          installCollection = collection: ''
+            if ! cscli collections list | grep -q "${collection}"; then
+              cscli collections install ${collection}
+            fi
+          '';
+          mkScript =
+            name: text:
+            lib.getExe (
+              pkgs.writeShellApplication {
+                inherit name text;
+              }
+            );
+          collectionsScript =
+            [
+              (lib.singleton "crowdsecurity/linux")
+              (lib.optional cfg.sources.iptables "crowdsecurity/iptables")
+              (lib.optional cfg.sources.caddy "crowdsecurity/caddy")
+              (lib.optional cfg.sources.sshd "crowdsecurity/sshd")
+            ]
+            |> lib.concatLists
+            |> lib.map installCollection
+            |> lib.concatLines
+            |> mkScript "crowdsec-install-collections";
+        in
+        lib.mkAfter collectionsScript;
+    };
+
+    custom.persistence.directories = [ "/var/lib/crowdsec" ];
+  };
+}

modules/system/web-services/grafana.nix

@@ -57,6 +57,9 @@ in
       victorialogs.enable = lib.mkEnableOption "" // {
         default = config.custom.web-services.victorialogs.enable;
       };
+      crowdsec.enable = lib.mkEnableOption "" // {
+        default = config.custom.services.crowdsec.enable;
+      };
     };
   };
 
@@ -173,6 +176,22 @@ in
             ''
           );
       };
+      # https://grafana.com/grafana/dashboards/19012-crowdsec-details-per-instance/
+      "grafana-dashboards/crowdsec-details-per-instance-patched.json" = {
+        enable = cfg.dashboards.crowdsec.enable;
+        source =
+          pkgs.fetchurl {
+            name = "crowdsec-details-per-instance.json";
+            url = "https://grafana.com/api/dashboards/19012/revisions/1/download";
+            hash = "sha256-VRPWAbPRgp+2pqfmey53wMqaOhLBzXVKUZs/pJ28Ikk=";
+          }
+          |> (
+            src:
+            pkgs.runCommand "crowdsec-details-per-instance-patched.json" { buildInputs = [ pkgs.gnused ]; } ''
+              sed 's/''${DS_PROMETHEUS}/Prometheus/g' ${src} > $out
+            ''
+          );
+      };
     };
 
     custom.services.caddy.virtualHosts.${cfg.domain}.port = cfg.port;