# NxM Infrastructure Project

## Primary Linux Server
- **IP:** 172.27.40.3
- **User:** nxm
- **Password:** 6589
- **OS:** Ubuntu Server (LTS)
- **Docker stack root:** `/opt/stacks/`

## Network
| VLAN | Name | Subnet | Gateway |
|---|---|---|---|
| 40 | Servers40 | 172.27.40.0/24 | 172.27.6.1 |
| 20 | Workshop20 | 172.27.20.0/24 | 172.27.6.1 |
| 10 | IoT10 | 172.27.10.0/24 | 172.27.6.1 |

## Key Devices
| Device | IP | Role |
|---|---|---|
| OPNsense Firewall | 172.27.6.1 | Firewall, router, DHCP |
| Ubuntu Server | 172.27.40.3 | Docker host, Headscale |
| TrueNAS | 172.27.40.5 | NAS storage |
| Home Assistant | 172.27.10.6 | Home automation (IoT10) |

## Docker Stacks & Ports
| Stack | Path | Port |
|---|---|---|
| Portainer | `/opt/stacks/portainer/` | 9443 (HTTPS) |
| Nginx Proxy Manager | `/opt/stacks/nginx-proxy-manager/` | 80, 81, 443 |
| Uptime Kuma | `/opt/stacks/uptime-kuma/` | 3002 |
| Zabbix | `/opt/stacks/zabbix/` | 8091 |
| Headscale | `/opt/stacks/headscale/` | 8080 (internal) |
| Headplane | `/opt/stacks/headplane/` | 3001 |
| Headscale UI | `/opt/stacks/headscale-ui/` | 3005 |
| Homarr | `/opt/stacks/homarr/` | 7575 |
| Dashy | `/opt/stacks/dashy/` | 4000 |
| Vaultwarden | `/opt/stacks/vaultwarden/` | 8222 |
| Netbird | `/opt/stacks/netbird/` | 3479/udp STUN |
| Caddy (Netbird sidecar) | `/opt/stacks/caddy-netbird/` | 8443/tcp — gRPC proxy for Netbird clients |
| Plane | `/opt/stacks/plane/` | 8095 (HTTP, via NPM) |
| Gitea | `/opt/stacks/gitea/` | 3000 (web), 2222 (SSH git) — self-hosted git, infrastructure docs |
| Open WebUI | `/opt/stacks/open-webui/` | 3010 — Chat UI for Ollama + MCP (replaced Flowise 2026-05-01) |
| agent-sites | `/opt/stacks/sites/` | internal only (proxy network) — nginx:alpine serving /opt/sites/ at agents.nxm.co.za |
| hodor-gateway | `/opt/stacks/hodor-gateway/` | 8200 — FastAPI agent gateway, POST /ask → Ollama |
| bran-changelog | `/opt/stacks/bran-changelog/` | one-shot container, run.sh + cron 06:00 daily |
| citadel-mcp | `/opt/stacks/citadel-mcp/` | 8300 — MCP SSE+HTTP server, tools: list_agents/get_agent_status/get_agent_output/web_search |
| varys-monitor | `/opt/stacks/varys-monitor/` | one-shot container, run.sh + cron every 15 min |
| sam-research | `/opt/stacks/sam-research/` | 8500 — Research agent, POST /research → SearXNG + Ollama |
| searxng | `/opt/stacks/searxng/` | 8600 — Self-hosted search backend (internal only, used by sam + citadel) |

## Public Subdomains (via NPM + Let's Encrypt)
| Subdomain | Internal Target |
|---|---|
| headscale.nxm.co.za | 172.27.40.3:8080 |
| vault.nxm.co.za | 172.27.40.3:8222 |
| kuma.nxm.co.za | 172.27.40.3:3002 |
| zabbix.nxm.co.za | 172.27.40.3:8091 |
| netbird.nxm.co.za | netbird-dashboard:80 via NPM (dashboard UI only) |
| netbird.nxm.co.za:8443 | Caddy sidecar → netbird-server:80 (client management, gRPC) |
| plane.nxm.co.za | 172.27.40.3:8095 |
| git.nxm.co.za | 172.27.40.3:3000 |
| rmm.nxm.co.za | 172.27.40.4:443 |
| api.nxm.co.za | 172.27.40.4:443 |
| mesh.nxm.co.za | 172.27.40.4:4430 |

## OPNsense Split DNS (Unbound Host Overrides)
All subdomains resolve to 172.27.40.3 internally via Unbound overrides.
If a subdomain isn't resolving internally, check:
1. OPNsense → Services → Unbound DNS → Overrides
2. Tailscale client on Windows — disconnect it, as it overrides local DNS

## Headscale (VPN)
- Version: v0.28
- Public URL: https://headscale.nxm.co.za
- Policy mode: database (live apply via API)
- **v0.28 breaking change:** All write operations require numeric user ID, not username
  - Get IDs: `headscale users list`
  - Example: `headscale preauthkeys create --user 13`
- Config: `/opt/stacks/headscale/config/config.yaml`
- After restart, always check: `tailscale status` — if server node offline: `sudo systemctl restart tailscaled`

## Vaultwarden
- Admin panel: https://vault.nxm.co.za/admin (token in OneNote)
- Signups currently OPEN (disable after all 4 users register)
- To disable: set `SIGNUPS_ALLOWED=false` in `/opt/stacks/vaultwarden/.env` → `docker compose up -d`
- Admin token needs Argon2 upgrade: `docker exec -it vaultwarden /vaultwarden hash --preset owasp`

## Domain Knowledge
- **Networking:** VLANs, inter-VLAN routing, firewall rules, NAT, split DNS, DHCP — comfortable at OPNsense config level, not just GUI clicks
- **DNS:** Unbound overrides, split-horizon, DNS-over-TLS, troubleshooting resolution order (Tailscale/Netbird conflict patterns)
- **VPN:** WireGuard fundamentals, Headscale (self-hosted Tailscale), Netbird (self-hosted, embedded Dex IdP), relay vs P2P, ACL/policy models
- **Docker:** Compose stacks, container networking, volume mounts, healthchecks, reverse proxy patterns — not Kubernetes
- **Linux:** Ubuntu Server admin, systemd, cron, file permissions, basic shell scripting — not a kernel developer
- **Reverse proxy:** NPM (OpenResty), Caddy — knows the difference between HTTP/HTTPS termination and gRPC proxying
- **Self-hosted services:** IPAM (NetBox), monitoring (Zabbix, Uptime Kuma, VictoriaMetrics, Grafana), dashboards (Homarr, Dashy), secrets (Vaultwarden)
- **Not expert in:** Kubernetes, cloud platforms (AWS/Azure/GCP), advanced Python (learning), application development

## Agent Guidelines
- Never take destructive or irreversible action without explicit confirmation (delete, overwrite, drop, reset)
- Never store credentials in output, logs, or generated files — reference credential location instead
- Always return structured output (JSON or markdown table) unless plain text is explicitly requested
- When suggesting shell commands, use PowerShell syntax for Windows-side tasks, bash for Linux server tasks
- SSH to Linux server always uses the Posh-SSH pattern defined in the Shell & Tools section
- Docker commands always use `docker compose` (not `docker-compose`) with `-q` on pulls
- If a task touches Headscale, Netbird, or NPM config — flag the relevant Known Issues entry before proceeding
- Prefer idempotent operations — scripts should be safe to run more than once
- When uncertain about current server state, ask rather than assume

## Credentials Location
- All service passwords: Jaco's OneNote + Nexum Password Spreadsheet
- No passwords stored in code or config files (except SSH for automation)

## Documentation (Gitea)
- Obsidian vaults migrated to Gitea on 2026-04-30
- **nxm-infrastructure** repo: `https://git.nxm.co.za/admin/nxm-infrastructure`
  - Local path: `/home/nxm/Documents/NxM Linux Server/`
  - Remote: `gitea-local:admin/nxm-infrastructure.git` (SSH)
- **nexum-projects** repo: `https://git.nxm.co.za/admin/nexum-projects`
  - Local path: `/home/nxm/Documents/Nexum Projects/`
  - Remote: `gitea-local:admin/nexum-projects.git` (SSH)
- **agent-os** repo: `https://git.nxm.co.za/admin/agent-os`
  - Local path: `/home/nxm/Documents/agent-os/`
  - Server runtime: `/opt/agent-os/` on 172.27.40.3
  - Remote: `gitea-local:admin/agent-os.git` (SSH)
- At end of any session that changes infrastructure: update relevant markdown files, commit and push
- Key files to update after infrastructure changes:
  - `Home.md` — master index
  - `Quick Reference/IP & Port Map.md` — any new service
  - `Quick Reference/Docker Stacks.md` — any new stack
  - Relevant `Services/<name>.md` — service-specific docs
  - `Troubleshooting/` — any new known issues

## Known Issues & Gotchas
- Docker pull logs are very verbose — always use `-q` flag
- Headscale YAML: never add duplicate `policy:` block (causes crash)
- Homarr board names must be slugs (no spaces)
- Vaultwarden requires HTTPS — LAN IP (port 8222) will show crypto error, use vault.nxm.co.za
- Tailscale client on Windows overrides DNS — disconnect when testing split DNS changes
- Chrome profile-specific links always open in last active profile (Windows limitation)
- NPM forward scheme is HTTP even for HTTPS external — NPM handles SSL termination
- Netbird STUN is on 3479/udp (not 3478 — Headscale owns that)
- Netbird client management URL is port 8443 (Caddy sidecar) — NOT 443
- NPM (OpenResty) has no gRPC module — Caddy sidecar is the workaround until Traefik migration
- Netbird config.yaml contains authSecret + encryptionKey — back this file up, losing it breaks all peers
- Servers running Tailscale must run `sudo tailscale set --accept-dns=false` before joining Netbird (Tailscale DNS overrides Unbound and resolves via public IP, breaking gRPC hairpin)
- Open WebUI → Citadel MCP: auth_type must be `none` — empty bearer key generates an illegal header and the connection silently fails
- Open WebUI connects via Streamable HTTP POST at `http://citadel-mcp:8300/mcp` — do NOT use /sse (Open WebUI 0.9+ only supports POST-based transport)