microservice-devops

microservice-devops — PM2 (dev) + Nomad (prod) operations playbook

This skill is the canonical doctrine for operating microservices and their web frontends in a hybrid model: PM2 for development, HashiCorp Nomad (with Consul, Vault, Traefik) for production, frequently colocated on a single shared host. This SKILL.md is the entry point — it tells future Claude what to read, in what order, and what hard rules to enforce. Deep procedures live in references/.

This skill is intentionally server- and domain-agnostic. Never bake in a specific IP address, hostname, domain, token, or credential. Always use placeholders: <host>, <service>, <domain>, <bridge-gateway-ip>, <token>. When you operate on a real server, read its actual state first — do not assume values from this doctrine.

When this skill applies

Trigger when the user is deploying, operating, hardening, or debugging services that use — or want to use — this stack:

• Development: Node/Go/other processes managed by PM2 (pm2 start, ecosystem.config.js, pm2 save/resurrect).
• Production: Nomad jobs (docker or exec driver), with Consul for service discovery, Vault for secrets, and Traefik (or nginx) for ingress.

Strong signals: an ecosystem.config.js; a *.nomad / *.nomad.hcl job spec with a template stanza using {{ range service "..." }} (Consul) or {{ with secret "..." }} (Vault); Consul/Nomad systemd units; host databases that containers reach via the container bridge gateway; or the user explicitly invoking this playbook.

If the project is unrelated to this stack, do not apply this skill — ask first.

Core mental model

                    ┌─────────────────── shared host ───────────────────┐
   DEV  (PM2)        │  pm2 ─┬─ svc-a (bind 127.0.0.1:PORT)              │
                     │       └─ svc-b (bind 127.0.0.1:PORT)              │
                     │                                                   │
   PROD (Nomad)      │  systemd ─ docker                                 │
                     │  systemd ─ consul ─┐ (service discovery)          │
                     │  systemd ─ vault ──┤ (secrets)                    │
                     │  systemd ─ nomad ──┴─ alloc ─ task (docker) ──┐   │
                     │                       │  reads Consul+Vault    │   │
                     │                       └─ registers in Consul   │   │
                     │  Traefik / nginx ── ingress ── :443 public ────┘   │
                     │  host db (mysql/postgres) bind 127.0.0.1,<bridge>  │
                     └───────────────────────────────────────────────────┘

Two control planes share one box. The dominant risks all stem from that: resource contention, port/exposure collisions, and blast radius from dev into prod. The hard rules below exist to contain those.

Hard rules (apply always)

Global numbering so you can cite "rule #N".

Isolation & safety

1. Dev never shares blast radius with prod. PM2 dev processes MUST bind to 127.0.0.1 (or a private interface), never 0.0.0.0, unless the user explicitly wants public exposure. Every PM2 app MUST set max_memory_restart so a leak cannot starve prod. See references/pm2-dev-workflow.md.
2. Production changes are confirmed before execution when irreversible or shared. Editing a Nomad job, restarting consul/nomad, reloading ingress, or rebinding a database are prod-affecting. Read current state, back up the file, validate (nginx -t, nomad job validate), then act. Localhost-only reversible reads need no confirmation.
3. Least exposure. Anything that does not need to be public binds to localhost or a private interface. A host service that containers consume must bind the container bridge gateway (see rule #6), not 0.0.0.0.

Service discovery, secrets, ordering

4. Nomad template dependencies are hard runtime dependencies. A task whose template renders {{ range service "X" }} (Consul) or {{ with secret "Y" }} (Vault) will NOT start until Consul/Vault are up and X/Y resolve. If a task is stuck "pending"/"failed" with a "Template failed" / "Missing: service" message, the root cause is upstream (Consul down, dependency service not healthy, Vault sealed/no token), not the task itself. See references/incident-playbook.md.
5. Platform daemons must survive reboot AND start in order. consul, vault, nomad, and docker MUST be systemctl enabled (verify with is-enabled), and Nomad MUST start after Consul and Docker. Use systemd drop-ins for ordering. If a Type=notify daemon flaps (systemd kills it every TimeoutStartUSec because it never signals READY), pin Type=exec via a drop-in. See references/platform-systemd.md.
6. Host services consumed by containers bind the bridge gateway. A database or cache on the host that a container connects to must listen on 127.0.0.1,<bridge-gateway-ip> (e.g. Docker's default bridge gateway), not 127.0.0.1 alone — otherwise containers get "connection refused". Add a systemd ordering drop-in so the DB starts after docker (so the bridge interface exists before the DB binds). See references/host-service-binding.md.

Production hygiene

7. Prod hosts carry runtime artifacts only. A deployed service directory keeps the built binary/bundle + its run spec (Nomad job / ecosystem.config.js / Dockerfile) + its .env/secret/config files. Delete source, build caches, go.mod/go.sum/package.json dev manifests, and never leave a .git folder on prod. See references/prod-hygiene.md.
8. Logs are bounded. Rotate or cap PM2 and container logs; an unbounded log file will silently eat the disk. See references/pm2-dev-workflow.md.

Security & maintenance

9. Patch on a schedule, reboot for kernels. Apply security updates (e.g. unattended-upgrades restricted to the distro's own origins, not random third-party repos), and reboot when a kernel/reboot-required flag is set — confirm timing with the user first. See references/security-maintenance.md.
10. Mitigate CVEs at the layer you control. When an upstream fix is unavailable or disabled in the installed package, reduce attack surface instead (e.g. disable an optional protocol feature on the affected vhost). Verify the actual installed package state before declaring "safe". See references/security-maintenance.md.

What to read, and when

Situation	Read
Setting up / managing dev processes	references/pm2-dev-workflow.md
Writing or operating a Nomad prod job	references/nomad-prod-workflow.md
Service discovery or secret rendering issues	references/consul-vault.md
Boot ordering, flapping daemon, enablement	references/platform-systemd.md
Container can't reach host DB/cache	references/host-service-binding.md
Cleaning a prod host / shipping a deploy	references/prod-hygiene.md
Patching, kernel reboot, CVE mitigation	references/security-maintenance.md
A service is down — diagnose root cause	references/incident-playbook.md

Operating discipline

• Read before you write. On a real host, inspect actual state (versions, binds, unit status, catalog) before acting. This doctrine describes patterns, not the truth of any given box.
• Validate before reload. nginx -t, nomad job validate, consul validate, vault operator ... dry-runs exist for a reason.
• Back up before edit. Copy a config to a timestamped .bak before changing it; record what you changed and how to revert.
• One change at a time when debugging prod, so you can attribute effects.