feat: new-service and delete-service scaffolder

One command from nothing to a running service: generate repo from template,
Actions secrets, namespace + unleash token, gitops render, kick CI, register
with ArgoCD once the first image exists, optional --wait health check.
delete-service reverses all of it. Idempotent, --dry-run supported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
nholtware 2026-11-05 12:03:55 -06:00
commit 253b6033c3
7 changed files with 809 additions and 0 deletions

1
.gitignore vendored Normal file
View file

@ -0,0 +1 @@
platform.env

2
.shellcheckrc Normal file
View file

@ -0,0 +1,2 @@
source-path=SCRIPTDIR
external-sources=true

96
README.md Normal file
View file

@ -0,0 +1,96 @@
# platform
One command from nothing to a running, observed service on the homelab k3s
cluster, and one to take it all away again. Bash + `curl`, `jq`, `git`,
`kubectl` (`kustomize` optional; falls back to `kubectl kustomize`).
```sh
new-service orders --kind backend --with postgres,redis --wait
new-service orders-web --kind frontend --api orders --wait
delete-service orders-web --yes && delete-service orders --yes
```
## Setup
```sh
git clone http://192.168.1.230:3000/nholtware/platform.git ~/Code/forgejo.local/nholtware/platform
cd ~/Code/forgejo.local/nholtware/platform
cp platform.env.example platform.env # gitignored; defaults match the homelab
export PATH="$PWD/bin:$PATH" # not installed globally
```
`platform.env` points at the homelab repo's gitignored `.secrets/` token files,
the gitops clone, the kubeconfig and the ingress domain. The scripts refuse to
run if any of them is missing, the gitops clone is dirty or not on `main`, the
Forgejo token is rejected, or the cluster is unreachable.
## new-service
```
new-service NAME --kind backend|frontend [--with postgres,redis] [--api BACKEND]
[--wait] [--timeout SECONDS] [--dry-run]
```
| Step | What | Skipped when |
|------|------|--------------|
| 1 | validate `NAME` (`^[a-z][a-z0-9-]{1,30}$`, not reserved), kind, flags; `--api` must be an existing backend | — |
| 2 | generate public repo from `service-template-express` / `service-template-react` | repo exists |
| 3 | set Actions secrets `REGISTRY_USER`, `REGISTRY_TOKEN`, `GITOPS_TOKEN` | never (PUT is idempotent) |
| 4 | create namespace + Secret `unleash-client-token` (copied from `unleash/unleash-init-tokens`) | they exist |
| 5 | render gitops `templates/<kind>` (+ `postgres`, `redis`) → `manifests/NAME`, `kustomize build` lint, commit, push | dir exists |
| 6 | clone repo to `$CODE_DIR/NAME`, push empty commit `chore: kick ci` | CI already deployed, or a kick is running/succeeded |
| 7 | wait for CI's `gitops-bump` to replace `newTag: latest` (fails fast if the CI run fails) | tag already bumped |
| 8 | write `apps/NAME.yaml`, commit, push, refresh ArgoCD `root` | file exists |
| 9 | `--wait`: ArgoCD app `Synced/Healthy`, then `/readyz` (backend) or `/healthz` + `/api/readyz` (frontend) → 200 | no `--wait` |
Then it prints the service, ArgoCD, Kibana (pre-filtered on `service:"NAME"`)
and Unleash URLs, and the elapsed time.
**Why `apps/NAME.yaml` comes last:** ArgoCD only sees the service once CI has
pushed a real image. Registering it earlier made the backend's migrate Job sit
in `ImagePullBackOff` on `:latest` and hold the sync forever (Plan 04 caveat).
**Resumable:** every step checks before acting. If a run dies (CI failure,
timeout, Ctrl-C), fix the cause and re-run the same command; a failed CI run is
re-kicked automatically.
**Kinds:**
- `backend` — Express/TS. Runtime config (`NAME-runtime-config` ConfigMap) is
always included. `--with postgres` adds a CNPG `NAME-db` cluster + migrate
Job, `--with redis` adds `NAME-redis`. Without them the env vars and Job are
dropped.
- `frontend` — Vite/React on nginx. `--api BACKEND` deploys into the backend's
namespace and routes `NAME.<domain>/api/*` to it (prefix stripped). Without
`--api` it gets its own namespace and no `/api` route.
## delete-service
```
delete-service NAME [--yes] [--dry-run] [--timeout SECONDS]
```
Removes `apps/NAME.yaml` + `manifests/NAME` from gitops (the Application's
`resources-finalizer` makes ArgoCD delete its workloads when `root` prunes it),
waits for the Application to go, deletes the namespace if it is the service's
own (CNPG PVCs go with it), deletes every version of the container package and
the Forgejo repo, and removes the local clone. Asks you to type the name unless
`--yes`. Refuses while a frontend still deploys into the service's namespace
(delete the frontend first). Leaves a shared namespace and its
`unleash-client-token` Secret alone.
## --dry-run
Runs every read-only check for real (config, repo exists?, namespace exists?,
render + `kustomize build` lint into a temp dir) and prints each mutating API
call (secret values redacted), git push and kubectl change instead of doing it.
## Notes
- The Forgejo admin token has no package scope; package list/delete uses the
registry push token (same user).
- Git pushes use the admin token through an inline credential helper; it is
never written into a remote URL or `argv`.
- Platform UIs: ArgoCD, Kibana and Unleash at `argocd.` / `kibana.` /
`unleash.192.168.1.83.nip.io`. Credentials and conventions: homelab
`PLATFORM.md`.

152
bin/delete-service Executable file
View file

@ -0,0 +1,152 @@
#!/usr/bin/env bash
# delete-service: reverse new-service. gitops entry (ArgoCD prunes), namespace
# (and with it CNPG PVCs), Forgejo repo, container package versions, local clone.
# delete-service orders-web --yes && delete-service orders --yes
set -euo pipefail
# shellcheck source=../lib/common.sh
source "$(dirname "$(readlink -f "$0")")/../lib/common.sh"
usage() {
cat <<'EOF'
usage: delete-service NAME [--yes] [--dry-run] [--timeout SECONDS]
--yes don't ask for confirmation
--dry-run print every API call, git push and kubectl mutation instead
--timeout per-wait timeout (default 600)
EOF
}
NAME='' YES=0 TIMEOUT=600
while (($#)); do
case $1 in
--yes | -y) YES=1; shift ;;
--dry-run) DRY_RUN=1; shift ;;
--timeout) TIMEOUT=${2:?--timeout needs a value}; shift 2 ;;
-h | --help) usage; exit 0 ;;
-*) usage >&2; die "unknown option: $1" ;;
*) [[ -z $NAME ]] || die "unexpected argument: $1"; NAME=$1; shift ;;
esac
done
[[ -n $NAME ]] || { usage >&2; die "NAME is required"; }
validate_name "$NAME"
is_dry && log "DRY RUN: nothing below is executed except read-only checks"
load_config
gitops_sync
# ------------------------------------------------------------ what exists
NS=$(app_namespace "$NAME")
if [[ -z $NS ]]; then
NS=$(kubectl get application -n argocd "$NAME" -o jsonpath='{.spec.destination.namespace}' 2>/dev/null || true)
fi
NS=${NS:-$NAME}
OWN_NS=0
[[ $NS == "$NAME" ]] && OWN_NS=1
# Refuse to pull a namespace out from under a frontend co-located with us.
dependents=()
for f in "$GITOPS_DIR"/apps/*.yaml; do
other=$(basename "$f" .yaml)
[[ $other != "$NAME" && $(app_namespace "$other") == "$NAME" ]] && dependents+=("$other")
done
((${#dependents[@]} == 0)) || die "${dependents[*]} deploy(s) into namespace $NAME; delete-service them first"
APP_DIR=$CODE_DIR/$NAME
cat >&2 <<EOF
This permanently deletes, for service '$NAME':
- gitops apps/$NAME.yaml and manifests/$NAME/ (ArgoCD prunes the workloads)
- $([[ $OWN_NS == 1 ]] && echo "namespace $NAME, including its Postgres data (PVCs)" || echo "nothing else in shared namespace $NS (left in place)")
- Forgejo repo $FORGEJO_OWNER/$NAME and its container package versions
- local clone $APP_DIR
EOF
if ((!YES)) && ! is_dry; then
read -r -p "Type the service name to confirm: " answer
[[ $answer == "$NAME" ]] || die "aborted"
fi
# ------------------------------------------------------------ 1. ArgoCD app + gitops
# The resources finalizer makes ArgoCD delete the app's workloads when the
# root app prunes the Application (older apps were created without it).
if kubectl get application -n argocd "$NAME" >/dev/null 2>&1; then
run kubectl patch application -n argocd "$NAME" --type merge \
-p '{"metadata":{"finalizers":["resources-finalizer.argocd.argoproj.io"]}}' >/dev/null
fi
paths=()
[[ -f $GITOPS_DIR/apps/$NAME.yaml ]] && paths+=("apps/$NAME.yaml")
[[ -d $GITOPS_DIR/manifests/$NAME ]] && paths+=("manifests/$NAME")
if ((${#paths[@]})); then
if is_dry; then
dry "git -C $GITOPS_DIR rm -r ${paths[*]}"
else
git -C "$GITOPS_DIR" rm -rq -- "${paths[@]}"
fi
gitops_commit_push "chore($NAME): delete service" "${paths[@]}"
argocd_refresh root
else
ok "gitops has no entry for $NAME"
fi
if ! is_dry; then
app_gone() { ! kubectl get application -n argocd "$NAME" >/dev/null 2>&1; }
if ! app_gone; then
# If root never managed it (e.g. scaffold stopped before registering), delete directly.
sleep 15
app_gone || kubectl get application -n argocd root -o json |
jq -e --arg n "$NAME" '.status.resources[]? | select(.kind == "Application" and .name == $n)' >/dev/null ||
kubectl delete application -n argocd "$NAME" --wait=false >/dev/null
wait_until "$TIMEOUT" 5 "ArgoCD to prune application $NAME and its resources" app_gone
else
ok "no ArgoCD application $NAME"
fi
fi
# ------------------------------------------------------------ 2. namespace
if ((OWN_NS)); then
if kubectl get namespace "$NAME" >/dev/null 2>&1; then
run kubectl delete namespace "$NAME" --wait=false >/dev/null
if ! is_dry; then
ns_gone() { ! kubectl get namespace "$NAME" >/dev/null 2>&1; }
wait_until "$TIMEOUT" 5 "namespace $NAME to terminate" ns_gone
fi
else
ok "namespace $NAME absent"
fi
fi
# ------------------------------------------------------------ 3. container package
# The admin token has no package scope; the registry push token (same user) does.
deleted=0
while :; do
api GET "/packages/$FORGEJO_OWNER?type=container&q=$NAME&limit=50" '' registry
[[ $API_CODE == 200 ]] || die "listing packages failed ($API_CODE): $API_BODY"
mapfile -t versions < <(jq -r --arg n "$NAME" '.[] | select(.name == $n) | .version' <<<"$API_BODY")
((${#versions[@]})) || break
for v in "${versions[@]}"; do
api_write DELETE "/packages/$FORGEJO_OWNER/container/$NAME/$(jq -rn --arg v "$v" '$v | @uri')" '' registry
[[ $API_CODE =~ ^20[04]$ || $API_CODE == 404 ]] || die "delete package $NAME/$v failed ($API_CODE): $API_BODY"
deleted=$((deleted + 1))
done
is_dry && break
done
ok "container package $NAME: $deleted version(s) deleted"
# ------------------------------------------------------------ 4. repo
if repo_exists "$NAME"; then
[[ $(jq -r .template <<<"$API_BODY") != true ]] || die "refusing to delete template repo $NAME"
api_write DELETE "/repos/$FORGEJO_OWNER/$NAME"
[[ $API_CODE == 204 || $API_CODE == 200 ]] || die "delete repo failed ($API_CODE): $API_BODY"
ok "repo $FORGEJO_OWNER/$NAME deleted"
else
ok "repo $FORGEJO_OWNER/$NAME absent"
fi
# ------------------------------------------------------------ 5. local clone
if [[ -d $APP_DIR ]]; then
origin=$(git -C "$APP_DIR" remote get-url origin 2>/dev/null || true)
[[ $origin == "$FORGEJO_URL/$FORGEJO_OWNER/$NAME.git" ]] ||
die "$APP_DIR origin is '$origin', not this service's repo; not deleting it"
run rm -rf -- "$APP_DIR"
ok "removed $APP_DIR"
fi
ok "$NAME deleted in $(elapsed)s"

312
bin/new-service Executable file
View file

@ -0,0 +1,312 @@
#!/usr/bin/env bash
# new-service: one command from nothing to a running, observed service.
# new-service orders --kind backend --with postgres,redis --wait
# new-service orders-web --kind frontend --api orders --wait
# Idempotent: every step checks before acting, so re-running after a failure
# continues where it stopped. See README.md.
set -euo pipefail
# shellcheck source=../lib/common.sh
source "$(dirname "$(readlink -f "$0")")/../lib/common.sh"
usage() {
cat <<'EOF'
usage: new-service NAME --kind backend|frontend [options]
--kind backend|frontend template to generate from (required)
--with LIST backend only: comma list of postgres,redis
(runtime-config is always included)
--api BACKEND frontend only: route /api to BACKEND and deploy
into its namespace
--wait after deploy, wait until Healthy and the health
endpoint answers; print elapsed time
--timeout SECONDS per-wait timeout (default 900)
--dry-run print every API call, git push and kubectl
mutation instead of executing it
-h, --help this text
EOF
}
NAME='' KIND='' WITH='' API_APP='' WAIT=0 TIMEOUT=900
while (($#)); do
case $1 in
--kind) KIND=${2:?--kind needs a value}; shift 2 ;;
--with) WITH=${2:?--with needs a value}; shift 2 ;;
--api) API_APP=${2:?--api needs a value}; shift 2 ;;
--wait) WAIT=1; shift ;;
--timeout) TIMEOUT=${2:?--timeout needs a value}; shift 2 ;;
--dry-run) DRY_RUN=1; shift ;;
-h | --help) usage; exit 0 ;;
-*) usage >&2; die "unknown option: $1" ;;
*) [[ -z $NAME ]] || die "unexpected argument: $1"; NAME=$1; shift ;;
esac
done
# ------------------------------------------------------------ 1. validate
[[ -n $NAME ]] || { usage >&2; die "NAME is required"; }
validate_name "$NAME"
[[ $TIMEOUT =~ ^[0-9]+$ ]] || die "--timeout must be a number of seconds"
WITH_PG=0 WITH_REDIS=0
case $KIND in
backend)
[[ -z $API_APP ]] || die "--api is for --kind frontend"
IFS=, read -ra _with <<<"$WITH"
for w in "${_with[@]}"; do
case $w in
postgres) WITH_PG=1 ;;
redis) WITH_REDIS=1 ;;
runtime-config | '') ;; # always on
*) die "--with: unknown component '$w' (postgres, redis)" ;;
esac
done
TEMPLATE_REPO=service-template-express
NS=$NAME
HEALTH_PATH=/readyz
;;
frontend)
[[ -z $WITH ]] || die "--with is for --kind backend"
TEMPLATE_REPO=service-template-react
if [[ -n $API_APP ]]; then
validate_name "$API_APP"
[[ $API_APP != "$NAME" ]] || die "--api must name a different service"
NS=$API_APP
else
NS=$NAME
fi
HEALTH_PATH=/healthz
;;
'') usage >&2; die "--kind is required" ;;
*) die "--kind must be backend or frontend" ;;
esac
is_dry && log "DRY RUN: nothing below is executed except read-only checks"
load_config
gitops_sync
if [[ -n $API_APP ]]; then
[[ -f $GITOPS_DIR/apps/$API_APP.yaml && -d $GITOPS_DIR/manifests/$API_APP ]] ||
die "--api $API_APP: no such service in gitops (create it first)"
[[ $(app_namespace "$API_APP") == "$API_APP" ]] ||
die "--api $API_APP: not a backend (its namespace is '$(app_namespace "$API_APP")')"
fi
if [[ -f $GITOPS_DIR/apps/$NAME.yaml && $(app_namespace "$NAME") != "$NS" ]]; then
die "gitops apps/$NAME.yaml exists with namespace '$(app_namespace "$NAME")', expected '$NS'"
fi
ok "validated $NAME (kind=$KIND ns=$NS postgres=$WITH_PG redis=$WITH_REDIS api=${API_APP:--})"
# ------------------------------------------------------------ 2. repo
if repo_exists "$NAME"; then
ok "repo $FORGEJO_OWNER/$NAME exists"
[[ $(jq -r .private <<<"$API_BODY") == false ]] || die "repo $FORGEJO_OWNER/$NAME is private; the cluster pulls images anonymously"
else
log "generating $FORGEJO_OWNER/$NAME from $TEMPLATE_REPO"
body=$(jq -nc --arg o "$FORGEJO_OWNER" --arg n "$NAME" '{owner:$o, name:$n, git_content:true, private:false}')
api_write POST "/repos/$FORGEJO_OWNER/$TEMPLATE_REPO/generate" "$body"
[[ $API_CODE == 201 || $API_CODE == 200 ]] || die "generate failed ($API_CODE): $API_BODY"
ok "repo created: $FORGEJO_URL/$FORGEJO_OWNER/$NAME"
fi
# ------------------------------------------------------------ 3. Actions secrets
set_secret() {
local body
body=$(jq -nc --arg d "$2" '{data:$d}')
api_write PUT "/repos/$FORGEJO_OWNER/$NAME/actions/secrets/$1" "$body" admin 1
[[ $API_CODE =~ ^20[014]$ ]] || die "set secret $1 failed ($API_CODE): $API_BODY"
}
set_secret REGISTRY_USER "$FORGEJO_OWNER"
set_secret REGISTRY_TOKEN "$REGISTRY_TOKEN"
set_secret GITOPS_TOKEN "$GITOPS_TOKEN"
ok "Actions secrets REGISTRY_USER, REGISTRY_TOKEN, GITOPS_TOKEN set"
# ------------------------------------------------------------ 4. namespace + unleash token
if kubectl get namespace "$NS" >/dev/null 2>&1; then
ok "namespace $NS exists"
else
run kubectl create namespace "$NS" >/dev/null
ok "namespace $NS created"
fi
if kubectl get secret -n "$NS" unleash-client-token >/dev/null 2>&1; then
ok "secret $NS/unleash-client-token exists"
else
# Copy the base64 values straight across; tokens are never decoded or put in argv.
manifest=$(kubectl get secret -n unleash unleash-init-tokens -o json | jq --arg ns "$NS" '{
apiVersion: "v1", kind: "Secret", type: "Opaque",
metadata: {name: "unleash-client-token", namespace: $ns,
labels: {"app.kubernetes.io/managed-by": "platform-new-service"}},
data: {token: .data.INIT_CLIENT_API_TOKENS, frontendToken: .data.INIT_FRONTEND_API_TOKENS}}')
[[ $(jq -r '.data.token // empty' <<<"$manifest") ]] || die "could not read unleash/unleash-init-tokens"
if is_dry; then
dry "kubectl create -f - (Secret $NS/unleash-client-token from unleash/unleash-init-tokens)"
else
kubectl create -f - <<<"$manifest" >/dev/null
fi
ok "secret $NS/unleash-client-token created"
fi
# ------------------------------------------------------------ 5. gitops manifests
# apps/<name>.yaml is written only after CI's first image exists (step 7), so
# ArgoCD never syncs `newTag: latest` before there is an image to pull (the
# migrate-Job ImagePullBackOff caveat from Plan 04).
render_manifests() {
local out=$1 t=$GITOPS_DIR/templates/$KIND
cp -r "$t/." "$out/"
rm -f "$out/README.md" "$out/application.yaml.tmpl"
if [[ $KIND == backend ]]; then
local extra=()
if ((WITH_PG)); then
cp "$GITOPS_DIR/templates/postgres/cluster.yaml" "$out/postgres.yaml"
extra+=(postgres.yaml)
else
rm "$out/migrate-job.yaml"
sed -i '/^ - migrate-job.yaml$/d' "$out/kustomization.yaml"
sed -i '/- name: DATABASE_URL$/,/secretKeyRef:/d' "$out/deployment.yaml"
fi
if ((WITH_REDIS)); then
cp "$GITOPS_DIR/templates/redis/redis.yaml" "$out/redis.yaml"
extra+=(redis.yaml)
else
sed -i '/- name: REDIS_URL$/,+1d' "$out/deployment.yaml"
fi
# Replace the "Added by the render recipe" comment with the actual list.
local add='' f
for f in "${extra[@]}"; do add+=" - $f\n"; done
sed -i "s|^ # Added by the render recipe.*$|${add%\\n}|" "$out/kustomization.yaml"
sed -i '/^$/d' "$out/kustomization.yaml"
sed -i "s/__APP__/$NAME/g" "$out"/*.yaml
else
if [[ -z $API_APP ]]; then
rm "$out/ingress-api.yaml" "$out/middleware.yaml"
sed -i '/ingress-api.yaml$/d; /middleware.yaml$/d; /# \/api -> backend/d' "$out/kustomization.yaml"
fi
sed -i "s/__APP__/$NAME/g; s/__API__/$API_APP/g" "$out"/*.yaml
fi
! grep -rn '__[A-Z]*__' "$out" || die "unrendered placeholder left in manifests"
kustomize_build "$out" >/dev/null || die "kustomize build of rendered manifests failed"
}
if [[ -d $GITOPS_DIR/manifests/$NAME ]]; then
ok "gitops manifests/$NAME exists"
else
stage=$(tmpdir)
render_manifests "$stage"
ok "rendered + linted manifests ($(cd "$stage" && echo *.yaml))"
if is_dry; then
dry "write $GITOPS_DIR/manifests/$NAME/{$(cd "$stage" && printf "%s," *.yaml | sed "s/,$//")}"
sed 's/^/ /' "$stage/kustomization.yaml" >&2
else
cp -r "$stage" "$GITOPS_DIR/manifests/$NAME"
chmod 755 "$GITOPS_DIR/manifests/$NAME"
fi
gitops_commit_push "feat($NAME): scaffold $KIND manifests" "manifests/$NAME"
fi
# ------------------------------------------------------------ 6. clone + kick CI
APP_DIR=$CODE_DIR/$NAME
REPO_URL=$FORGEJO_URL/$FORGEJO_OWNER/$NAME.git
deployed_tag() { # tag CI has written to gitops origin/main ('' if none yet)
git_auth -C "$GITOPS_DIR" fetch -q origin main 2>/dev/null || return 0
git -C "$GITOPS_DIR" show "origin/main:manifests/$NAME/kustomization.yaml" 2>/dev/null |
awk '/newTag:/{print $2}' | grep -vx latest || true
}
if [[ -d $APP_DIR/.git ]]; then
ok "local clone $APP_DIR exists"
elif is_dry; then
dry "git clone $REPO_URL $APP_DIR"
else
git_auth clone -q "$REPO_URL" "$APP_DIR"
ok "cloned to $APP_DIR"
fi
# A CI task is live/finished for the kick sha?
ci_state() { # prints: none | running | success | failure
api GET "/repos/$FORGEJO_OWNER/$NAME/actions/tasks?limit=20"
[[ $API_CODE == 200 ]] || { echo none; return; }
jq -r --arg sha "$1" '[.workflow_runs[] | select(.head_sha == $sha) | .status] as $s
| if ($s | length) == 0 then "none"
elif any($s[]; . == "failure" or . == "cancelled") then "failure"
elif any($s[]; . == "running" or . == "waiting" or . == "blocked") then "running"
elif any($s[]; . == "success") and ($s | length) >= 2 then "success"
else "running" end' <<<"$API_BODY"
}
TAG=$(deployed_tag)
if [[ -n $TAG ]]; then
ok "CI already deployed $NAME:$TAG"
elif is_dry; then
dry "git -C $APP_DIR commit --allow-empty -m 'chore: kick ci' && git push origin main"
dry "poll gitops origin/main manifests/$NAME/kustomization.yaml until newTag != latest"
else
git_auth -C "$APP_DIR" pull -q --ff-only origin main
KICK_SHA=$(git -C "$APP_DIR" log -1 --format=%H --grep='^chore: kick ci$' origin/main || true)
if [[ -n $KICK_SHA && $(ci_state "$KICK_SHA") != failure ]]; then
ok "CI already kicked (${KICK_SHA:0:8})"
else
[[ -n $KICK_SHA ]] && warn "previous CI run for ${KICK_SHA:0:8} failed; kicking again"
git -C "$APP_DIR" -c user.name="$FORGEJO_OWNER" -c user.email="$FORGEJO_OWNER@forgejo.local" \
commit -q --allow-empty -m 'chore: kick ci'
git_auth -C "$APP_DIR" push -q origin main
KICK_SHA=$(git -C "$APP_DIR" rev-parse HEAD)
ok "pushed 'chore: kick ci' (${KICK_SHA:0:8}); CI: $FORGEJO_URL/$FORGEJO_OWNER/$NAME/actions"
fi
# ---------------------------------------------------------- 7. wait for first image
ci_done() {
TAG=$(deployed_tag)
[[ -n $TAG ]] && return 0
[[ $(ci_state "$KICK_SHA") != failure ]] ||
die "CI failed for ${KICK_SHA:0:8}: $FORGEJO_URL/$FORGEJO_OWNER/$NAME/actions (re-run new-service to kick again)"
return 1
}
wait_until "$TIMEOUT" 10 "CI to build $NAME and bump its tag in gitops" ci_done
ok "CI deployed $NAME:$TAG at $(elapsed)s"
fi
# ------------------------------------------------------------ 8. register with ArgoCD
if [[ -f $GITOPS_DIR/apps/$NAME.yaml ]]; then
ok "gitops apps/$NAME.yaml exists"
else
is_dry || git_auth -C "$GITOPS_DIR" pull -q --ff-only origin main
if is_dry; then
dry "write $GITOPS_DIR/apps/$NAME.yaml (namespace $NS)"
else
sed "s/__APP__/$NAME/g; s/__NS__/$NS/g" "$GITOPS_DIR/templates/$KIND/application.yaml.tmpl" \
>"$GITOPS_DIR/apps/$NAME.yaml"
fi
gitops_commit_push "feat($NAME): register ArgoCD application" "apps/$NAME.yaml"
fi
# Don't wait for ArgoCD's 3-min poll.
argocd_refresh root
# ------------------------------------------------------------ 9. wait for healthy
HOST=$NAME.$INGRESS_DOMAIN
if ((WAIT)) && ! is_dry; then
app_ok() {
local s
s=$(kubectl get application -n argocd "$NAME" -o json 2>/dev/null |
jq -r '"\(.status.sync.status)/\(.status.health.status)"') || return 1
[[ $s == Synced/Healthy ]]
}
wait_until "$TIMEOUT" 10 "ArgoCD app $NAME Synced/Healthy" app_ok
http_ok() {
local p
for p in "$@"; do
[[ $(curl -s -o /dev/null -w '%{http_code}' --max-time 5 \
--resolve "$HOST:80:$INGRESS_IP" "http://$HOST$p") == 200 ]] || return 1
done
}
paths=("$HEALTH_PATH")
[[ -n $API_APP ]] && paths+=(/api/readyz)
wait_until 180 5 "http://$HOST${paths[*]} -> 200" http_ok "${paths[@]}"
for p in "${paths[@]}"; do
printf ' GET %s -> %s\n' "$p" "$(curl -s --max-time 5 --resolve "$HOST:80:$INGRESS_IP" "http://$HOST$p")" >&2
done
fi
print_urls "$NAME" "$KIND"
if is_dry; then
log "dry run complete"
elif ((WAIT)); then
ok "$NAME is live: $(elapsed)s from invocation"
else
ok "$NAME registered; ArgoCD deploys it shortly (re-run with --wait to block until healthy)"
fi

230
lib/common.sh Normal file
View file

@ -0,0 +1,230 @@
# shellcheck shell=bash
# shellcheck disable=SC2034 # globals here are read by bin/* (DRY_RUN, API_BODY, GITOPS_TOKEN)
# Shared helpers for bin/new-service and bin/delete-service.
# Sourced, not executed. Callers set `set -euo pipefail` themselves.
PLATFORM_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
DRY_RUN=${DRY_RUN:-0}
START_TS=$(date +%s)
# ---------------------------------------------------------------- logging
if [[ -t 2 ]]; then
_c_info=$'\033[1;34m' _c_ok=$'\033[1;32m' _c_warn=$'\033[1;33m' _c_err=$'\033[1;31m' _c_dim=$'\033[2m' _c_off=$'\033[0m'
else
_c_info='' _c_ok='' _c_warn='' _c_err='' _c_dim='' _c_off=''
fi
_ts() { local s=$(($(date +%s) - START_TS)); printf '%02d:%02d' $((s / 60)) $((s % 60)); }
log() { printf '%s[%s] ==>%s %s\n' "$_c_info" "$(_ts)" "$_c_off" "$*" >&2; }
ok() { printf '%s[%s] ok%s %s\n' "$_c_ok" "$(_ts)" "$_c_off" "$*" >&2; }
warn() { printf '%s[%s] !!%s %s\n' "$_c_warn" "$(_ts)" "$_c_off" "$*" >&2; }
die() {
printf '%s[%s] ERR%s %s\n' "$_c_err" "$(_ts)" "$_c_off" "$*" >&2
exit 1
}
dry() { printf '%s[dry-run]%s %s\n' "$_c_dim" "$_c_off" "$*" >&2; }
is_dry() { [[ $DRY_RUN == 1 ]]; }
elapsed() { echo $(($(date +%s) - START_TS)); }
# run CMD...: execute, or print it under --dry-run.
run() {
if is_dry; then
dry "$(printf '%q ' "$@")"
else
"$@"
fi
}
# ---------------------------------------------------------------- config
_cleanup_files=()
_cleanup() { ((${#_cleanup_files[@]})) && rm -rf "${_cleanup_files[@]}"; return 0; }
trap _cleanup EXIT
tmpfile() {
local f
f=$(mktemp "${TMPDIR:-/tmp}/platform.XXXXXX")
_cleanup_files+=("$f")
echo "$f"
}
tmpdir() {
local d
d=$(mktemp -d "${TMPDIR:-/tmp}/platform.XXXXXX")
_cleanup_files+=("$d")
echo "$d"
}
need_cmd() { command -v "$1" >/dev/null 2>&1 || die "missing prerequisite: $1"; }
# kustomize_build DIR: standalone kustomize if installed, else kubectl's.
kustomize_build() {
if command -v kustomize >/dev/null 2>&1; then kustomize build "$1"; else kubectl kustomize "$1"; fi
}
# load_config: source platform.env, check every path/tool/credential, export
# what children need. Refuses to run if anything is missing.
load_config() {
local env_file=${PLATFORM_ENV:-$PLATFORM_ROOT/platform.env}
[[ -f $env_file ]] || die "config not found: $env_file (copy platform.env.example)"
# shellcheck source=/dev/null
source "$env_file"
local v
for v in FORGEJO_URL FORGEJO_OWNER FORGEJO_ADMIN_TOKEN_FILE REGISTRY_PUSH_TOKEN_FILE \
GITOPS_TOKEN_FILE GITOPS_DIR CODE_DIR KUBECONFIG INGRESS_DOMAIN INGRESS_IP; do
[[ -n ${!v:-} ]] || die "platform.env: $v is not set"
done
for v in FORGEJO_ADMIN_TOKEN_FILE REGISTRY_PUSH_TOKEN_FILE GITOPS_TOKEN_FILE KUBECONFIG; do
[[ -r ${!v} ]] || die "platform.env: $v=${!v} is not a readable file"
done
[[ -d $GITOPS_DIR/.git ]] || die "platform.env: GITOPS_DIR=$GITOPS_DIR is not a git clone"
[[ -d $CODE_DIR ]] || die "platform.env: CODE_DIR=$CODE_DIR does not exist"
for v in curl jq git kubectl; do need_cmd "$v"; done
export KUBECONFIG
FORGEJO_TOKEN=$(tr -d '\n' <"$FORGEJO_ADMIN_TOKEN_FILE")
REGISTRY_TOKEN=$(tr -d '\n' <"$REGISTRY_PUSH_TOKEN_FILE")
GITOPS_TOKEN=$(tr -d '\n' <"$GITOPS_TOKEN_FILE")
# Read by the inline git credential helper (git_auth) without hitting argv.
export PLATFORM_GIT_USER=$FORGEJO_OWNER PLATFORM_GIT_TOKEN=$FORGEJO_TOKEN
# Header files keep tokens out of `ps` output.
ADMIN_HDR=$(tmpfile)
REGISTRY_HDR=$(tmpfile)
chmod 600 "$ADMIN_HDR" "$REGISTRY_HDR"
printf 'Authorization: token %s\n' "$FORGEJO_TOKEN" >"$ADMIN_HDR"
printf 'Authorization: token %s\n' "$REGISTRY_TOKEN" >"$REGISTRY_HDR"
api GET /user
[[ $API_CODE == 200 ]] || die "Forgejo admin token rejected ($API_CODE) at $FORGEJO_URL"
kubectl get namespace argocd >/dev/null 2>&1 || die "kubectl cannot reach the cluster (KUBECONFIG=$KUBECONFIG)"
}
# ---------------------------------------------------------------- Forgejo API
# api METHOD PATH [JSON] [admin|registry]: sets API_CODE and API_BODY.
# Never fails on HTTP status; callers check API_CODE.
API_CODE='' API_BODY=''
api() {
local method=$1 path=$2 body=${3:-} as=${4:-admin} hdr out
[[ $as == registry ]] && hdr=$REGISTRY_HDR || hdr=$ADMIN_HDR
out=$(tmpfile)
local args=(-sS -o "$out" -w '%{http_code}' -X "$method" -H "@$hdr" -H 'Accept: application/json')
[[ -n $body ]] && args+=(-H 'Content-Type: application/json' --data-binary "$body")
API_CODE=$(curl "${args[@]}" "$FORGEJO_URL/api/v1$path") || die "curl failed: $method $path"
API_BODY=$(cat "$out")
}
# api_write METHOD PATH [JSON] [as] [redact]: a mutating call. Under --dry-run
# it is printed (with the body redacted if asked) and reports success.
api_write() {
local method=$1 path=$2 body=${3:-} as=${4:-admin} redact=${5:-0}
if is_dry; then
local shown=$body
[[ $redact == 1 ]] && shown='{"data":"<redacted>"}'
dry "$method $FORGEJO_URL/api/v1$path ${shown}"
API_CODE=200 API_BODY='{}'
return 0
fi
api "$method" "$path" "$body" "$as"
}
repo_exists() {
api GET "/repos/$FORGEJO_OWNER/$1"
[[ $API_CODE == 200 ]]
}
# ---------------------------------------------------------------- git
# git_auth ARGS...: git with the admin token as an inline credential helper
# (overrides the user's global helper; the token never lands in a remote URL).
git_auth() {
# shellcheck disable=SC2016 # expanded by the helper's shell, from the env
git -c credential.helper= \
-c 'credential.helper=!f() { echo "username=$PLATFORM_GIT_USER"; echo "password=$PLATFORM_GIT_TOKEN"; }; f' "$@"
}
# gitops_sync: GITOPS_DIR must be a clean checkout of main; fast-forward it.
gitops_sync() {
local branch
branch=$(git -C "$GITOPS_DIR" rev-parse --abbrev-ref HEAD)
[[ $branch == main ]] || die "gitops clone is on '$branch', expected main"
[[ -z $(git -C "$GITOPS_DIR" status --porcelain) ]] || die "gitops clone $GITOPS_DIR has uncommitted changes"
git_auth -C "$GITOPS_DIR" pull -q --ff-only origin main || die "gitops pull failed"
}
# gitops_commit_push MESSAGE PATH...: stage PATHs, commit, push; rebase and
# retry if CI's gitops-bump pushed in between.
gitops_commit_push() {
local msg=$1
shift
if is_dry; then
dry "git -C $GITOPS_DIR add -A $* && git commit -m '$msg' && git push origin main"
return 0
fi
git -C "$GITOPS_DIR" add -A -- "$@"
if git -C "$GITOPS_DIR" diff --cached --quiet; then
ok "gitops: nothing to commit for: $msg"
return 0
fi
git -C "$GITOPS_DIR" commit -q -m "$msg"
local i
for i in 1 2 3 4 5; do
if git_auth -C "$GITOPS_DIR" push -q origin main; then
ok "gitops: pushed '$msg' ($(git -C "$GITOPS_DIR" rev-parse --short HEAD))"
return 0
fi
warn "gitops push rejected (attempt $i); rebasing on origin/main"
git_auth -C "$GITOPS_DIR" pull -q --rebase origin main
done
die "gitops push failed after 5 attempts"
}
# app_namespace NAME: destination namespace from gitops apps/NAME.yaml (empty if absent).
app_namespace() {
local f=$GITOPS_DIR/apps/$1.yaml
[[ -f $f ]] || return 0
awk '/^ destination:/{d=1; next} d && /^ namespace:/{print $2; exit}' "$f"
}
# ---------------------------------------------------------------- cluster
argocd_refresh() {
run kubectl annotate application -n argocd "$1" argocd.argoproj.io/refresh=normal --overwrite >/dev/null
}
# wait_until TIMEOUT_S INTERVAL_S DESCRIPTION CMD...: poll CMD until it succeeds.
wait_until() {
local timeout=$1 interval=$2 desc=$3
shift 3
local deadline=$(($(date +%s) + timeout))
log "waiting for $desc (timeout ${timeout}s)"
until "$@"; do
(($(date +%s) < deadline)) || die "timed out after ${timeout}s waiting for $desc"
sleep "$interval"
done
ok "$desc"
}
validate_name() {
[[ $1 =~ ^[a-z][a-z0-9-]{1,30}$ ]] || die "invalid name '$1': must match ^[a-z][a-z0-9-]{1,30}\$"
[[ $1 != *- ]] || die "invalid name '$1': must not end with '-'"
case $1 in
argocd | default | logging | unleash | platform | gitops | workflows | demo-app | \
kube-* | cnpg-system | elastic-system | service-template-*)
die "name '$1' is reserved"
;;
esac
}
# Print the URLs a human wants after a deploy.
print_urls() {
local name=$1 kind=$2
local host="$name.$INGRESS_DOMAIN"
local kql
kql=$(jq -rn --arg s "$name" '"service:\"\($s)\"" | @uri')
cat <<EOF
service : http://$host/
health : http://$host/$([[ $kind == backend ]] && echo readyz || echo healthz)
repo : $FORGEJO_URL/$FORGEJO_OWNER/$name
argocd : http://argocd.$INGRESS_DOMAIN/applications/argocd/$name
kibana : http://kibana.$INGRESS_DOMAIN/app/discover#/?_g=(time:(from:now-1h,to:now))&_a=(query:(language:kuery,query:'$kql'))
unleash : http://unleash.$INGRESS_DOMAIN/projects/default
EOF
}

16
platform.env.example Normal file
View file

@ -0,0 +1,16 @@
# Copy to platform.env (gitignored) and adjust. Sourced by bin/* as bash.
# Every file below must exist or the scripts refuse to run.
HOMELAB_SECRETS=$HOME/Code/github.com/nholtware/homelab/.secrets
FORGEJO_URL=http://192.168.1.230:3000
FORGEJO_OWNER=nholtware
FORGEJO_ADMIN_TOKEN_FILE=$HOMELAB_SECRETS/forgejo-admin-token # repo create/delete, Actions secrets, git push
REGISTRY_PUSH_TOKEN_FILE=$HOMELAB_SECRETS/registry-push-token # becomes REGISTRY_TOKEN; also lists/deletes packages
GITOPS_TOKEN_FILE=$HOMELAB_SECRETS/ci-bot-gitops-token # becomes GITOPS_TOKEN (ci-bot)
GITOPS_DIR=$HOME/Code/forgejo.local/nholtware/gitops # clean clone of nholtware/gitops on main
CODE_DIR=$HOME/Code/forgejo.local/nholtware # new service repos are cloned here
KUBECONFIG=$HOME/.kube/homelab.yaml
INGRESS_DOMAIN=192.168.1.83.nip.io # services at <name>.$INGRESS_DOMAIN
INGRESS_IP=192.168.1.83 # health checks use curl --resolve, no DNS needed