AHL Pod Down After Job Crash | Generated by AI

Home 2026.07

The pod has gone down and hasn’t come back up. Here’s the situation:

🚨 Server is Down

SSH port 31005 — Connection refused Tunnel URL — 404 Not Found

🔍 What Likely Happened

The AHL video generation job was submitted and went to processing state, but then errored out (went to “Not found”). This likely crashed the AHL server or caused an OOM kill, since the AHL process was using ~35GB of RAM. The pod then restarted or was terminated.

🛠️ Possible Causes

  1. OOM (Out of Memory) — The video generation model (FLUX) is very memory-hungry, and the pod may have been killed when it exceeded limits
  2. Pod restart — If the Jupyter process (PID 1) exited, the pod restarts
  3. Job backend crash — The gen-video processing encountered an error and crashed

Next Steps

Try these once the pod comes back up:

  1. Check if the pod is still alive from the platform console

  2. If recreated, re-run the tunnel setup:

    ssh -p 31005 root@36.150.116.206
export FRP_BROKER_URL=$(grep -z FRP_BROKER_URL /proc/1/environ | tr "\0" "\n" | cut -d= -f2-)
export FRP_BROKER_TLS_SERVER_NAME=$(grep -z FRP_BROKER_TLS_SERVER_NAME /proc/1/environ | tr "\0" "\n" | cut -d= -f2-)
export PATH="$HOME/.local/bin:$PATH"
/var/run/secrets/frp-self-service/install
rc-tunnel expose --port 8000

  3. Check AHL server logs:

    curl http://127.0.0.1:8000/health  # if the server is up

Would you like me to keep trying to reconnect periodically?

Generated by AI. Curating and sharing still takes effort. If you find it useful, feel free to donate. WeChat: @lzwjavaWeChat QR · X: @lzwjava · Say hi 👋

Back Donate