AINode Release Notes

Fixed

A decision is answered by the node that owns the model, wherever the request enters (#285). /v1/decide and /v1/systemone for a model the receiving node does not serve (the master, usually) are forwarded whole to the serving node's AINode over the fleet key, so the answer carries that node's temperatures and warm grammar instead of calibration: {applied: false, temperatures: null}. Replicas are tried in turn when one is down; a forwarded request is never forwarded again and does not count against the owner's rate limit; with no owner answering, the node calls the engines itself as before. A locally served model is unchanged.

When the receiving node does have to call the engines itself, it tempers with the serving node's table (#284). A node with no copy of the model fetches the table from the node it routes to over the fleet key (GET /api/decide/calibration?model=<id>, new) and caches it for five minutes.

A slow resolver can no longer freeze a node (#284). /api/server/status looked up this host's name on the event loop, so a node whose hostname is not in /etc/hosts, with a nameserver that does not answer, stopped serving every route for 20 s per poll. On 2026-09-26 that made the master's peers go stale, and it stopped routing to them. The lookup now runs in a thread, is cached for five minutes, and a request waits at most one second for it.

A decision model kept across a restart on a stacked port warms its answer grammar again (#284). Adoption skipped the #277 warm-up, so /api/status reported warm: null for it and the first real question paid the compile.

Image: ghcr.io/getainode/ainode:0.5.33

Decisions use the model's own calibration

/v1/decide and /v1/systemone apply a decision adapter's fitted temperatures and say so in every answer, a decision model warms its answer grammar the moment it binds, the decision bench scores any model on the public Jevals sets with the boards' own formulas, and a master that serves no model is safe to run, which is the first step of moving the fleet's control plane to Atlas.

Added

/v1/decide and /v1/systemone apply a decision adapter's own temperatures (#276, #281). When the served model's directory carries temperatures.json, each question's label logprobs are divided by the temperature fitted for its kind (choice, noul, score) before the softmax. Both responses report calibration: {applied, temperatures}, and "calibration": "raw" returns the engine's own spread.

A decision model warms its answer grammar as soon as its engine binds (#277, #281). One small constrained question per kind is sent in the background, with each compile time logged. /api/status gains instances[] with warm and warm_compile_seconds per instance.

The decision bench scores models on the public question sets the independent Jevals boards use, with the boards' own formulas (#272), so an AINode-served model can be compared with Jev and its clones on one recipe. ainode-bench decide --suite all --transport decide|systemone runs PubMedQA (noul, 300), Banking77 (choice, 300, K=77), HelpSteer2 helpfulness (score, 300) and LocalLLaMA/typed-decisions (mixed, 2000 decisions over 400 cases) at 5 repeats per question, against either AINode's own POST /v1/decide or any endpoint speaking the Jev POST /v1/systemone wire format. Per set it reports the Decision Score against the label prior (100 perfect, 0 the base rates, negative worse), accuracy with its guessing floor, the calibration gap in points over ten bins with the reliability table behind it, the hand-off share at 95 percent, the published gate with this run's coverage, pick flips and confidence swing across the repeats, p50 and p95 latency, questions per second, malformed answers, and cost when a price is given. --questions <file> scores any question file in the same shape, so a private blind set never has to be committed. The recipe, the date each source page was read and every deviation from it are in bench/decide/JEVALS.md; the sets are bench/decide/sets/ and ainode-bench decide download fetches the item text, which is not committed, and verifies every state against its published hash.

Changed

A decision request that meets a cold grammar compile gets a 503 saying so, and has longer to finish (#281). The engine-call limit is 300 s, up from 180, and a call that times out after connecting reads "the engine is compiling the answer grammar, retry" instead of "unreachable".

ainode doctor passes on a node that loads no model (#282). "No model loaded" is INFO, and docker and engine-backend findings are INFO on such a node, so a routing-only node whose GPU belongs to something else no longer fails. A node that pins a model still fails exactly as before.

Fixed

A request with no model on a node that serves no model of its own gets a 400 missing_model_field naming the field (#282), instead of being routed to a model the node does not serve. Nodes that serve a primary keep their default.

Unloading or ejecting a node's primary no longer promotes another instance (#282), for example Whisper, into config.model with the old primary's engine parameters. The primary slot and its overrides are cleared, and stacked instances stay on their own ports.

An empty account list is never replicated (#282). The master's push, a worker's pull, the CLI push and /api/auth/users/sync (409 empty_account_list) all refuse it and log how to fix the store, so a master with an empty users.json can no longer sign every dashboard user out of the fleet.

Image: ghcr.io/getainode/ainode:0.5.32

Engine activity as the liveness signal, ranks kept in step through autotune, OpenCode in the harness bench, catalog shapes in every record.

Added

ainode-bench harness --claude-effort <level> sets the reasoning effort Claude Code runs with (#127). Claude Code sends effort "high" by default, and a served chat template does not have to accept that value: Qwen3.8-Flash-Next takes only xhigh, medium and low, so every request came back API Error: 400 Unexpected reasoning effort high, the harness crashed in about 0.3 s per attempt and the model scored 0/10 on a suite the other three harnesses were passing. The same ten tasks then scored 8/10 and 10/10 at medium. The level is appended to the claude argv as --effort <level> and nothing else about the invocation moves; unset stays the default and sends nothing, so every number already recorded (Ornith, Qwen3.8 27B) keeps its meaning. It is written down where a reader will find it: the run's settings as claude_effort and the claude block's options as effort, both absent when the flag was not used, because a run with the agent's own default is a different statement from a null. Docs: the claude section of bench/harness/README.md.

Fixed

The harness bench record's placement.node now names the node that serves the model, not the master the CLI talked to. ainode-bench harness built the placement block by calling describe_via_http with the --ainode web base as both the control-plane and engine URL, so when DeepSeek V4 Flash was served on Spark-2 while the CLI pointed at Spark-1, the record's placement.node read "Spark-1". The CLI now resolves the serving node through the master's /api/server/status loaded_models (the same fleet view the speed bench and the in-product bench use), calls describe_via_http against that node's own web base, and records the engine port in placement. A model served on the master is unchanged; an unloaded model produces the same "does not report serving" warning.

The harness bench's OpenCode adapter now runs in the task's directory instead of the one the bench was started from, which is the whole of #118. Every --harness opencode invocation from the runner exited 1 in about 1 s with {"type":"error",...,"message":"Unexpected server error. Check server logs for details."} on stdout and an empty log, while the identical command run by hand passed, so the adapter was carried as unverified and the DeepSeek harness record's OpenCode rows are marked not measured. --print-logs --log-level DEBUG off a runner-shaped invocation named it in one run: opencode creates its instance in the working directory, reads the run's project-local opencode.json there, and then creates a second instance and builds the session against the directory named by PWD - which a subprocess started with cwd= still inherits from its parent, so it was the shell the bench was launched in. That instance has no ainode-bench provider, so the session died with ProviderModelNotFoundError: Model not found: ainode-bench/<model id>, reported on stdout as the generic server error. By hand it passed because a person cds in first and PWD is then correct; that asymmetry, not flakiness, is why it looked intermittent, and it is why the other four harnesses were never affected (none of them reads PWD to find its project). Two fixes, one cause: the argv now states --dir <workdir>, and _launch sets PWD to the launch cwd and drops a stale OLDPWD for every harness, since the wrong value also reaches whatever shell an agent spawns for its own tools. Verified by the runner itself, --only-tasks isogram against Ornith 1.5 35B A3B NVFP4 on the fleet: three consecutive runs, pass@1 1/1 at 25.5 s, 26.7 s and 26.5 s, zero crashes, where the same command on the previous commit crashed in 0.9 s twice out of two.

An OpenCode bench run no longer reads the operator's global config, and a failed one now says what failed. OPENCODE_CONFIG_DIR overrides the XDG config search outright, so the four isolated XDG_* variables the adapter already set were bypassed by any exported value (Orca exports one) and somebody's global config, plugins and agents were inside the measurement; the overlay now points that variable at the run's own empty config directory too, the same rule DSH_HOME and CLAUDE_CONFIG_DIR follow. The argv also carries --print-logs --log-level ERROR, which is diagnostics rather than behaviour - the logs go to stderr, stdout stays the NDJSON event stream the adapter parses - so a future failure reaches the record's stderr_tail with a named cause instead of the opaque "Unexpected server error" that #118 had to work from; verified against the old broken shape, where it prints the ProviderModelNotFoundError in 1388 bytes.

The bind wait watches what the engine is DOING; log silence is only the second opinion (#112). vLLM prints nothing at all while it loads weight shards, runs torch.compile and autotunes FlashInfer, so "no log line for N seconds" could never tell a wedged engine from a working one: three healthy starts in two days were declared silent (Qwen3.8 27B NVFP4 quiet for 206 s between shard updates, Nemotron 3.5 Lightning for 363 s inside autotune and graph capture, then a 48-minute autotune), and the window had to grow 120 to 360 to 900 s to keep up. Since 0.5.13 that verdict broke nothing (the relaunch is a no-op while the container is alive) but it spent the engine's single retry and wrote a wrong reason into the log. Backends now publish activity_mark(): for NvidiaBackend the last time THAT engine's container was seen burning CPU, read from the container's cgroup cpu.stat when the tree is visible and otherwise from docker stats --no-stream, cached so one bind poll costs one probe and run off the event loop thread. A log line OR an advancing activity mark resets the silence clock, and the start is called dead only when both have been quiet for the whole budget, which is why the budget comes back down to 300 s (engine_bind_log_silence_seconds): a truly wedged engine does no work at all and is caught in five minutes again instead of fifteen. The hard death signal (container exited) and the 1800 s ceiling are unchanged, and a backend that cannot see its engine's container reports None and keeps the old log-driven behaviour. GPU utilization is deliberately not part of this: the pynvml path reads device 0 as a whole, so it cannot say which stacked engine is busy, and it reads 0% during exactly the phases that need covering.

engine_ready in /api/nodes means the engine answers /v1/models, not that the node is reachable (the second half of #112). The field was derived from status, which is heartbeat health (online/stale/offline) and true of every node whose orchestrator is up, because ClusterNode.from_discovered replaces the announced engine state with that health: Spark-4 was published engine_ready: true while its engine was still loading and its loaded_models was empty. ClusterNode now carries the node's own word on its engine (engine_status, straight from the announcement, which that node's sync loop already sets from a live /v1/models probe), the local row answers from a fresh localhost probe rather than the ready latch, and a member node (which runs no local vLLM) still reads ready once discovered. status is untouched, so the topology's online count keeps working.

The two ranks of an mp launch stay in step through FlashInfer autotune, instead of gloo killing the pair 48 minutes in (#134). Kernel warmup runs per rank. On Qwen3.8-Flash-Next the head, warm from an earlier launch, finished autotune in tens of minutes while the peer tuned 856 profiles from cold, sat 1800 s on its first trtllm::fused_moe::gemm1 profile, and the barrier after autotune died with Connection closed by peer on one side and Application timeout caused pair closure on the other. 1800 s is not a kernel hang: it is PyTorch's default timeout for the CPU (gloo) group, which is the group the barrier and the autotune sync both use. Three things change. The head now ships its compiled-kernel cache to a peer before that peer's container starts (_ensure_peer_has_jit_cache): one transfer per child of the recipe's VLLM_CACHE_ROOT, skipped per child when the peer already has it, and a transfer that fails is a warning rather than a refused launch, since a missing cache costs minutes and a refused launch costs the model. The Flash-Next entry turns autotune off with --no-enable-flashinfer-autotune (the off form of KernelConfig.enable_flashinfer_autotune, which kernel_warmup.py reads to skip the whole phase) and raises both collective floors to 5400 s, above the worst warmup measured on the pair; the stated cost is that its fused MoE and fp4 GEMMs now run FlashInfer's heuristic tactic rather than a measured one, so decode is slower than the 26.6 tok/s of the launch that got through. It also states the cache roots it was missing, so its JIT output persists per node inside the mounted HF cache instead of dying with the container and re-tuning from cold every launch. The bind wait's ceiling goes from 1800 s to 3600 s, because a launch that took 36 minutes to ready was being killed by the ceiling as well; the ceiling is the last resort and the activity signal from #112 still calls a genuinely wedged engine dead inside five minutes, so nothing waits an hour on a dead start. Worth writing down for the next person: seeding a peer's autotune file cannot help on its own. Only rank 0 reads that file and broadcasts its bytes to the other ranks, and the path holds a sha256 of the whole VllmConfig hash, so a hand-copied directory is both unread and easy to orphan. DeepSeek V4 Flash keeps its own autotune and timeouts exactly as proven.

Catalog entries state their active parameter count and architecture, so a bench record stops reporting a MoE as dense. ModelInfo had no field for either, and the record's model block was filled by reading the A<n>B marker off the model id, which works for Ornith-1.5-35B-A3B and not at all for an id that does not carry one: DeepSeek V4 Flash (284B total, 13B active) and Qwen3.8-Flash-Next (125B, 6B) were both written down as params_b 125/284, active_b 125/284, arch dense, which on GB10 is the one number that predicts decode speed, said wrong. ModelInfo now carries active_params_b and arch ("moe" / "dense"), every curated and fallback entry states its shape, and the model block takes both from the catalog when it has them and falls back to the id only for a model the catalog does not describe. An entry that says MoE without an active count (GLM-5.2 REAP, where pruning moves it) leaves active_b out and both renderers print 504B MoE rather than claiming dense. The catalog cache round-trips the new fields and a cache file written before they existed still loads. Applies to the throughput bench, the in-product /api/bench runs and the harness bench alike: all three build the block through ainode/bench/fleet.py.

Image: ghcr.io/getainode/ainode:0.5.20

The fleet endpoint forwards the Anthropic Messages API; Claude Code is a harness bench adapter; Qwen3.8-Flash-Next entry pinned and proven.

Fixed

The fleet endpoint forwards the Anthropic Messages API. vLLM serves POST /v1/messages natively alongside its OpenAI paths, but AINode's port 3000 registered only /v1/chat/completions and /v1/completions, so /v1/messages was a 404 on every node: a client that speaks Messages and nothing else (Claude Code, the Anthropic SDKs) had to be aimed at one engine's :8000 and lost the model-id lookup, capability-aware ordering and failover that every other protocol gets for free. /v1/messages and /v1/messages/count_tokens now go through proxy_to_vllm, the same handler as chat completions, so routing on the body's model, transport failover across every node serving it, the 404 for a model nobody serves, SSE streaming and header passthrough (x-api-key, anthropic-version, anthropic-beta reach the engine untouched) are the code that already existed rather than a second implementation. is_multimodal_request now also reads the Messages API's own spelling of media (an {"type": "image"} or {"type": "document"} block, including one nested in a tool_result), so an image sent over /v1/messages is ordered onto an instance that accepts images instead of being routed on the model id alone. The proxy also stops dropping the caller's query string (path_qs, not path): Claude Code posts to /v1/messages?beta=true. The Server view's endpoint list no longer calls /v1/messages planned.

Added

Harness bench adapter: claude, Claude Code driven against a served model. The first harness here that does not speak the OpenAI protocol, and the reason the proxy fix above exists. --harness claude runs claude -p "<prompt>" --model <id> --dangerously-skip-permissions --output-format json --max-turns 12 with ANTHROPIC_BASE_URL set to --endpoint minus its trailing /v1 (Claude Code appends /v1/messages itself, so the /v1 form would produce /v1/v1/messages); --endpoint stays the /v1 form on the command line for every harness. The placeholder key goes in both ANTHROPIC_API_KEY and ANTHROPIC_AUTH_TOKEN, the served model fills both the main and small-fast model slots, and telemetry and non-essential traffic are off. CLAUDE_CONFIG_DIR points at the run's own scratch directory so a bench run never reads or writes the operator's Claude Code profile (settings, hooks, MCP servers, sessions, credentials), which is both hygiene and honesty, since a personal settings.json changes what the agent does; it needs no seed file (verified: claude 2.1.272 creates the whole tree itself from a directory that never existed and never prompts for a login). is_error: true in the result JSON is recorded as a crash even when the exit code is 0, because Claude Code reports a run it could not finish and still exits cleanly. Verified end to end on claude 2.1.272: 6/6 hidden tests on isogram driving Qwen3.8 27B NVFP4 on the fleet, 294 s over 6 turns. Known gap, documented rather than fixed: a CLAUDE.md in a parent of the working directory is still discovered, so a --work-dir inside a checkout that has one puts that file in the run.

Image: ghcr.io/getainode/ainode:0.5.19

Fixed

The distributed launch serves a model downloaded through AINode; harness bench; Qwen3.8-Flash-Next entry.

A distributed launch now serves a model that was downloaded through AINode instead of making every node re-download it. POST /api/models/download-repo writes a flat <models_dir>/<owner--name> directory, not the Hugging Face cache layout, and the solo launch has served that directory for releases. The distributed shapes did not: both served the repo id, the head and peer containers mounted only the HF cache, and _ensure_peer_has_model distributed only a hub/models--<owner>--<name> entry, so a 133 GB model already sitting on the head was pulled again on every rank, over the WAN, for every launch. Both shapes now go through one resolver: when that flat directory exists on the head and the mount is trustworthy (the same AINODE_HOST_HOME test the solo path uses), every rank serves /ainode-models/<owner--name> with --served-model-name <repo id> so /v1/models is unchanged, and every rank mounts its own copy of the store there: the head's models_dir, a peer's /home/<ssh_user>/ainode-nvidia-models (chosen like the peer HF cache, because we ssh in as that user). _ensure_peer_has_model ships the flat directory over the fabric (rsync when present, else tar over ssh, skipped when the peer already has it) instead of the hub entry, and the peer's mkdir -p covers the new path so docker can never invent a root-owned empty bind source. With no such local copy, nothing changes: the repo id is served and only the HF cache is mounted. A recipe that states --served-model-name itself still wins, in solo too.

Added

Catalog entry: Qwen3.8-Flash-Next (NVFP4), nvidia/Qwen3.8-Flash-Next-NVFP4, as a two-node mp launch. Frontier MoE, 125B total and 6B active per token, plus a 51B PLE n-gram embedding and a 4B MTP module; 133 GB of mixed-precision weights (NVFP4 routed experts, FP8 elsewhere, modelopt), so it does not fit one 121 GB GB10 node. The entry pins vllm/vllm-openai:v0.29.0: the architecture and the FP8-PLE loader for mixed ModelOpt checkpoints landed in 0.29, and the fleet's 0.27.1 default and the local 0.28.0 images do not know it. Carries the card's recipe minus what the backend emits itself (--quantization modelopt, --enable-prefix-caching, --reasoning-parser qwen3, --tool-call-parser qwen3_coder, --enable-auto-tool-choice), fp8 KV cache, 262K context, --trust-remote-code, gmu 0.85. MTP speculative decoding is deliberately not enabled: it wants --enable-expert-parallel, which hangs on this MoE and hardware, so it is a follow-up. verified=false and recommended=false until it is served end to end on the pair. It is the strongest coding model in the Qwen3.8 line, ahead of the 27B and DeepSeek V4 Flash on three of the four coding rows in Qwen's own table.

A harness bench: scripts/ainode-bench.py harness measures whether a served model can drive a coding agent to passing tests, as opposed to how fast it generates. Ten Python practice exercises vendored from exercism/python (MIT, the same source Aider's polyglot benchmark draws on) are handed to a real agent CLI, and the score is what pytest says afterwards: pass@1, pass@2, mean wall seconds and crash count per harness. Adapters ship for aider, dsh, pi and opencode; every one is pointed at an AINode endpoint (http://<node>:3000/v1) and needs no API key beyond a placeholder. The hidden tests are copied into the working directory only after the harness has exited and removed again before the next attempt, so a harness can never read the assertions it is being judged on, and the second attempt is given the real test output the way Aider's protocol does it. --dry-run prints the exact argv, environment and config for every task and harness without running anything, and is the only part that works in a sandboxed shell: the agents themselves need normal network and home-directory access, and a sandboxed run hangs rather than failing loudly. dsh gets its own DSH_HOME because it validates every configured provider route at boot, so one stale entry in a person's ~/.dsh/settings.yaml would end every run with a transport error whatever provider the run selected. Records are schema 1 with a new top-level harness block and no results block, so scripts/render-bench-table.py skips them rather than rendering a coding run as a very slow model. Docs: bench/harness/README.md.

Image: ghcr.io/getainode/ainode:0.5.18