AI Foundry lab

AI Foundry lab, 14 October 2026: the dashboard's pages checked, and a todo: list

← AI Foundry lab dashboard › Worklog

Covers Wed 14 Oct 2026 (week 42): the lab dashboard, every page it links, the three machines aifoundry1, aifoundry2 and aifoundry3, and what watches them. Written Wed 14 Oct 2026, about 16:00 PDT, from the live pages and their histories, the repository, and read-only checks on the three machines. No card was opened, and nothing that needs root was done.

The new-user page offers aifoundry2 again, and the dashboard has a todo: list. The page's Machine field had left aifoundry2 out since 2 October, through both of the card's returns to service. Eighteen pages the dashboard links are now up to date; most of them still described a card as it was a week or two ago. The checks also found the wired network down overnight with the dashboard reading "All clear", and aifoundry1's card-use log four days from its size cap. Those, and everything else still open, are on the new list.

The new-user page

New user? Start here has a Machine field that fills in the commands. It was added on 2 October, while aifoundry2's card was out of service, and offered only aifoundry3 and aifoundry1. The card came back on 6 October and again on 9 October, and both updates changed the brief, the dashboard and the docs, but not that field, so for five days a new user could not choose aifoundry2. It now offers aifoundry3, aifoundry2 and aifoundry1, in the brief's order. aifoundry2 is ready for newcomers: its tmux is found by a plain ssh command, and the page's restart-proof setup was tested there on 13 October.

Two things keep it from happening again:

On the same page, the footer link, broken since 9 October, opens the session timeline again, and "If a step fails" says how to stop the Remote Control server as well as the restarts.

The dashboard

Under the Worklog there is now a todo: list: 41 things we want to do in the lab, oldest first, from 25 September to today, each with the day it was added and who added it. It gathers what the lab problems ledger, the review's list, the 11 October sweep, the fan swap and today's checks left open. The longer ledgers stay where they are, and it links to them.

The Worklog gained Tuesday 13 and Wednesday 14 October: the new-user steps that survive a restart, the chip diagram's Up bar and readout, the 3D views, and this page. Its intro no longer claims that every page it lists links back here. A standing note that folded away card 0's PCIe errors is gone: aifoundry1's card 0 logged about 4,000 corrected errors an hour in late September and 44 in all since the 7 October restart, so a return will now show as an alert.

Pages brought up to date

Each change is a dated line in the page's own style; nothing dated was rewritten.

What changed Pages
aifoundry2's card is in service since 9 October the reports hub (four places, three of them work that waited on the card), the 2 October and 6 October pages, the energy tutorial, the review's list, Requests for Roman (H28, SH5)
aifoundry1's card 0 is in service since 2 October the reports hub, the energy manual, heat placement, the DVFS loop, the Horace experiment, running without a heatsink (its floor table too), and the two 25 September pages on aifoundry1 (what was broken, the fix)
the wired network (below) the 7 October and 11 October pages, Requests for Roman (H11, U29, SH4)
aifoundry3's wired address and monitor fix went on on 13 October; only its reboot waits for its user the 11 October page's summary and sections 5 and 6
the line back to the dashboard and its Worklog added to the 2 October page, the idle runaway of 7 October, the two aifoundry1 pages and the link-retrain hang; completed on the 6 and 7 October pages and the energy tutorial
Requests for Roman's state, from 8 to 14 October C15 (card 0's link errors), C30, H1 and H37 re-checked
the repository's own record the energy tutorial and its three scripts filed, with a row in the mirror index, as are the agents page and the heat-aware compiler page; the 9 October page's step 7, added on the live page today, folded in; aifoundry2's banner no longer sends newcomers elsewhere; the lab tools' notes say the daily health check is installed (it is, on all three); the history page says every 15 minutes, as it has been since 8 October; and its publisher refuses an unknown argument instead of publishing

Found, and on the todo: list

Not done, and why

Sources