← AI Foundry lab dashboard › Worklog
Covers Wed 14 Oct 2026 (week 42): the lab dashboard, every page it links, the three machines aifoundry1, aifoundry2 and aifoundry3, and what watches them. Written Wed 14 Oct 2026, about 16:00 PDT, from the live pages and their histories, the repository, and read-only checks on the three machines. No card was opened, and nothing that needs root was done.
The new-user page offers aifoundry2 again, and the dashboard has a todo: list. The page's Machine field had left aifoundry2 out since 2 October, through both of the card's returns to service. Eighteen pages the dashboard links are now up to date; most of them still described a card as it was a week or two ago. The checks also found the wired network down overnight with the dashboard reading "All clear", and aifoundry1's card-use log four days from its size cap. Those, and everything else still open, are on the new list.
The new-user page
New user? Start here has a Machine field that fills in the commands. It was added on 2 October, while aifoundry2's card was out of service, and offered only aifoundry3 and aifoundry1. The card came back on 6 October and again on 9 October, and both updates changed the brief, the dashboard and the docs, but not that field, so for five days a new user could not choose aifoundry2. It now offers aifoundry3, aifoundry2 and aifoundry1, in the brief's order. aifoundry2 is ready for newcomers: its tmux is found by a plain ssh command, and the page's restart-proof setup was tested there on 13 October.
Two things keep it from happening again:
- The note. The checklist for a card going out of service or coming back now lists the Machine field and the login banners' "New here?" lines with everything else to change in one commit (docs/lab-start/README.md, Keeping it current; AGENT.md §4 points to it).
- A check. The page's rebuild now refuses to run while the Machine field and the brief's per-machine list disagree, in either direction: tested against the old page (it asks to add aifoundry2) and against the brief of 7 October, when the card was out (it asks to remove it).
On the same page, the footer link, broken since 9 October, opens the session timeline again, and "If a step fails" says how to stop the Remote Control server as well as the restarts.
The dashboard
Under the Worklog there is now a todo: list: 41 things we want to do in the lab, oldest first, from 25 September to today, each with the day it was added and who added it. It gathers what the lab problems ledger, the review's list, the 11 October sweep, the fan swap and today's checks left open. The longer ledgers stay where they are, and it links to them.
The Worklog gained Tuesday 13 and Wednesday 14 October: the new-user steps that survive a restart, the chip diagram's Up bar and readout, the 3D views, and this page. Its intro no longer claims that every page it lists links back here. A standing note that folded away card 0's PCIe errors is gone: aifoundry1's card 0 logged about 4,000 corrected errors an hour in late September and 44 in all since the 7 October restart, so a return will now show as an alert.
Pages brought up to date
Each change is a dated line in the page's own style; nothing dated was rewritten.
| What changed | Pages |
|---|---|
| aifoundry2's card is in service since 9 October | the reports hub (four places, three of them work that waited on the card), the 2 October and 6 October pages, the energy tutorial, the review's list, Requests for Roman (H28, SH5) |
| aifoundry1's card 0 is in service since 2 October | the reports hub, the energy manual, heat placement, the DVFS loop, the Horace experiment, running without a heatsink (its floor table too), and the two 25 September pages on aifoundry1 (what was broken, the fix) |
| the wired network (below) | the 7 October and 11 October pages, Requests for Roman (H11, U29, SH4) |
| aifoundry3's wired address and monitor fix went on on 13 October; only its reboot waits for its user | the 11 October page's summary and sections 5 and 6 |
| the line back to the dashboard and its Worklog | added to the 2 October page, the idle runaway of 7 October, the two aifoundry1 pages and the link-retrain hang; completed on the 6 and 7 October pages and the energy tutorial |
| Requests for Roman's state, from 8 to 14 October | C15 (card 0's link errors), C30, H1 and H37 re-checked |
| the repository's own record | the energy tutorial and its three scripts filed, with a row in the mirror index, as are the agents page and the heat-aware compiler page; the 9 October page's step 7, added on the live page today, folded in; aifoundry2's banner no longer sends newcomers elsewhere; the lab tools' notes say the daily health check is installed (it is, on all three); the history page says every 15 minutes, as it has been since 8 October; and its publisher refuses an unknown argument instead of publishing |
Found, and on the todo: list
- The wired network dropped overnight. All three wired ports lost their link at about 22:35 on 13 October and got it back at 13:56 on 14 October, within seven seconds of each other, which points at the switch or its power. Each port has lost its link twice since its machine last started. The dashboard read "All clear" throughout: its health check reports the link only as information, which the page does not show. The cable and its switch need a look on site.
- aifoundry1's card-use log reaches its 500 MB cap about Saturday 18 October, growing about 30 MB a day, and from then on deletes its oldest days without saying so. The fix written on 11 October needs its review's blocking issue solved (a joined minute counted as held time) and a second review.
- The card-use logger is at its 64 MB memory limit on aifoundry2 and aifoundry3; an out-of-memory stop would end card-use logging there.
- aifoundry1's login banner still tells people not to use card 0, in service since 2 October. The corrected banners, the root-login notice, the card-open tracer and the current
et-lab-startare in the repository and need root on the machines. - The history page has no keeper, standby, outside watcher or visibility check, unlike the dashboard.
- Which firmware two cards run. The new-user brief says aifoundry3's card and aifoundry1's card 1 run 1.4.0 since 7 October; the dashboard and the docs say 1.3.1 and 1.2.0. It needs a reading on the cards before the numbers change.
Not done, and why
- Nothing that needs root, a reboot or a visit: the banners and
et-lab-starton the machines, the kernel reboots waiting on aifoundry1 (since 9 October) and aifoundry3 (since 10 October, for its user), and the cable. - No card was opened, so the firmware question waits for a reading taken under the card etiquette.
Sources
- The live pages and their version histories, and the repository, et-soc1-prototyping (request Q114 in 02-requests.md).
- Read-only checks on aifoundry1, aifoundry2 and aifoundry3: each machine's network state and its counters, systemd, the logs readable without root, and the installed tools and banners against the repository's copies.
- The dashboard's own data. Written by agents in one session: seven checkers, each re-checked by a second agent, then five fixers, each reviewed by another before anything was published.