Living with it · Chapter 29 · Time: 2 hours, plus about an hour for a backup · Level: Beginner · Status: Partly test-built
The routine - keep the robot's copies of the repo current, read the whole system back in one command, deploy one part at a time, find the logs, back up after a good change, and get the robot back when its Wi-Fi drops.
The robot never pulls code from GitHub. It runs copies of files from the repo on your laptop, put there by a few
scripts in scripts/ops/ that you run from the Mac. Day-to-day work is those scripts: read everything back, deploy one
part, set the microphone array's levels, choose where Buddy hears, switch the display mode. They are written so that a
small model, or you on a tired evening, can run them without improvising: docs/runbook.md says which command does
what and what the output must look like.
Every ops script below was run on this robot on 2026-10-06 and 2026-10-07; the outputs are real. Two things in this
chapter were not done during the rebuild: the one-command copy into ~/setup (only its dry run was done) and the two
project agents (written, never used in a session).
| Repo (on the Mac) | On the robot | How it gets there |
|---|---|---|
scripts/*.sh, scripts/*.py, scripts/setup/*.sh, systemd/* |
~/setup/ |
rsync, next section |
scripts/tests/ |
~/setup/tests/ |
rsync |
scripts/make_golden.sh |
~/golden/ |
rsync |
vision/mind.py, world.py, reappear.py, look_learner.py |
~/vision/ |
deploy.sh mind |
vision/think.py, attention.py, scan.py, mind_pause.py |
~/vision/ |
not covered by deploy.sh: scp |
buddy_link/robot_api.py |
~/buddy_link/ |
deploy.sh api |
buddy_link/light_hori.py |
~/buddy_link/ |
scp (chapter 30) |
ros2/rosorin_base/rosorin_base/*.py, config/*.yaml, launch/*.py |
~/ros2_ws/src/rosorin_base/ |
deploy.sh driver or deploy.sh nav |
ros2/rosorin_base/bt/, urdf/, config/robot_model.json |
~/ros2_ws/src/rosorin_base/ |
not covered by deploy.sh: scp, then deploy.sh nav to rebuild |
behavior/*.py |
~/behavior/ |
deploy.sh explore |
selfcare/*.py |
~/selfcare/ |
deploy.sh selfcare |
selfcare/expected.json |
~/selfcare/ |
not covered by deploy.sh: scp |
robot_face/face_fb.py |
~/face/face_fb.py |
nothing; the robot's copy differs from the repo (chapter 31) |
On 2026-10-07 the files in bt/, urdf/ and config/robot_model.json were identical on both sides (md5 read back).
Every install script that installs a unit takes the unit file from /home/burgerbarn/setup/, for example
install -m 0644 /home/burgerbarn/setup/rosorin-api.service /etc/systemd/system/rosorin-api.service in
scripts/install_api_service.sh. The one exception is install_base_service.sh, which takes rosorin-base.service
from ~/ros2_ws/src/rosorin_base/systemd/. The scripts themselves are run from there too
(sudo bash ~/setup/install_ros2.sh, as their headers say). Running units also execute scripts from there:
rosorin-nav runs nav_stop.sh, rosorin-explore and rosorin-practice run explore_run.sh, rosorin-task@ runs
come_run.sh, the rosorin-base drop-in runs wait_addresses.sh, and robot_home.sh calls go_home.sh. No script
in the repo copies these files. During the rebuild they went over one at a time, for example
scp -q scripts/go_home.sh rosorin-wifi:setup/.
Do the copy as one dry run first. -a keeps modes and times, -v lists files, -n changes nothing, -c compares
by checksum so identical files are not listed. systemd/ with the trailing slash copies the folder's contents, not the
folder.
On your laptop:
cd ~/CCode/rosorin-pro
rsync -avnc --exclude=__pycache__ scripts/*.sh scripts/*.py scripts/setup/*.sh systemd/ rosorin-wifi:setup/
rsync -avnc --exclude=__pycache__ scripts/tests/ rosorin-wifi:setup/tests/
rsync -avnc scripts/make_golden.sh rosorin-wifi:golden/
Check
The dry run on 2026-10-07 listed 29 files for~/setup. Most were never copied (for example
rosorin-vslam.service,rosorin-practice.service,rosorin-face.service,rosorin-cam.service,
install_vision.sh,robot_bt_rtk.sh,wifi_powersave_off.sh); two are older on the robot
(install_isaac_ros.sh,rollback_isaac_ros.sh). It listed 9 files for~/setup/testsand nothing for
~/golden. None of the scripts a running unit uses (nav_stop.sh,explore_run.sh,come_run.sh,
go_home.sh,wait_addresses.sh) was listed: they are identical.
If the list holds only what you expect, run the same three lines without -n:
On your laptop:
cd ~/CCode/rosorin-pro
rsync -avc --exclude=__pycache__ scripts/*.sh scripts/*.py scripts/setup/*.sh systemd/ rosorin-wifi:setup/
rsync -avc --exclude=__pycache__ scripts/tests/ rosorin-wifi:setup/tests/
rsync -avc scripts/make_golden.sh rosorin-wifi:golden/
Not test-built
Only the dry run was done (2026-10-07, the Mac's openrsync against rsync 3.2.7 on the robot). The real copy has
not been run on this robot.
If it fails
- Never add
--delete. Some files exist only on the robot:install_come_service.sh,
rollback_come_service.sh,rosorin-come.service,chain_probe.py,webtest/,logs/(the stop-proof logs
of 2026-10-02),wheels/(the cuVSLAM wheelinstall_isaac_ros.shdownloads) and
face_kiosk.sh.unused-20261007.- If the dry run ever lists
nav_stop.sh,explore_run.sh,come_run.sh,go_home.shorwait_addresses.sh,
the copy changes what a running unit does at its next start. Compare first:
ssh rosorin-wifi cat setup/explore_run.sh | diff - scripts/explore_run.sh.- A unit file copied into
~/setupchanges nothing until its install script runs
(sudo bash ~/setup/install_<name>.sh), and that script runssystemctl enable --now.
scripts/ops/readback.sh is the first and last thing you run in every session. It opens one ssh connection to the
robot and one to bigbuddy, prints one screen, and changes nothing. The file is already in the repo; read it before you
run it.
On your laptop:
cat -n ~/CCode/rosorin-pro/scripts/ops/readback.sh
The whole file, 20 lines:
On your laptop:
#!/bin/bash
# Tier 1. The robot and bigbuddy, read back in one screen. Run from the Mac. Changes nothing.
R=${ROBOT:-rosorin-wifi}; B=${BIGBUDDY:-bigbuddy}
ssh -o ConnectTimeout=8 $R 'source /opt/ros/humble/setup.bash 2>/dev/null
echo "== robot $(hostname) $(uptime -p) load $(cut -d" " -f1-3 /proc/loadavg)"
echo "services: $(systemctl is-active rosorin-base rosorin-nav rosorin-mind rosorin-camera rosorin-vslam rosorin-api rosorin-contact | tr "\n" " ")"
echo "tasks: explore $(systemctl is-active rosorin-explore) home $(systemctl is-active rosorin-task@home) come $(systemctl is-active rosorin-task@come)"
echo "driver: $(timeout 5 ros2 topic echo --once /rosorin_board_driver/state --field data 2>/dev/null | grep -o "\"plugged\": [a-z]*\|\"enabled\": [a-z]*\|\"estop\": [a-z]*\|\"arm_enabled\": [a-z]*\|\"servo_silent\": \[[0-9, ]*\]" | tr "\n" " ")"
echo "nav: $(timeout 5 ros2 topic echo --once /nav/status --field data 2>/dev/null | head -1 | cut -c1-150)"
echo "mind: $(timeout 5 ros2 topic echo --once /rosorin_mind/mode --field data 2>/dev/null | head -1 | cut -c1-120)"
echo "devices: array usb $(lsusb | grep -c Seeed) card $(grep -c Array /proc/asound/cards) | camera $(lsusb | grep -c Aurora) | lidar $(ls /dev/lidar 2>/dev/null | wc -l) | screen $(cat /sys/class/drm/card1-DP-1/status 2>/dev/null)"
C=$(grep -i Array /proc/asound/cards | head -1 | awk "{print \$1}")
[ -n "$C" ] && echo "array: alsa $(amixer -c $C cget numid=5 2>/dev/null | grep -o "values=[0-9,]*" | tail -1) $(amixer -c $C cget numid=6 2>/dev/null | grep -o "values=[0-9,]*" | tail -1) | fw $(sudo ~/tools/reSpeaker_XVF3800_USB_4MIC_ARRAY/host_control/jetson/xvf_host VERSION 2>/dev/null | tail -1) | mic gain $(sudo ~/tools/reSpeaker_XVF3800_USB_4MIC_ARRAY/host_control/jetson/xvf_host AUDIO_MGR_MIC_GAIN 2>/dev/null | tail -1) | usb speed $(cat /sys/bus/usb/devices/1-*/speed 2>/dev/null | paste -sd, )"
echo "voice: api gain stage $(grep -o "ROBOT_SPEECH_GAIN_DB., .[0-9]*" ~/buddy_link/robot_api.py | grep -o "[0-9]*$") dB"'
ssh -o ConnectTimeout=8 $B 'bash -s' <<'B'
echo "== bigbuddy: buddy-voice $(systemctl --user is-active buddy-voice) | robot-mic $(systemctl --user is-active robot-mic) | pipewire $(systemctl --user is-active pipewire pipewire-pulse wireplumber | tr '\n' ' ')"
echo "ears: $(pactl list short sources 2>/dev/null | grep -i 'robot_mic\|xvf_asr\|buddy_aec_source' | awk -F'\t' '{printf "%s(%s) ", $2, $5}')"
echo "mouth: $(grep -h BUDDY_MOUTH ~/.config/systemd/user/buddy-voice.service.d/*.conf 2>/dev/null | cut -d= -f3 || echo robot) | wake th $(grep -h BUDDY_WAKE_TH ~/.config/systemd/user/buddy-voice.service.d/*.conf 2>/dev/null | cut -d= -f3)"
journalctl --user -u buddy-voice --since '-10 min' --no-pager -o cat 2>/dev/null | grep -v -i token | grep -i 'WAKE\|heard\|reply' | tail -3 | cut -c1-110
B
Two details in it matter. The robot part is one long single-quoted string sent to the robot's shell, which is why the
inner quotes are double quotes. The bigbuddy part goes through bash -s with a quoted heredoc, because bigbuddy's
login shell is fish and would not parse bash syntax.
Run it:
On your laptop:
bash ~/CCode/rosorin-pro/scripts/ops/readback.sh
Check
Real output on 2026-10-07, about 20:29 UTC, robot parked on the charger:== robot rosorin up 2 hours, 38 minutes load 12.84 9.86 8.43 services: active active active active active active active tasks: explore inactive home inactive come inactive driver: "enabled": false "estop": null "plugged": true "arm_enabled": false "servo_silent": [] nav: {"state": "ready", "ready": true, "heals": 0, "t": 1791405007.4, "pose": [-0.187, 0.242, 8.0], "fit": 0.83, "probe_fails": 0, "mapping": false} mind: {"mode": "sleep", "why": "nobody for 120 s", "enabled": true, "have_arm": false} devices: array usb 1 card 2 | camera 1 | lidar 1 | screen array: alsa values=60,60 values=60 | fw VERSION 2 1 1 | mic gain AUDIO_MGR_MIC_GAIN 135 | usb speed 12,480,12,480,480,480,12 voice: api gain stage 12 dB == bigbuddy: buddy-voice active | robot-mic active | pipewire active active active ears: robot_mic(RUNNING) xvf_asr(RUNNING) buddy_aec_source(IDLE) mouth: | wake th
What each line means and what good looks like:
| Line | What it reads | Good | Not good |
|---|---|---|---|
== robot |
uptime and load average | load around 7-10 at idle with navigation up (status.md 2026-10-05); self-care flags load above 18 | load far above that for minutes: read journalctl -u rosorin-selfcheck |
services: |
base, nav, mind, camera, vslam, api, contact, in that order | seven times active |
failed or activating for more than a minute: escalate (runbook) |
tasks: |
the wheel skills | inactive while parked |
active means the robot is driving; failed means the last run of that skill ended in failure, read its log in ~/explore/ |
driver: |
board driver state | "estop": null, "servo_silent": []; "enabled": false while parked |
a servo number in servo_silent: escalate; empty line in the first minute after boot is normal |
nav: |
/nav/status from nav_supervisor |
"ready": true, "heals": 0, "probe_fails": 0, fit around 0.8-0.9; "mapping": false while plugged in |
"state": "lost" ("no single place explains the scan"), or starting_nav for more than about 3 minutes after boot |
mind: |
/rosorin_mind/mode |
"enabled": true; mode sleep, curious or track |
"enabled": false after a test: someone left it disabled |
devices: |
USB and device nodes | array usb 1, card 2, camera 1, lidar 1; screen empty in light mode |
any 0 |
array: |
the reSpeaker's levels | alsa values=60,60 values=60, fw ... 2 1 1, mic gain ... 135 |
other levels: run audio_levels.sh robot |
voice: |
the speech gain stage in the deployed robot_api.py | 12 dB (code default) |
a different number: someone deployed a different robot_api.py |
== bigbuddy: |
Buddy's services | all active |
anything else |
ears: |
the PipeWire sources | robot_mic(RUNNING) xvf_asr(RUNNING) |
robot_mic missing or SUSPENDED: run buddy_ears.sh robot |
mouth: and wake th |
drop-ins on bigbuddy | both empty today, which means the code defaults: mouth on the robot, threshold 0.5 | evo after mouth: Buddy speaks on the Evo, not the robot |
| last lines | WAKE, heard, reply in the last 10 minutes |
present if someone spoke | none, if nobody spoke |
If it fails
- The runbook's expected values are partly stale.
docs/runbook.mdsays "array card 1" and a wake threshold
of 0.4 from20-wake.conf. Live, the array is card 2, and20-wake.confwas moved to
~/voice/reverted-20261006/on bigbuddy when the audio was reverted to the ears-and-mouth milestone (commit
448b466), so the threshold is the code default 0.5 andwake thprints nothing. Trust the table above.usb speedlists every device on USB bus 1, not only the array. A12in the list does not mean the array
fell to full speed. Read the array alone (command from the record):
ssh rosorin-wifi 'for d in /sys/bus/usb/devices/*; do [ -f $d/product ] && grep -qi xvf3800 $d/product && echo "array speed $(cat $d/speed)"; done'.
Good:array speed 480.ssh: connect to host 192.168.1.108 port 22: Host is down: the robot fell off Wi-Fi. See the last section.
Do not run it in a tight loop
The script starts three shortros2 topic echoprocesses on the robot. Each one is a new DDS participant. On
2026-10-06 the navigation "deaf to the map transform" fault recurred right after another node appeared, and the
eProsima shared-memory port check that runs when a participant opens a port is a candidate mechanism - not proven
(status.md 2026-10-06, research hour 5). Run the read-back when you need it, not every few seconds.
scripts/ops/deploy.sh <part> copies one part of the repo to the robot, restarts what has to restart, waits 20 s and
runs readback.sh. It changes into the repo root itself, so you can run it from anywhere. ROBOT=rosorin uses the
wired address instead of Wi-Fi.
| Target | Copies | Then |
|---|---|---|
mind |
vision/mind.py world.py reappear.py look_learner.py to ~/vision |
py_compile, restart rosorin-mind (the arm is released and taken again, about 100 s) |
api |
buddy_link/robot_api.py to ~/buddy_link |
py_compile, SIGKILL then restart rosorin-api (an open /audio stream hangs a plain stop) |
driver |
ros2/rosorin_base/rosorin_base/*.py and config/*.yaml |
colcon build --packages-select rosorin_base, stop rosorin-mind, restart rosorin-base, wait 10 s, start rosorin-mind (the mind hangs when the driver restarts under it) |
nav |
the same plus launch/*.py |
colcon build, restart rosorin-nav |
explore |
behavior/*.py to ~/behavior |
nothing to restart: the skills start on demand |
selfcare |
selfcare/*.py to ~/selfcare |
nothing to restart: the timer runs it every 15 min |
On your laptop:
bash ~/CCode/rosorin-pro/scripts/ops/deploy.sh mind
Check
Real output from 2026-10-07 01:37 UTC (the camera watchdog deploy; the record kept only these four lines):deployed mind; reading back in 20 s services: active active active active active active active mind: {"mode": "sleep", "why": "start", "enabled": true, "have_arm": false} devices: array usb 1 card 2 | camera 1 | lidar 1 | screen
"why": "start"is the mind's first state after a restart. Then read the mind's own log for errors:
ssh rosorin-wifi 'journalctl -u rosorin-mind --since "-60s" --no-pager -o cat | grep -iE "error|traceback" | tail -3'
must print nothing.
If it fails
py_compileonly proves the file parses. On 2026-10-02 an untested line in board_driver crashed the driver
the moment the wheels were enabled (lessons.md). Run driver changes on the robot, wheels in the air, before any
run on the floor. On 2026-10-06 a method inserted intocome_here.pyswallowed the next__init__line and
every "come here" aborted with "missing inputs": after editing a class, read the whole__init__back and run
the task's--drymode.- There is no automatic rollback. The script makes
~/retired/<date>on the robot but copies nothing into it;
the old file is overwritten. Roll back by deploying the previous version from git, as the record did on
2026-10-07:git checkout milestone-ears-mouth-on-robot-2026-10-06 -- buddy_link/robot_api.py, then deploy.- Files deploy.sh does not copy are listed in the first table. A change to
vision/think.pydeployed with
deploy.sh mindstays on the Mac.
After a re-plug, a reboot or a new array, the reSpeaker XVF3800's settings must be put back. audio_levels.sh sets the
two ALSA playback controls to 60 (0 dB) and stores them, sets the mic gain (default 135) and the LED effect (default 2,
rainbow, the owner's choice on 2026-10-07), sets the codec output levels to 9 and saves them on the board, then prints
what it reads back.
On your laptop:
bash ~/CCode/rosorin-pro/scripts/ops/audio_levels.sh robot
The arguments are robot|bigbuddy [mic_gain] [led]. On the robot it uses Seeed's Jetson xvf_host binary with sudo;
the codec commands exist only in Seeed's Python control, which runs from the ~/tools/xvfenv venv (chapter 27).
Check
Real output, 2026-10-07 17:41 UTC:array on robot: fw VERSION 2 1 1 | alsa values=60,60 values=60 | mic gain AUDIO_MGR_MIC_GAIN 135 | LED LED_EFFECT 2 | usb speed 10000,12,480 (12 = re-plug it)The
usb speedfield here is the set of speeds of every USB device on the robot, so12is always in it. The
array itself readarray speed 480a few seconds later. Use the array-only command from the read-back section.
If it fails
no array on robot: no ALSA card named Array. Checklsusb | grep Seeed. If the ring flashes red, the array
booted its factory image because Mute was held while plugging it in: re-plug without touching Mute.- Array at 12 (full speed) after a firmware flash or a reboot: playback stalls and capture runs at half rate. On
2026-10-07 a USB unbind/bind did not help; only a physical re-plug brought it back to 480. Do not unbind the array
(lessons.md 2026-10-06 night).- The robot's voice is too loud or too quiet. The PulseAudio sink volume is the same ALSA control as numid 5
(measured: Pulse 90 % = 58,58), and this script puts it back to 100 %. The voice level isSPEECH_OUT_DBin
buddy_link/robot_api.py(-2.75 dB since 2026-10-07), deployed withdeploy.sh api.
buddy_ears.sh robot makes the robot's array Buddy's ears and mouth; buddy_ears.sh bigbuddy moves both to an array
plugged into bigbuddy and the Evo speaker. On bigbuddy it copies one of two saved versions of
60-buddy-echo-cancel.conf over the active one, adds or removes the 30-mouth.conf drop-in, enables or disables
robot-mic.service, restarts PipeWire, then robot-mic and buddy-voice.
On your laptop:
bash ~/CCode/rosorin-pro/scripts/ops/buddy_ears.sh robot
Check
Real output, 2026-10-07 17:41 UTC (bigbuddy prints its local time, PDT):ears on robot: robot_mic(RUNNING) xvf_asr(RUNNING) buddy_aec_source(IDLE) 10:42:05 ready: wake yo_buddy_v3 @ 0.5, STT on GPU, LLM google/gemma-4-12b, voice am_onyx, mic robot_mic
Safety
The script restarts PipeWire on bigbuddy, so any sound playing there stops for a few seconds. It depends on two
files that exist only on bigbuddy, in~/.config/pipewire/pipewire.conf.d/:
60-buddy-echo-cancel.conf.robotmic-20261006(not in the repo) and60-buddy-echo-cancel.conf.bak-20261006-robotmic
(the same as the repo'svoice/60-buddy-echo-cancel.conf). Keep both.
Not test-built
buddy_ears.sh bigbuddyhas not been run since the drop-in60-mic.conf(BUDDY_MIC=robot_mic) was added on
bigbuddy on 2026-10-07 01:29 UTC. The script does not touch that drop-in, so inbigbuddymode Buddy would still
readrobot_mic. After running it, theready:line must end inmic buddy_aec_source; if it says
mic robot_mic, Buddy is listening to a source nothing feeds. The record has no step for this case.
The standing configuration since 2026-10-07 is "light mode": audio and voice, no screen, rosorin-face disabled.
robot_mode.sh face brings Buddy's face back on the robot's touchscreen. Both survive a reboot. Audio, voice, mind and
navigation are the same in both modes.
On your laptop:
bash ~/CCode/rosorin-pro/scripts/ops/robot_mode.sh light
Check
Real output, 2026-10-07 02:12 UTC:mode: face service inactive / disabled | voice active | mind active
If it fails
From a clean boot the touchscreen's NVIDIA display driver is not loaded, so in face mode the face draws for a
1280x1024 fallback. Loadingnvidia-drmby hand coincided with a reset of the robot on 2026-10-07: do not do it
blind (status.md, pause point 2026-10-06 late). The robot's copy offace_fb.pyis older than the repo's (chapter 31).
The runbook lists scripts/ops/robot_home.sh as a Tier 1 procedure: it stops the wheel skills, pauses the mind, sends
one home goal through ~/setup/go_home.sh, and turns the mind back on. Expected: SUCCEEDED, wheels false, mind
true.
Safety
This script moves the wheels. Run it only with the robot unplugged, on the floor, the path clear, and you watching
with the gamepad e-stop in reach.
docs/runbook.md sorts every routine task by the model that may do it:
| Tier | Who | May do |
|---|---|---|
| 1 | Haiku, or you | Run the runbook's commands and compare the output with its "expected" column. Nothing else typed on the robot or bigbuddy. |
| 2 | Sonnet | One known change in one file, deployed with deploy.sh, read back, a status note. The array firmware flash. No arm or wheel code, nothing that changes what the robot decides. |
| 3 | Opus or Fable | Unknowns, diagnosis, safety, anything that writes to the arm or wheels or changes a decision the robot makes. |
Escalate to tier 3 when any output says Traceback, refused, lost, healing, servo_silent is not empty,
navigation is not ready for more than 3 minutes, a service is failed or activating for more than a minute, the task
needs the floor, the arm, the wheels, a new rule or a part decision, or two runs of the same procedure disagree.
The repo holds two Claude Code agents for tiers 1 and 2, in .claude/agents/:
ops.md: model: haiku, tools Bash, Read. Reads the runbook, runs exactly the listed command from the repo root,ESCALATE: for anything else.dev.md: model: sonnet, tools Bash, Read, Edit, Grep, Glob. One known change in one file, py_compile,deploy.sh, readback.sh, a measured result, a dated line in docs/status.md, commit and push to rebuild, withThey load when Claude Code is started in the repo folder (cd ~/CCode/rosorin-pro, then claude); the main session
hands routine work to them by name.
Not test-built
Both agents were committed on 2026-10-07 (commit bfb61d8) and have not been used in a session. The ops agent
compares against the runbook's "expected" column, which is stale on two lines (array card, wake threshold): it
would report MISMATCH on a healthy robot until the runbook is corrected.
The robot's clock and its log folders use UTC. bigbuddy prints local time (PDT).
| What | Where | Read it with |
|---|---|---|
| Any robot service | persistent system journal (632 MB on 2026-10-07) | journalctl -u rosorin-nav --since '-10 min' -o cat |
| The previous boot | the same journal | journalctl --list-boots, then journalctl -b -1 |
| The mind's events | ~/vision/mind/<UTC day>/events.jsonl |
see below |
| The mind's thoughts | ~/vision/think/<UTC day>/thoughts.jsonl, and thinks: lines in journalctl -u rosorin-mind |
see below |
| Self-care | ~/selfcare/data/latest.json, reports/<day>.jsonl, actions.jsonl, say_queue.json |
see below |
| Battery | ~/battery/<day>.csv, ~/battery/plugged.json |
tail |
| Drives | ~/explore/<run>/log.jsonl |
tail |
| Buddy | bigbuddy user journal | runbook command below |
Warnings and errors from one robot service in the last ten minutes (the runbook's command; any unit works):
On your laptop:
ssh rosorin-wifi "journalctl -u rosorin-nav --since '-10 min' -o cat | grep -i 'warn\|error' | tail -20"
The mind writes one JSON line per event. The newest file and its last lines:
On the robot:
f=$(ls -t ~/vision/mind/*/events.jsonl | head -1); echo $f; tail -3 $f | cut -c1-200
Check
Lines look like these (2026-10-07):/home/burgerbarn/vision/mind/20261007/events.jsonl {"t": 1791337079.705, "event": "start", "mode": "sleep"} {"t": 1791337086.371, "event": "thought", "mode": "sleep", "skill": "identify", "args": {"object": "rock wall", "where": "center"}, "thought": "I see ...
tis Unix time in seconds. Event names includestart,mode,thought,found,lost,track,
heartbeat,arm_refused,camera_silent,camera_back,heard,owner_stop,body_skill_started,
body_skill_done(all fromvision/mind.py).
What the mind is thinking right now, from its journal:
On the robot:
journalctl -u rosorin-mind --since '-5 min' -o cat | grep thinks | tail -3
A line reads like 17:43:56 thinks: The owner is looking directly at me and seems to be interacting with something in his hands. -> keep_watching {}.
The latest self-care round (built from the two read-only commands the record used on 2026-10-05):
On the robot:
python3 -c "
import json,os; d=json.load(open(os.path.expanduser('~/selfcare/data/latest.json'))); print('healthy:', d['healthy'], '| disabled:', d['disabled'], '| took_s:', d['took_s'], '| evidence items:', len(d['evidence']))
for f in d['findings']: print(' ', json.dumps(f)[:170])"
Check
On 2026-10-05 21:53 UTC it printed:healthy: True | disabled: True | took_s: 14.1 | evidence items: 10 {"what": "the last three recorded drives failed (none in the last two hours)", "evidence": ["E10"], "action": "none: old news", "result": "noted"}
disabled: Truemeant the owner's off switch~/selfcare/DISABLEDexisted then. Since 2026-10-07 the file is
DISABLED.off-20261007and self-care acts again; today the line must saydisabled: False. The keys of
latest.jsonaret, time, took_s, healthy, driving, disabled, evidence, findings.
If it fails
healthy: Truedoes not mean the robot is well. On 2026-10-05 self-care said healthy while the claw servo was
silent, the arm had been refused 4,800 times and the wheels could not be enabled: it had no check for the arm
(whole_project_review.md, findings 12 and 42). Read the mind'sarm_refusedevents and the driver line of the
read-back as well.
Buddy's side, on bigbuddy (the runbook's command; the grep -v -i token keeps any line that mentions a key out of
your terminal):
On your laptop:
ssh bigbuddy "journalctl --user -u buddy-voice --since '-10 min' -o cat | grep -v -i token | tail -20"
A change that passed its read-back is worth a golden set (chapter 8): something you can restore if the next change goes
wrong. The newest set is 20260929T1829Z. Nothing after 2026-09-29 is in any set: not Docker, not Isaac ROS, not
navigation always-on, self-care, the robot's audio, Bluetooth or the Wi-Fi fix.
Check the room first. The robot keeps its sets in ~/golden; bigbuddy keeps copies in ~/rosorin-golden.
On your laptop:
ssh rosorin-wifi 'df -h / | tail -1; du -sh ~/golden'
ssh bigbuddy 'bash -s' <<'EOF'
df -h /home | tail -1; du -sh ~/rosorin-golden
EOF
Check
On 2026-10-07: robot/dev/nvme0n1p16 352G 100G 237G 30% /and28G /home/burgerbarn/golden(five sets);
bigbuddy/dev/nvme1n1p3 929G 476G 451G 52% /homeand28G /home/burgerbarn/rosorin-golden.
The next set will be much larger than the last one
make_golden.sharchives the whole root file system except~/golden,/tmpand/var/tmp. Of the 100 GB used
today, 28 GB is~/golden; the rest includes 29 GB under/var/lib/containerd(the vLLM image of the disabled
rosorin-brain, 31.1 GB as Docker counts it), 17 GB in~/bagsand 4.2 GB in~/hf_cache. The set of 2026-09-29
was 9.2 GB and took about 16 minutes. A set of today's system has never been made; its size and time are unknown.
The script has no exclusion for these folders, and the record has no decision on removing the unused image.
The record ran the backup over the wired link (rosorin, 192.168.1.109). Plug the Ethernet cable in, then start the
script in the background on the robot and wait for it:
On your laptop:
ssh rosorin 'nohup sudo -n bash ~/golden/make_golden.sh > /tmp/golden.log 2>&1 & echo $! > /tmp/golden.pid; sleep 2; head -3 /tmp/golden.log'
ssh rosorin 'P=$(cat /tmp/golden.pid); while kill -0 $P 2>/dev/null; do sleep 15; done; tail -2 /tmp/golden.log'
Then check the set on the robot, copy it to bigbuddy through the Mac (scp -3), and check it again there. This is the
command used for 20260929T1829Z:
On your laptop:
G=$(ssh rosorin 'basename $(ls -dt ~/golden/2*/ | head -1)'); echo "set $G"
ssh rosorin "cd ~/golden/$G && sha256sum -c --quiet SHA256SUMS && echo robot SHA256 OK; du -sh ."
ssh bigbuddy "mkdir -p ~/rosorin-golden/$G" && scp -3 -q "rosorin:golden/$G/*" "bigbuddy:rosorin-golden/$G/" && ssh bigbuddy "cd ~/rosorin-golden/$G && sha256sum -c --quiet SHA256SUMS && echo bigbuddy SHA256 OK; du -sh .; df -h / | tail -1"
Check
For20260929T1829Zthe record shows:== GPT The operation has completed successfully. == boot partitions p2..p15 (raw, zstd) ... GOLDEN_DONE /home/burgerbarn/golden/20260929T1829Z set 20260929T1829Z robot SHA256 OK 9.2G .and
bigbuddy SHA256 OKafter the copy. Then add a row for the new set to the table indocs/restore.md.
If it fails
root is not /dev/nvme0n1p16, abort: the robot did not boot from the rebuilt partition. Stop and read
chapter 8.rootfs archive FAILED: tar returned 2 or more. Exit code 1 is normal on a running system (a file changed
while it was read); those files are listed intar_warnings.txtin the set.- bigbuddy's disk was 98 % full on 2026-09-29 (21 GB free) until old snapshots were deleted on 2026-10-01. Check
dfbefore copying.- The script unmounts
/boot/efiwhile it images partitions 2-15 and mounts it again on exit.
On 2026-10-07 at 17:44 UTC the Wi-Fi firmware (rtw89, RTL8852CE) hung while leaving power save and did not recover.
The robot kept running everything - navigation, mind, self-care - but nobody could reach it, and Buddy lost the robot's
microphone. From the Mac (ping, arp -an, ssh; the Mac prints local time) it looked like this:
On your laptop:
10:44:53 192.168.1.108:down 192.168.1.109:down
? (192.168.1.108) at (incomplete) on en0 ifscope [ethernet]
ssh: connect to host 192.168.1.108 port 22: Host is down
The robot's kernel log, read after the reboot:
On the robot:
Oct 07 17:44:12 rosorin kernel: rtw89_8852ce_git 0001:01:00.0: SER catches error: 0x1000
Oct 07 17:44:12 rosorin kernel: rtw89_8852ce_git 0001:01:00.0: firmware failed to ack for leaving ps mode
The fix is scripts/setup/wifi_powersave_off.sh (chapter 7). Check that it is in place:
On the robot:
NetworkManager --print-config 2>/dev/null | grep wifi.powersave
iw dev wlP1p1s0 get power_save
Check
wifi.powersave=2 Power save: off
If it fails
wifi.powersave=3with the fix file present: the file sorts before Ubuntu's
default-wifi-powersave-on.confand loses. On 2026-10-07 the first try was named99-wifi-powersave-off.conf
and NetworkManager still printedwifi.powersave=3; renamed tozz-wifi-powersave-off.confit printed 2.- The fix is not yet shown to stop the hangs (hardware.md). The driver also resets the firmware about twice an hour
on its own (SER catches error, 23 times on 2026-10-05/06); those recover by themselves.
When ping 192.168.1.108 fails and the robot's address shows (incomplete) in arp -an, there is no way in over
Wi-Fi. The record has two ways back:
ssh rosorin); the link takes several.109 was down too.Reloading the Wi-Fi driver without a reboot was never tried. After either way back, read the kernel log for the
minutes before the drop. Use a time window, as the record did: on 2026-10-07 journalctl -b -1 landed on a boot from
the day before, not on the one that dropped. The robot's clock is wrong early in boot (lessons.md); whether that is
the reason was not checked.
On the robot:
journalctl -k --since "2026-10-07 17:40:00" --until "2026-10-07 17:51:00" --no-pager -o short | grep -E "SER catches|failed to ack" | tail -5
Put your own drop's time in the window.
Then run readback.sh, and audio_levels.sh robot if the array line is wrong.
Safety
Remove the Ethernet cable before the robot drives (hardware.md).
readback.sh shows seven active services, "ready": true, "servo_silent": [], fw VERSION 2 1 1, mic gainrobot_mic(RUNNING) on bigbuddy.deploy.sh target covers the file you change, and which files it does not copy.disabled: False.iw dev wlP1p1s0 get power_save says Power save: off.20260929T1829Z exists on the robot and on bigbuddy, or you know why you have not made one.Where this comes from
scripts/ops/readback.sh,deploy.sh,audio_levels.sh,buddy_ears.sh,robot_mode.sh,robot_home.sh;
scripts/make_golden.sh;scripts/setup/wifi_powersave_off.sh;scripts/install_*.sh(paths in~/setup);
docs/runbook.md(tiers, expected values, escalation);.claude/agents/ops.mdanddev.md(commit bfb61d8,
2026-10-07);docs/restore.md(golden sets);docs/hardware.md(Wi-Fi 2026-10-07 17:44 UTC, array speed, Pulse =
numid 5);docs/lessons.md2026-10-02 (driver crash), 2026-10-06 (come_here, array re-plug);docs/status.md
2026-10-05 (load, self-care), 2026-10-06 research hour 5 (deaf transform), pause point 2026-10-06 late (face);
docs/research/whole_project_review.mdfindings 12 and 42; commit 448b466 (audio revert). Command log
(cmdlog_robot.md): readback 2026-10-07 01:37, 17:32, 17:41, 17:52; audio_levels 17:41; Wi-Fi drop 17:44-17:52;
golden 2026-09-29 18:29; deploy 2026-10-07 01:37. survey_live_robot.md (~/setup contents, kiosk files),
survey_bigbuddy.md (drop-ins, 60-mic.conf). Read-only lookups on 2026-10-07: rsync dry run,df,du, the
readback run quoted above.
← Face, screen and media (optional) · Contents · The work light →