The robot's own mind · Chapter 21 · Time: 2 hours to install and check; plan an evening to read the code · Level: Advanced · Status: Done on this robot
rosorin-mind.service running vision/mind.py - one always-on loop that owns the robot's head, sleeps, wakes, follows Matt, searches for him, looks around the room, keeps a world model, learns from its own logs every night and asks the language model on bigbuddy for ideas - and the list of what it still gets wrong.
The mind is one Python program, vision/mind.py, that runs all the time as rosorin-mind.service. In normal
life it is the only thing that moves the robot's head (the arm with the camera on it): it watches for Matt, follows
him, looks around when he is gone, and lets the arm rest when nobody has been around. It is built as one loop that
owns the head because the driver e-stops when two programs command the arm at once, and because the head must give
way at once to the gamepad, the wheels and any script that needs the arm. Everything it needs to react runs on the
robot; the ideas it gets from the language model on bigbuddy are optional, and it carries on without them.
Every step in this chapter was done on this robot (service installed 2026-09-30 00:38 UTC, timers 2026-09-30 and
2026-10-01, the latest change deployed 2026-10-07 18:59 UTC). The code has grown since; this chapter describes the
version in the repo at commit bfb61d8, which is byte-identical to what runs on the robot.
The mind uses almost everything built so far. Required:
rosorin-base running, with the arm services (arm/recover, arm/enable, arm/read_state,reset_estop) and the arm_controller/joint_jog input. The owner's arm limits file must exist at~/ros2_ws/src/rosorin_base/config/arm_limits.yaml; the mind reads it when it starts.rosorin-camera publishing /aurora/rgb/image_raw, /aurora/depth/image_raw, /aurora/rgb/camera_info.~/vision/venv, the patch32 engine, and the vision files in ~/vision.sudo systemctl to restart the camera and to startOptional (the mind runs without them, with less):
/nav/status and the map frame it never drives and its world model stays~/selfcare/say.py it cannot ask to be heard (it logs say_failed).rosorin-practice and rosorin-explore.The arm moves by itself
From the moment the mind runs, the arm can move at any time: when it sees a person, once an hour to look around,
when Buddy hears "yo buddy". Keep the space around the arm clear and keep the gamepad within reach: any button
except MODE is an e-stop, and the mind pauses while the gamepad is not in its locked mode. The head moves at up to
72 deg/s (docs/decisions.md2026-10-01).
New idea: a control loop
A robot that reacts does the same three things over and over: sense (read the latest camera frame and
state), decide (what to do now), act (send a command). One pass is one step. The mind's step runs
once per new camera frame, about 14 times a second while awake, 2 times a second asleep. Each step must be short:
a step that waits for something slow (the network, a person) stalls everything after it.
New idea: proportional control, dead zones and hysteresis
To keep Matt in the middle of the picture, the mind measures how far off-centre he is (the error, as an
angle) and turns the head at a speed proportional to it: speed = gain x error, capped at a top speed. Far off
means fast, nearly centred means slow, so the head lands softly. A dead zone means "close enough, do not
move". Hysteresis means using two different limits for starting and stopping: the mind starts re-centring
when he is more than 10 deg off and stops only when he is within 2 deg, so it does not twitch at the boundary.
New idea: velocity commands for joints
The mind does not say "go to position X". It sends joint speeds (rad/s) onarm_controller/joint_jog, a
control_msgs/JointJogmessage, many times a second. The base driver integrates them at 25 Hz with an
acceleration limit, checks every step against the arm limits and the arm-collision model, and glides to a stop
when commands stop arriving. That gives smooth motion and keeps every safety check in the driver.
New idea: latched topics
A normal ROS 2 topic delivers messages only to subscribers that exist when the message is sent. A topic with
TRANSIENT_LOCAL durability keeps its last message and hands it to anyone who subscribes later, like a
notice pinned to a board. The mind publishes its mode that way (/rosorin_mind/mode) and reads navigation's
status that way (/nav/status), so a script that starts late still learns the current state at once.
New idea: learning from its own logs
Most of what the mind decides is fixed code (its "body"). A few decisions are learned from its own records with
no human labels: where Matt usually is (presence map), where he reappears after leaving the picture (reappear),
what each head direction is worth looking at (look learner), and what his face looks like (owner faces). Some of
these update live, two retrain every night. Whether each one helps is measured in the record, and two of them do
not yet (see "Known weaknesses").
The mind subscribes to the camera, LiDAR and driver topics through the Scan node from vision/scan.py
(lines 113-124), which it reuses. It drops that node's arm/goal publisher on purpose: "the driver e-stops on >1
arm/goal publisher" (mind.py lines 112-113). The robot API (chapter 22) reads /rosorin_mind/mode and
/rosorin_mind/seen and publishes /buddy/heard.
The vision files went to ~/vision in chapter 20. The mind imports world.py, reappear.py, look_learner.py,
attention.py, scan.py, scan_filter.py and think.py from the same folder. Copy the unit files and install
scripts into ~/setup, where every install script looks for them:
On your laptop:
cd ~/CCode/rosorin-pro
scp systemd/rosorin-mind.service systemd/rosorin-owner-faces.service systemd/rosorin-owner-faces.timer \
systemd/rosorin-reappear.service systemd/rosorin-reappear.timer \
scripts/install_mind_service.sh scripts/rollback_mind_service.sh \
scripts/install_owner_faces_timer.sh scripts/install_reappear_timer.sh rosorin-wifi:setup/
If it fails
No script copies files into~/setup; on this robot it was always done withscpas above, a few files at a
time. If an install script stops withinstall: cannot stat '/home/burgerbarn/setup/...', the file is not there.
Run every new version of the mind once by hand before it becomes a service. The record has two reasons: on
2026-10-01 a deploy crashed the mind on its first frame, and on 2026-10-02 the service crash-looped 33 times because
a variable was used before it existed, waking the arm every 27 s (docs/lessons.md 2026-10-01, docs/decisions.md
2026-10-02).
On the robot:
source /opt/ros/humble/setup.bash; source ~/ros2_ws/install/setup.bash
cd ~/vision && . venv/bin/activate
timeout -s INT 70 python mind.py 2>&1 | grep -v "Warn\|meshgrid\|_VF\|warn\|TRT\|detach\|out +=" | tail -12
tail -n 12 $(ls -d ~/vision/mind/*/ | tail -1)events.jsonl | cut -c1-200
timeout -s INT 70 stops it after 70 s with Ctrl-C's signal, which the mind treats as "release the arm and exit".
On a robot where the service already runs, stop it first (sudo systemctl stop rosorin-mind) so that only one mind
commands the head, and start it again afterwards.
Check
The terminal showsmind runningand noTraceback. The event log of that run starts and stops in sleep mode
(2026-09-30):On the robot:
{"t": 1790728524.44, "event": "start", "mode": "sleep"} {"t": 1790728583.803, "event": "stop", "mode": "sleep"}If someone was in front of the camera, you will also see the head turn to them and
tracklines.
If it fails
FileNotFoundErrorforarm_limits.yaml: the mind reads the owner's limits at import (mind.pylines 75-80).
Deploy the driver package first (chapter 11).RuntimeError: timeout waiting for topics: no camera frame, camera info or driver state within 60 s
(main(), line 842). Checkrosorin-cameraandrosorin-base.- On 2026-09-30 the first version printed a traceback from
rclpy.shutdown()whentimeoutstopped it. The
currentmain()takes over SIGINT and SIGTERM itself and shuts ROS down only if it is still running
(lines 836-838 and 854).
The unit, systemd/rosorin-mind.service:
On the robot:
[Unit]
Description=ROSOrin robot mind (attention: watch for the owner, track, curious, sleep) - arm only via joint_jog
After=rosorin-base.service rosorin-camera.service
Wants=rosorin-base.service rosorin-camera.service
[Service]
User=burgerbarn
Environment=ROS_DOMAIN_ID=0
ExecStart=/bin/bash -c 'source /opt/ros/humble/setup.bash && source /home/burgerbarn/ros2_ws/install/setup.bash && source /home/burgerbarn/vision/venv/bin/activate && cd /home/burgerbarn/vision && exec python mind.py'
KillSignal=SIGINT
TimeoutStopSec=15
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
After= / Wants=: start after the driver and the camera, and pull them in if they are not running.User=burgerbarn: the mind writes its files in /home/burgerbarn/vision.ExecStart: source ROS 2, then the robot's own workspace (the same environment the driver runs in), then theexec python mind.py so systemd signals Python directly.KillSignal=SIGINT: stopping the service is a Ctrl-C, which runs the finally: block in main() that releasesTimeoutStopSec=15 gives it time to do so.Restart=always, RestartSec=10: a crash restarts it after 10 s, forever. This is why a crash loop shows up as theInstall it with the script (scripts/install_mind_service.sh):
On the robot:
#!/bin/bash
# Install + enable rosorin-mind.service (vision/mind.py). Rollback: sudo bash ~/setup/rollback_mind_service.sh
set -euo pipefail
[ "$(id -u)" = 0 ] || { echo "run with sudo"; exit 1; }
install -m 0644 /home/burgerbarn/setup/rosorin-mind.service /etc/systemd/system/rosorin-mind.service
systemctl daemon-reload
systemctl enable --now rosorin-mind.service
sleep 10; systemctl --no-pager --lines=6 status rosorin-mind.service | head -10
On the robot:
sudo bash ~/setup/install_mind_service.sh 2>&1 | tail -6
Check
The status block ends with the running process (2026-09-30):On the robot:
Tasks: 27 (limit: 18452) Memory: 470.7M CPU: 9.767s CGroup: /system.slice/rosorin-mind.service └─28349 python mind.pyThen, a minute later:
On the robot:
source /opt/ros/humble/setup.bash systemctl is-active rosorin-mind journalctl -u rosorin-mind --since "-4 min" --no-pager -o cat | grep -E "mind running| -> " timeout 6 ros2 topic echo --once /rosorin_mind/mode --field dataOn the robot:
active mind running {"mode": "sleep", "why": "start", "enabled": true, "have_arm": false}
If it fails
systemctl status rosorin-mindshowsactivating (auto-restart)again and again: it is crash-looping. Read the
traceback withjournalctl -u rosorin-mind -n 50 --no-pager -o cat | grep -A 15 Traceback, fix, and run it in
the foreground (step 2) before starting the service again.
The mind is always in one of four modes (mind.py line 3). It starts in sleep (line 123).
| Mode | The arm | The camera | What it does |
|---|---|---|---|
paused |
not touched | ignored | Someone else owns the arm or the wheels. |
sleep |
released (the driver tucks it after 2 minutes) | 2 detections per second | Waits for a person, an hourly glance, an hourly practice look-around, or a voice event. |
curious |
held | every frame | Slow looks around, searches for Matt, room scans. Back to sleep after 120 s with nobody. |
track |
held | every frame while re-centring, 4 per second while holding still | Keeps Matt centred. |
"busy" is any of: /rosorin_mind/enable set to false (disabled), an e-stop latched (e-stop), the wheels enabled
(wheels enabled), or the gamepad not in its locked mode (gamepad in use). Each change of mode is printed to the
journal as HH:MM:SS old -> new (why) and written to the event log. These are real lines from the robot's journal:
On the robot:
paused -> sleep (resume)
curious -> track (found you (person))
sleep -> curious (looking around the room)
track -> sleep (arm released by driver)
sleep -> paused (disabled)
track -> curious (lost you)
sleep -> curious (glance around)
curious -> sleep (nobody for 120 s)
main() (mind.py lines 834-855) sets up ROS, builds the Mind, waits up to 60 s for a camera frame, the camera
info and the driver state, then calls step() after every short spin:
On the robot:
def main():
# own the SIGINT/SIGTERM: the context must stay alive so the finally-block can release the arm
rclpy.init(signal_handler_options=SignalHandlerOptions.NO)
import signal
signal.signal(signal.SIGTERM, lambda *_: (_ for _ in ()).throw(KeyboardInterrupt()))
mind = Mind()
mind.event('start')
mind.publish_mode()
mind.n.spin_until(lambda: all(k in mind.n.last for k in ('rgb', 'info', 'state')), 60, 'topics')
print('mind running', flush=True)
try:
while rclpy.ok():
rclpy.spin_once(mind.n, timeout_sec=0.02) # was 5 ms: a 200 Hz poll for a ~15 fps camera
mind.step()
except KeyboardInterrupt:
pass
finally:
mind.release_arm()
mind.event('stop')
mind.n.destroy_node()
if rclpy.ok():
rclpy.shutdown()
rclpy.spin_once delivers whatever messages have arrived (each subscription stores the latest one); step() then
works on the latest state. SIGTERM is turned into the same KeyboardInterrupt as SIGINT, so any stop runs the
finally: block and releases the arm.
Mind.__init__ (lines 109-160) builds the detector (chapter 20: NanoOWL patch32 with the attention prompts, the
taught words and 'an object'), the service /rosorin_mind/enable, the latched mode publisher, the thinker thread,
the look learner, the world model, and loads the reappear model and the presence map from disk. It sets the first
idle glance and the first practice look-around one hour after start (lines 128 and 136).
step() is mind.py lines 281-511. Read it with this list beside it.
Housekeeping (285-286). body_step checks whether a wheel skill it started has ended and logs the result
(561-578). camera_watch restarts a silent camera (256-279, see "The camera watchdog").
Owner stop words (287-289). Any speech Buddy heard that contains stop, stay, halt or freeze ends its wheel
skills and sets "stay" for 30 minutes (stop_body, 580-585). This runs even while paused.
Clearing its own contact reflex (290-297). If the e-stop reason is 'estop topic' (the contact monitor's
stop, chapter 19), no skill of its own is running, and 20 s have passed since the last try, it calls
reset_estop. A gamepad e-stop is the owner's and is never cleared here.
Yield (298-310). The block that makes the mind give way:
On the robot:
busy = (not self.enabled and 'disabled') or (st.get('estop') and 'e-stop') or \
(st.get('enabled') and 'wheels enabled') or (st.get('pad_mode') not in (None, 'locked') and 'gamepad in use')
if busy:
if self.mode != 'paused':
if busy == 'disabled':
self.release_arm() # deliberate handoff (scripts/Buddy): give the arm back
else:
self.send(0.0, 0.0)
self.have_arm = False # gamepad/wheels/e-stop own it now: don't touch
self.set_mode('paused', busy)
return
if self.mode == 'paused':
self.set_mode('sleep', 'resume'); self.last_det = 0.0
A deliberate handoff (disabled) releases the arm properly. For the gamepad, the wheels and an e-stop it sends
one zero-speed command and then keeps its hands off: someone else owns the arm now.
Heard events (311-320). Events from /buddy/heard are handled by on_heard (624-682, below). If the driver
dropped the arm (for example the auto-tuck), curious or track falls back to sleep ("arm released by
driver").
Perception gate (321-329). Nothing happens without a new frame. Asleep: at most SLEEP_FPS = 2 detections a
second. Tracking and holding still: at most HOLD_FPS = 4. Otherwise every frame.
Detection (330-341). The frame goes through the detector with per-label thresholds (chapter 20), and
aim_point() picks "Matt" from the hits. Every 0.5 s it publishes /rosorin_mind/seen with its mode, the hits
and where the world model last saw Matt; Buddy reads that through the robot API.
Inspect (342-351). For 5 s after Buddy's robot_see tool asks it to inspect ("what am I holding"), and only
while it sees him, it picks a point with inspect_point() (93-105): the object nearest a hand, else the hand, else
the best-scoring object anywhere in view. If that point is within 15 deg it adjusts onto it; otherwise it holds
still so the picture is taken from where it already looks.
Thinking (352-369). Scene change = mean absolute difference of a 64x48 grey thumbnail against one from about a
minute ago. It hands the frame and a state summary to the thinker thread (thinker.feed), applies any decision
that came back (apply_thought), and writes a heartbeat event once a minute.
Sleep branch (371-389). Shown below.
Awake (391-511): servo temperatures every 120 s (over 55 C: release and sleep 10 minutes), the room scan if
one is running, objects into the world model every 2 s, then tracking, follow-through, losing him, going back to
sleep, the call search, the lost-look, a thinker-chosen look direction, and finally the slow curious sweep.
The sleep branch, lines 371-389:
On the robot:
if self.mode == 'sleep':
self.wake_hits = self.wake_hits + 1 if ap else 0
if self.wake_hits >= WAKE_HITS and now >= self.cool_until:
self.wake_hits = 0
if self.take_arm():
self.awake_since, self.next_temp = now, now + TEMP_EVERY
self.set_mode('track', f'saw you ({ap[2]})')
self.acquire(rgb, ap, dets, now)
elif st.get('plugged') and now >= self.next_practice and now >= self.cool_until:
# on the charger and alone: practise looking - its own inquisitive look-around, learning from each look
self.next_practice = now + PRACTICE_EVERY
self.on_heard({'event': 'scan_room', 'why': 'practice'}, now, st)
elif now >= self.next_glance and now >= self.cool_until:
self.next_glance = now + GLANCE_EVERY
if self.take_arm():
self.awake_since, self.next_temp, self.glance_until = now, now + TEMP_EVERY, now + GLANCE_S
self.pause_until, self.sweep_until = 0.0, 0.0
self.set_mode('curious', 'glance around')
return
Two detections in a row wake it straight into track (WAKE_HITS = 2, so one flicker does not). Otherwise, on the
charger, it practises looking around once an hour; off the charger it glances around for 20 s once an hour.
take_arm() (214-237) asks the driver to recover the arm from its tuck if needed, then to enable it. If enabling is
refused with "outside limits" (a servo sagged a few pulses past a limit while resting) it makes one slow recover move
and tries again; if enabling is still refused it backs off for 30 s (it used to retry every second all day:
docs/lessons.md 2026-10-01). release_arm() (239-243) sends zero speed, waits 0.4 s and disables the arm; the driver then tucks it
after 2 minutes.
The robot API turns Buddy's events into messages on /buddy/heard (chapter 22). on_heard() (lines 624-682) handles
five kinds:
| Event | What the mind does |
|---|---|
wake ("yo buddy") |
Dropped if older than 5 s (queued while paused). Tracking: an acknowledging head tilt. Otherwise wakes or stays curious ("heard yo buddy") and starts the call search. |
speech (Buddy's transcript) |
Kept as owner_said for the thinker for 60 s; the stop words end its wheel skills. |
learn (the owner named an object) |
Adds the name as a new detector word at once (add_labels, chapter 20), with "a " or "an " in front, and saves it to ~/vision/learned_objects.json. |
scan_room |
Starts a room scan, taking the arm first if asleep. |
inspect (Buddy's robot_see) |
Looks at what he holds for 5 s; if it is not tracking him, it searches for him first. |
aim_point() in vision/attention.py (lines 33-47) treats the owner as one thing. A face wins; otherwise a point
15 % down from the top of the person box (about where the head is); otherwise a hand:
On the robot:
def aim_point(dets):
"""(u, v, what) for the owner-as-one-entity, or None."""
faces = [d for d in dets if d[0] == 'a face']
people = [d for d in dets if d[0] == 'a person']
hands = [d for d in dets if d[0] == 'a hand']
if faces:
_, _, (x0, y0, x1, y1) = max(faces, key=lambda d: d[1])
return (x0 + x1) / 2, (y0 + y1) / 2, 'face'
if people:
_, _, (x0, y0, x1, y1) = max(people, key=lambda d: d[1])
return (x0 + x1) / 2, y0 + 0.15 * (y1 - y0), 'person'
if hands:
_, _, (x0, y0, x1, y1) = max(hands, key=lambda d: d[1])
return (x0 + x1) / 2, (y0 + y1) / 2, 'hand'
return None
The first version flipped between face and person every frame (attention.py docstring); this is the fix.
The pixel (u, v) is turned into two angles with the camera's intrinsics K from /aurora/rgb/camera_info
(focal length about 420 px): atan((u - cx) / fx) sideways and the same for up and down. Then, mind.py lines
417-436:
On the robot:
u, v, w = ap
rx = math.atan((u - K[0, 2]) / K[0, 0]); ry = math.atan((v - K[1, 2]) / K[1, 1])
# owner 2026-10-01: "moving all over the place ... wasted energy". The aim point flips face <-> person
# (4-6 deg apart, log) and the 1.5 deg dead zone chased every flicker. Now: smoothed error + hold still
# while the target is within HOLD_DEG; once it leaves, re-centre until within SETTLE_DEG, then hold.
fe = getattr(self, 'f_err', None)
self.f_err = (rx, ry) if fe is None or now - self.last_seen > LOST_S else \
(fe[0] + ERR_ALPHA * (rx - fe[0]), fe[1] + ERR_ALPHA * (ry - fe[1]))
ex, ey = self.f_err
# tilt at the owner's limit (WRIST_LIM, enforced in send) can't close the vertical error: count that axis as done,
# else re-centring never ends and the arm keeps pushing (2026-10-01: wrist 280, owner standing above it)
wr = ((n.state() or {}).get('arm_pose') or {}).get('4', 178)
if (wr >= WRIST_LIM[1] and ey < 0) or (wr <= WRIST_LIM[0] and ey > 0): ey = 0.0
off = math.degrees(math.hypot(ex, ey))
rc = getattr(self, 'recentring', False)
self.recentring = (off > SETTLE_DEG) if rc else (off > HOLD_DEG)
if self.recentring:
v1 = max(-VMAX, min(VMAX, GAIN * ex)); v4 = max(-VMAX, min(VMAX, GAIN * ey))
else:
v1 = v4 = 0.0
| Constant | Value | Where | Meaning |
|---|---|---|---|
GAIN |
1.8 1/s | attention.py 29 |
joint speed (rad/s) per radian of error. 2.5 overshot by +-18 deg. |
VMAX |
1.25 rad/s | attention.py 29 |
top speed, about 72 deg/s |
ERR_ALPHA |
0.4 | mind.py 72 |
smoothing: each frame moves the error 40 % of the way to the new measurement |
HOLD_DEG |
10 deg | mind.py 72 |
start re-centring when he is further off than this |
SETTLE_DEG |
2 deg | mind.py 72 |
stop re-centring when he is this close |
LOST_S |
1.5 s | attention.py 30 |
how long without a detection counts as "lost" |
Signs, measured on the robot: target to the right means joint 1 +rad (the pan pulse goes down); target below means
joint 4 +rad (the wrist pulse goes down). One pulse is 0.24 deg (RAD_PER_PULSE, line 68); pan 500 faces straight
ahead. send() (192-206) also keeps the tilt inside the owner's limits (arm_limit_4: [121, 935]) by allowing only
motion back toward the range, and brings the wrist roll back to level after a curious head tilt.
While tracking, the mind also records where he is: Matt's position on the map at most once a second (412-416), +1
in the presence map every 10 s while he is centred (439-446), and, when the aim point is a face, a picture every 5 s,
at most 500 a day (450-453), for the nightly face job.
Check
Stand in front of the robot at a few metres. Within a second or two the head turns to you and the journal shows
a change totrack. Walk slowly sideways: the head follows, then holds still while you stay within about 10 deg
of the centre. On 2026-09-29 the measured tracking error was a median of 1.6 deg sideways and 1.0 deg up-down
(docs/status.md).On the robot:
journalctl -u rosorin-mind --since "-2 min" --no-pager -o cat | grep " -> "
track. Between 0.4 and 1.3 sFOLLOW_V = 0.9 rad/s, easing out, then holds still.curious ("lost you"). It logs a lost event with the side he left by andlost_step, 789-817). It waits until 2.1 s after he left, then picks one target: where the worldSLEEP_AFTER), or 20 s after an idle glance.on_heard, 664-682, and 480-491). On "yo buddy" (a wake event) or an inspect request from Buddy'srobot_see tool when it is not already tracking him, it searches: first where it last had him centred (else the reappear prediction), then up to fourThe record says this works partly: after a loss it found him again at a learned spot 57 of 80 times on 2026-10-05,
because he had not gone anywhere (docs/research/whole_project_review.md, finding 28).
A room scan is about 25 s of looking around the room with the arm only, going after whatever looks interesting. It
starts on a scan_room event: from Buddy's "what do you see in the room" (chapter 26), from the explorer at each new
vantage point (chapter 24), as hourly practice on the charger, or when the thinker picks practice_looking.
room_scan_step() (707-768) on every frame:
room_mem, cells of 70 pulses, about 17 deg)._interest() (694-705): its score, plus up to 1.0 for novelty (never seen'an object' (something it has no name for), and 0 if it already looked there in this scan.SCAN_S = 25 s or MAX_LOOKS = 6 close looks. room_scan_finish() (770-787) writes a 3x2 mosaicmeta.json and ~/vision/room_memory.json; the robot API serves the mosaic to Buddy.Every settled look also goes to the look learner (vision/look_learner.py): it counts the 5 cm cells of the room
the depth camera showed that were new in this scan, the floor area seen, the share of valid depth, and the share of
LiDAR returns left while the head is there (the arm can block the LiDAR). value() turns that into one number per
head direction, which the glance choice uses:
On the robot:
def value(self, key):
"""0..1: how much a look there has been worth (what it shows minus the LiDAR it takes). Unknown = 1."""
m = self.model['looks'].get(key)
if not m:
return 1.0
best = max(1.0, self.model.get('best_new_cells', 1.0))
base = max((v.get('lidar_valid', 0.0) for v in self.model['looks'].values()), default=0.0)
cost = max(0.0, base - m.get('lidar_valid', base)) / base if base > 0 else 0.0
return float(max(0.0, min(1.0, m.get('new_cells', 0.0) / best - cost)))
Ask for one scan by hand. The arm will move for about 25 s:
On the robot:
source /opt/ros/humble/setup.bash
ros2 topic pub --once /buddy/heard std_msgs/msg/String "{data: \"{\\\"event\\\": \\\"scan_room\\\", \\\"why\\\": \\\"test\\\"}\"}"
sleep 40
grep -E "room_scan_done|look_learned" ~/vision/mind/$(date -u +%Y%m%d)/events.jsonl | tail -3 | cut -c1-200
ls ~/vision/roomscans | tail -1
python3 -c "import json;m=json.load(open('/home/burgerbarn/practice/head_model.json'));print('head directions learned:', len(m['looks']))"
Check
Aroom_scan_doneevent naming a new folder in~/vision/roomscans/withmosaic.jpgandmeta.jsonin it,
andlook_learnedevents. The first look-around with the learner on 2026-10-02 printed
head directions learned: 23(docs/status.mdrecords 9 from an earlier run the same day). The first
inquisitive scan on 2026-10-01 made 5 close looks and 11 glances: guitar, guitar, rug, beanbag chair, chair.
If it fails
Nothing happens: the mind ispaused(gamepad not locked, wheels enabled, e-stop) or cooling after hot servos;
check/rosorin_mind/mode. A refused arm shows asarm_refusedevents; read the driver log (chapter 13).
vision/world.py keeps what the robot knows as positions on its own map (the map frame of the live
slam_toolbox map, chapter 17) in ~/vision/world.json: Matt's last position and every object, with when it was last
seen and how often. The key function places a detection on the map:
On the robot:
def locate(tf, scan, u, K):
"""Map-frame (x, y) of what is at image column u, or None. Bearing: right of centre = negative yaw."""
rp, cy = robot_pose(tf), camera_yaw(tf)
if rp is None or cy is None:
locate.why = 'no robot pose' if rp is None else 'no camera TF'; return None
bearing = cy - math.atan((u - K[0, 2]) / K[0, 0])
d = lidar_range(scan, bearing)
if d is None:
locate.why = f'no LiDAR return at bearing {math.degrees(bearing):.0f} deg'; return None
locate.why = None
x, y, th = rp
return x + d * math.cos(th + bearing), y + d * math.sin(th + bearing)
The direction comes from TF (where the camera points on the robot, base_link -> camera_link0, plus the pixel's
angle). The distance comes from the LiDAR, not the depth camera, because the depth camera has no reading on dark or
shiny things (chapter 20): lidar_range() takes the LiDAR returns within 5 deg of that bearing, between 0.25 and
6 m, and uses the 20th percentile (the near side of whatever is there). The robot's position comes from TF
map -> base_link; when navigation is down (on the charger) there is no position and nothing is added.
Same-label sightings closer than MERGE_M = 0.7 m are the same thing (averaged); something seen only once and not
again within a day is forgotten (FORGET_ONCE_S). The see_object docstring and docs/decisions.md say 0.4 m; the
docs differ, the code uses 0.7.
The mind writes Matt at most once a second while tracking a face or a person (412-416), and objects with score 0.3
or more every 2 s (400-407). It reads it when it loses him (world_look, 797-802), for /rosorin_mind/seen, and in
what it tells the thinker (owner_last_seen, my_place_on_map).
Check
With navigation up and the robot off the charger, objects appear in~/vision/world.json. After the world model
was put back on 2026-10-06:On the robot:
objects placed on its map in the last 5 min: [('a shelf', 10, 3), ('a beanbag chair', 2, 3)] objects known in all: 47 of 11 kindsOn the robot:
python3 -c "import json;d=json.load(open('/home/burgerbarn/vision/world.json'));print('matt:', d.get('matt'));print({k: len(v) for k, v in d['objects'].items()})"
Why it was taken out and put back
The world model went in on 2026-10-02 and was removed the same night by a "reset to known-good" commit
(06e493b); the docs kept describing it, so for three days the record and the robot disagreed
(whole_project_review.mdsection 3). It was restored on 2026-10-06 (commit60d3a0a). The other part removed
that night, "head leads, body follows" (the base turning under the head), has not been put back, although
docs/decisions.md2026-10-02 calls it a standing rule.
| Learner | Updated | Data | Model file | Used for |
|---|---|---|---|---|
| Presence map | live, every 10 s while centred on Matt | head pose | ~/vision/presence.json |
the call search (learned spots) |
| Look learner | live, every settled look in a room scan | depth, LiDAR, TF | ~/practice/head_model.json |
where to glance in a room scan |
| Reappear | nightly 03:20, rosorin-reappear.timer |
lost and found events |
~/vision/reappear.json |
where to look after losing him |
| Owner faces | nightly 03:00, rosorin-owner-faces.timer |
the mind's face pictures | ~/vision/owner/ |
nothing yet |
vision/reappear.py is small enough to read in full and shows the pattern of all the robot's self-training: read its
own logs, count, write a model file, let the mind read it.
The header and constants. A lost followed by a found within 2 minutes is one example; the model is a histogram of
where (head pan, in bins of 50 pulses = 12 deg) he reappeared, one per exit side; newer examples weigh more, halving
every 14 days:
On the robot:
"""Where does the owner reappear after the robot loses them? Learned by the robot from its own experience.
Data: the mind's events.jsonl ('lost' = head pose + exit side when the owner left view, 'found' = head pose where they
were seen again + seconds later). Each lost->found pair within FOUND_WINDOW_S is one example. Model: per exit side, a
weighted histogram over head pan (pulses) of where the owner turned up, with the mean wrist per bin; newer examples
weigh more (half-life HALF_LIFE_D days). No labels, no Claude: the robot trains this itself, nightly
(rosorin-reappear.timer) and the mind reloads it at start. Untrained / too little data -> the mind falls back to
"keep looking toward where you left".
Run: python reappear.py (reads ~/vision/mind/*/events.jsonl, writes ~/vision/reappear.json)"""
import glob, json, math, os, time
HERE = os.path.dirname(os.path.abspath(__file__))
MODEL = os.path.join(HERE, 'reappear.json')
FOUND_WINDOW_S = 120.0 # a 'found' later than this after 'lost' is not the same episode
BIN = 50 # pan bin width (pulses, 12 deg)
HALF_LIFE_D = 14.0
MIN_EXAMPLES = 3 # below this per side the mind uses its fallback
train() walks every day's event log in order and pairs each lost with the next found:
On the robot:
def train(root=os.path.join(HERE, 'mind')):
examples = []
for f in sorted(glob.glob(os.path.join(root, '*', 'events.jsonl'))):
lost = None
for line in open(f):
try:
e = json.loads(line)
except ValueError:
continue
if e.get('event') == 'lost':
lost = e
elif e.get('event') == 'found' and lost and 0 < e['t'] - lost['t'] < FOUND_WINDOW_S:
examples.append((lost['side'], lost['pan'], e['pan'], e['wrist'], e['t'] - lost['t'], e['t']))
lost = None
Then, per side, it adds each example's weight 0.5 ** (age_in_days / 14) to the bin of the pan where he turned up,
keeps the weighted mean wrist and delay per bin, and writes the model to a temporary file that is then renamed over
the old one (a rename is atomic, so the mind never reads half a file):
On the robot:
now = time.time()
model = {'trained': now, 'examples': len(examples), 'sides': {}}
for side in (-1, 1):
bins = {}
for s, _, pan, wrist, dt, t in examples:
if s != side:
continue
w = 0.5 ** ((now - t) / 86400.0 / HALF_LIFE_D)
b = bins.setdefault(str(int(pan // BIN)), {'w': 0.0, 'wrist': 0.0, 'n': 0, 'dt': 0.0})
b['w'] += w; b['wrist'] += w * wrist; b['dt'] += w * dt; b['n'] += 1
for b in bins.values():
b['wrist'] /= b['w']; b['dt'] /= b['w']
model['sides'][str(side)] = {'n': sum(b['n'] for b in bins.values()), 'bins': bins}
json.dump(model, open(MODEL + '.tmp', 'w'), indent=1); os.replace(MODEL + '.tmp', MODEL)
return model
The mind calls load() once at start (mind.py line 147) and predict() when it needs a place to look: the centre
of the heaviest bin for that side, or None with fewer than 3 examples:
On the robot:
def load():
try:
return json.load(open(MODEL))
except (OSError, ValueError):
return None
def predict(model, side):
"""(pan, wrist) where the owner most likely reappears after leaving toward `side`, or None if not learned yet."""
s = (model or {}).get('sides', {}).get(str(side))
if not s or s['n'] < MIN_EXAMPLES:
return None
k, b = max(s['bins'].items(), key=lambda kv: kv[1]['w'])
return (int(k) + 0.5) * BIN, b['wrist']
if __name__ == '__main__':
m = train()
print(json.dumps({'examples': m['examples'], 'per_side': {k: v['n'] for k, v in m['sides'].items()},
'predict': {s: predict(m, s) for s in (-1, 1)}}))
The complete file, ~/vision/reappear.py:
On the robot:
"""Where does the owner reappear after the robot loses them? Learned by the robot from its own experience.
Data: the mind's events.jsonl ('lost' = head pose + exit side when the owner left view, 'found' = head pose where they
were seen again + seconds later). Each lost->found pair within FOUND_WINDOW_S is one example. Model: per exit side, a
weighted histogram over head pan (pulses) of where the owner turned up, with the mean wrist per bin; newer examples
weigh more (half-life HALF_LIFE_D days). No labels, no Claude: the robot trains this itself, nightly
(rosorin-reappear.timer) and the mind reloads it at start. Untrained / too little data -> the mind falls back to
"keep looking toward where you left".
Run: python reappear.py (reads ~/vision/mind/*/events.jsonl, writes ~/vision/reappear.json)"""
import glob, json, math, os, time
HERE = os.path.dirname(os.path.abspath(__file__))
MODEL = os.path.join(HERE, 'reappear.json')
FOUND_WINDOW_S = 120.0 # a 'found' later than this after 'lost' is not the same episode
BIN = 50 # pan bin width (pulses, 12 deg)
HALF_LIFE_D = 14.0
MIN_EXAMPLES = 3 # below this per side the mind uses its fallback
def train(root=os.path.join(HERE, 'mind')):
examples = []
for f in sorted(glob.glob(os.path.join(root, '*', 'events.jsonl'))):
lost = None
for line in open(f):
try:
e = json.loads(line)
except ValueError:
continue
if e.get('event') == 'lost':
lost = e
elif e.get('event') == 'found' and lost and 0 < e['t'] - lost['t'] < FOUND_WINDOW_S:
examples.append((lost['side'], lost['pan'], e['pan'], e['wrist'], e['t'] - lost['t'], e['t']))
lost = None
now = time.time()
model = {'trained': now, 'examples': len(examples), 'sides': {}}
for side in (-1, 1):
bins = {}
for s, _, pan, wrist, dt, t in examples:
if s != side:
continue
w = 0.5 ** ((now - t) / 86400.0 / HALF_LIFE_D)
b = bins.setdefault(str(int(pan // BIN)), {'w': 0.0, 'wrist': 0.0, 'n': 0, 'dt': 0.0})
b['w'] += w; b['wrist'] += w * wrist; b['dt'] += w * dt; b['n'] += 1
for b in bins.values():
b['wrist'] /= b['w']; b['dt'] /= b['w']
model['sides'][str(side)] = {'n': sum(b['n'] for b in bins.values()), 'bins': bins}
json.dump(model, open(MODEL + '.tmp', 'w'), indent=1); os.replace(MODEL + '.tmp', MODEL)
return model
def load():
try:
return json.load(open(MODEL))
except (OSError, ValueError):
return None
def predict(model, side):
"""(pan, wrist) where the owner most likely reappears after leaving toward `side`, or None if not learned yet."""
s = (model or {}).get('sides', {}).get(str(side))
if not s or s['n'] < MIN_EXAMPLES:
return None
k, b = max(s['bins'].items(), key=lambda kv: kv[1]['w'])
return (int(k) + 0.5) * BIN, b['wrist']
if __name__ == '__main__':
m = train()
print(json.dumps({'examples': m['examples'], 'per_side': {k: v['n'] for k, v in m['sides'].items()},
'predict': {s: predict(m, s) for s in (-1, 1)}}))
Run it once by hand (it only reads the event logs and rewrites ~/vision/reappear.json):
On the robot:
cd ~/vision && venv/bin/python reappear.py
Check
On a new robot there are no examples yet. This is what it printed the first time, on 2026-10-01:On the robot:
{"examples": 0, "per_side": {"-1": 0, "1": 0}, "predict": {"-1": null, "1": null}}After weeks of use the counts grow: 335 examples on 2026-10-05, 1020 on 2026-10-07 (569 for side -1, 451 for
side 1), read live from~/vision/reappear.json.
If it fails
The first version, deployed on 2026-10-01, crashed with
UnboundLocalError: local variable 'now' referenced before assignment(a tuple assignment usednowbefore it
was set). The lesson indocs/lessons.md: run every new script once before you deploy it.
The mind reads reappear.json only when it starts (line 147). A model retrained at 03:20 is used from the next
restart of the mind (a deploy or a reboot).
vision/owner_faces.py (86 lines) finds every face in the mind's saved pictures with MTCNN, turns each into a
512-number embedding with InceptionResnetV1 (VGGFace2 weights), and calls the owner "the largest chain of similar
faces": faces with cosine similarity of at least --sim are linked, links are followed (single linkage), and the
biggest group wins. It writes ~/vision/owner/owner.npz (the group's embeddings and their mean), owner.json
(statistics) and sheet.jpg (every face, green border = owner group). The code default is --sim 0.5; the
docstring says 0.6, which was the first value tried on 2026-09-30 (it gave a group of 8; 0.5 gave 13). The docs
differ; the code uses 0.5.
This model is wrong today
Live on 2026-10-07,owner.jsonreads: 4451 frames, 1467 faces, owner group 1464, lowest similarity inside the
group -0.58. Single linkage chained almost every face into one group, so it says that everyone and everything is
the owner. Nothing in the mind uses it (whole_project_review.mdfinding 10). The timer still runs it every
night. Read it before you trust it:On the robot:
python3 -c "import json;print(json.load(open('/home/burgerbarn/vision/owner/owner.json'))['stats'])"
Both are oneshot services run by a timer, as burgerbarn, at lower CPU priority (Nice=10), in the vision venv.
systemd/rosorin-reappear.service and its timer:
On the robot:
[Unit]
Description=ROSOrin self-training: where the owner reappears after leaving view (vision/reappear.py)
[Service]
Type=oneshot
User=burgerbarn
Nice=10
ExecStart=/bin/bash -c 'source /home/burgerbarn/vision/venv/bin/activate && cd /home/burgerbarn/vision && python reappear.py'
On the robot:
[Unit]
Description=Nightly reappear self-training (03:20)
[Timer]
OnCalendar=*-*-* 03:20:00
Persistent=true
[Install]
WantedBy=timers.target
systemd/rosorin-owner-faces.service and .timer are the same with python owner_faces.py and
OnCalendar=*-*-* 03:00:00. Persistent=true means: if the robot was off at that time, run the job at the next boot.
Install both with their scripts:
On the robot:
sudo bash ~/setup/install_owner_faces_timer.sh
sudo bash ~/setup/install_reappear_timer.sh
systemctl list-timers rosorin-owner-faces.timer rosorin-reappear.timer --no-pager | head -4
Check (live, 2026-10-07)
On the robot:NEXT LEFT LAST PASSED UNIT ACTIVATES Thu 2026-10-08 03:00:00 UTC 6h left Wed 2026-10-07 03:00:03 UTC 17h ago rosorin-owner-faces.timer rosorin-owner-faces.service Thu 2026-10-08 03:20:00 UTC 6h left Wed 2026-10-07 03:20:00 UTC 17h ago rosorin-reappear.timer rosorin-reappear.serviceRight after installing,
LASTreadsn/a.
If it fails
On 2026-10-05 the robot was off at 03:00, soPersistent=trueran the face job at boot: 10 minutes of CPU, in the
middle of the first floor run, which failed with late control loops (docs/research/doing_it_itself_reference.md
section 1). Since thenscripts/explore_run.shstopsrosorin-selfcheck,rosorin-owner-facesand
rosorin-reappearat the start of every drive (chapter 24).
Once every few seconds the mind shows the language model on bigbuddy the current camera frame and a short summary of
its state, and asks it to pick one skill from a fixed menu. The model never sends motor commands: the mind carries
out the chosen skill through its own paths, with all the driver's checks. If bigbuddy is off or slow, the mind does
everything else as before. The code is vision/think.py, 119 lines.
Why the model is on bigbuddy, not on the robot
On 2026-09-30 the robot ran its own vision-language model (Qwen3-VL-4B in NVIDIA's vLLM container,
rosorin-brain.service). The Jetson's memory is shared between CPU and GPU; the model's reservation left too
little, the kernel killed a robot process, the LiDAR reader stalled and the driver e-stopped (docs/lessons.md
2026-09-30). The owner decided the same day: bigbuddy and server are the brain, the robot is navigation and
vision.rosorin-brainwas disabled (it frees about 5 GB: MemAvailable 4.3 -> 12.3 GB). The first thought
served by bigbuddy took 1.37 s; on the robot it had taken 5-6 s (docs/decisions.md2026-09-30). The robot knows
the brain only asBRAIN_URLandBRAIN_MODEL, so a different brain machine is a configuration change.
The header and imports:
On the robot:
"""The robot's thinking - served by the BRAIN on bigbuddy (vision-language model, OpenAI API; LM Studio Gemma 4 12B).
Runs in a thread inside the mind. Every PERIOD s (awake) / SLEEP_PERIOD s (asleep) it shows the model the current
camera frame + a short state summary and asks for ONE skill from a fixed menu, as JSON. The model never commands
motors: the mind executes skills through its own safe paths (joint_jog + driver gates + arm-model veto).
If the brain is down or slow, the mind carries on without thoughts (reflexes and perception never wait on it).
Every thought is logged with its frame (~/vision/think/<UTC day>/) = the data the teacher (Claude) reviews and labels
to train the robot's models (distillation); questions for the owner go to ~/vision/questions.jsonl."""
import base64, json, os, threading, time
import cv2
import numpy as np
import requests
Where the brain is, and how often to ask. LM Studio speaks the OpenAI chat API on port 1234 (chapter 25). It asks every
4 s awake, every 60 s asleep, gives up on an answer after 30 s, and skips a turn unless the view changed by 6 grey
levels on average or a minute has passed:
On the robot:
HERE = os.path.dirname(os.path.abspath(__file__))
# The brain lives on bigbuddy (owner 2026-09-30: bigbuddy + server = brain/knowledge; robot = nav/vision).
URL = os.environ.get('BRAIN_URL', 'http://192.168.1.111:1234/v1/chat/completions')
MODEL = os.environ.get('BRAIN_MODEL', 'google/gemma-4-12b')
PERIOD, SLEEP_PERIOD, TIMEOUT = 4.0, 60.0, 30.0
SCENE_DIFF, MAX_QUIET = 6.0, 60.0 # grey levels (0-255) of change that count as a new scene; s
The menu. Each entry is a skill name and the sentence the model reads about it:
On the robot:
SKILLS = {
'keep_watching': 'keep doing what you are doing (the default)',
'look_toward': 'turn the head toward a direction: args {"direction": "left|right|up|down|center"}',
'look_around': 'slowly look around the room for something new',
'identify': 'you see something worth remembering: args {"object": "<what it is>", "where": "<where in the image>"}',
'ask_owner': 'you are unsure about something and want the owner to tell you: args {"question": "<short>"}',
'rest': 'nothing to do: rest (sleep) for a while',
# the body is yours (owner 2026-10-02: control is delegated to the robot; Claude builds and trains, never drives)
'practice_looking': 'practise looking around with your arm to learn what each direction shows you (arm only, any time)',
'practice_moving': 'practise small moves with your wheels to learn what your body does on this floor '
'(only when state.can_drive; a few minutes, stays near where you are)',
'explore_room': 'drive around the room to map it and see what is there '
'(only when state.can_drive and state.ready_to_explore)',
}
The prompt. The state (a JSON object built by the mind) and the menu are filled in at {state} and {skills}; the
doubled braces are literal braces in the reply format:
On the robot:
PROMPT = """You are the mind of a small home robot: part of "Buddy", the owner's assistant. You live in the owner's
den, on a rug. Your camera is on your arm, so where you look is visible to everyone. You are highly curious: you seek out
what you don't know, you never guess - if unsure, you ask the owner. Your body is yours to use: you move your head any
time, and you drive when state.can_drive is true. Nobody drives you. When the owner is not asking anything of you, you
decide how to use the time - looking, practising, exploring or resting - and you learn from what happens. If the owner
says stop or stay, you stay.
Current state: {state}
Look at the camera image and choose ONE skill:
{skills}
Reply with ONLY JSON: {{"thought": "<one short sentence: what you notice / why>", "skill": "<name>", "args": {{...}}, "confidence": <0-1>}}"""
The Thinker runs in its own thread so a slow answer never blocks the mind's loop. The mind hands it the newest
frame with feed() and collects the newest decision with take(); a lock keeps the two threads from touching the
same variables at once:
On the robot:
class Thinker:
def __init__(self):
self.lock = threading.Lock()
self.frame, self.state, self.asleep = None, {}, False
self.latest = None # (time, decision dict)
self.day, self.log, self.dir = None, None, None
self.ok_count = self.fail_count = 0
threading.Thread(target=self.run, daemon=True).start()
def feed(self, rgb, state, asleep):
with self.lock:
self.frame, self.state, self.asleep = rgb, dict(state), asleep
def take(self):
"""The newest decision not yet taken (or None)."""
with self.lock:
d, self.latest = self.latest, None
return d
def _logfile(self):
day = time.strftime('%Y%m%d', time.gmtime())
if day != self.day:
self.day = day
self.dir = os.path.join(HERE, 'think', day); os.makedirs(self.dir, exist_ok=True)
self.log = open(os.path.join(self.dir, 'thoughts.jsonl'), 'a')
return self.log
The thread's loop: wait for its turn, skip if the scene has not changed, shrink the frame to 448 px wide and send it
as a JPEG inside the chat request. reasoning_effort: 'none' turns off Gemma 4's hidden thinking; without it the
replies came back empty (docs/status.md 2026-09-29):
On the robot:
def run(self):
next_t = time.time() + 5.0
while True:
time.sleep(0.5)
with self.lock:
frame, state, asleep = self.frame, self.state, self.asleep
if frame is None or time.time() < next_t:
continue
next_t = time.time() + (SLEEP_PERIOD if asleep else PERIOD)
# efficiency (owner 2026-10-01): the same scene gave the same thought every 4 s. Think only when the view
# changed (mean abs diff of a 32x24 grey thumbnail) or MAX_QUIET s passed since the last thought.
thumb = cv2.resize(cv2.cvtColor(frame, cv2.COLOR_RGB2GRAY), (32, 24), interpolation=cv2.INTER_AREA).astype(np.int16)
last = getattr(self, 'last_thumb', None)
if last is not None and np.abs(thumb - last).mean() < SCENE_DIFF and time.time() - self.last_think < MAX_QUIET:
continue
self.last_thumb, self.last_think = thumb, time.time()
small = cv2.resize(frame, (448, int(448 * frame.shape[0] / frame.shape[1])))
jpg = cv2.imencode('.jpg', cv2.cvtColor(small, cv2.COLOR_RGB2BGR), [cv2.IMWRITE_JPEG_QUALITY, 85])[1].tobytes()
prompt = PROMPT.format(state=json.dumps(state), skills='\n'.join(f'- {k}: {v}' for k, v in SKILLS.items()))
body = {'model': MODEL, 'max_tokens': 160, 'temperature': 0.3, 'reasoning_effort': 'none', # Gemma 4: no hidden thinking
'messages': [{'role': 'user', 'content': [
{'type': 'image_url', 'image_url': {'url': 'data:image/jpeg;base64,' + base64.b64encode(jpg).decode()}},
{'type': 'text', 'text': prompt}]}]}
Then: take the first { to the last } of the reply as JSON, reject skills that are not on the menu, log every
thought with its picture (errors too), copy ask_owner questions to ~/vision/questions.jsonl, and hand the decision
to the mind:
On the robot:
t0 = time.time()
try:
txt = requests.post(URL, json=body, timeout=TIMEOUT).json()['choices'][0]['message']['content'] or ''
d = json.loads(txt[txt.index('{'):txt.rindex('}') + 1])
if d.get('skill') not in SKILLS:
raise ValueError(f'unknown skill {d.get("skill")}')
d['latency_s'] = round(time.time() - t0, 2)
self.ok_count += 1
except Exception as e:
self.fail_count += 1
d = {'error': f'{type(e).__name__}: {str(e)[:120]}', 'latency_s': round(time.time() - t0, 2)}
fn = time.strftime('%H%M%S', time.gmtime()) + '.jpg'
log = self._logfile()
with open(os.path.join(self.dir, fn), 'wb') as f:
f.write(jpg)
log.write(json.dumps({'t': round(time.time(), 2), 'state': state, 'frame': fn, **d}) + '\n'); log.flush()
if 'error' in d:
continue
if d['skill'] == 'ask_owner':
with open(os.path.join(HERE, 'questions.jsonl'), 'a') as f:
f.write(json.dumps({'t': round(time.time(), 2), 'question': d.get('args', {}).get('question'),
'frame': os.path.join('think', self.day, fn)}) + '\n')
with self.lock:
self.latest = (time.time(), d)
The complete file, ~/vision/think.py:
On the robot:
"""The robot's thinking - served by the BRAIN on bigbuddy (vision-language model, OpenAI API; LM Studio Gemma 4 12B).
Runs in a thread inside the mind. Every PERIOD s (awake) / SLEEP_PERIOD s (asleep) it shows the model the current
camera frame + a short state summary and asks for ONE skill from a fixed menu, as JSON. The model never commands
motors: the mind executes skills through its own safe paths (joint_jog + driver gates + arm-model veto).
If the brain is down or slow, the mind carries on without thoughts (reflexes and perception never wait on it).
Every thought is logged with its frame (~/vision/think/<UTC day>/) = the data the teacher (Claude) reviews and labels
to train the robot's models (distillation); questions for the owner go to ~/vision/questions.jsonl."""
import base64, json, os, threading, time
import cv2
import numpy as np
import requests
HERE = os.path.dirname(os.path.abspath(__file__))
# The brain lives on bigbuddy (owner 2026-09-30: bigbuddy + server = brain/knowledge; robot = nav/vision).
URL = os.environ.get('BRAIN_URL', 'http://192.168.1.111:1234/v1/chat/completions')
MODEL = os.environ.get('BRAIN_MODEL', 'google/gemma-4-12b')
PERIOD, SLEEP_PERIOD, TIMEOUT = 4.0, 60.0, 30.0
SCENE_DIFF, MAX_QUIET = 6.0, 60.0 # grey levels (0-255) of change that count as a new scene; s
SKILLS = {
'keep_watching': 'keep doing what you are doing (the default)',
'look_toward': 'turn the head toward a direction: args {"direction": "left|right|up|down|center"}',
'look_around': 'slowly look around the room for something new',
'identify': 'you see something worth remembering: args {"object": "<what it is>", "where": "<where in the image>"}',
'ask_owner': 'you are unsure about something and want the owner to tell you: args {"question": "<short>"}',
'rest': 'nothing to do: rest (sleep) for a while',
# the body is yours (owner 2026-10-02: control is delegated to the robot; Claude builds and trains, never drives)
'practice_looking': 'practise looking around with your arm to learn what each direction shows you (arm only, any time)',
'practice_moving': 'practise small moves with your wheels to learn what your body does on this floor '
'(only when state.can_drive; a few minutes, stays near where you are)',
'explore_room': 'drive around the room to map it and see what is there '
'(only when state.can_drive and state.ready_to_explore)',
}
PROMPT = """You are the mind of a small home robot: part of "Buddy", the owner's assistant. You live in the owner's
den, on a rug. Your camera is on your arm, so where you look is visible to everyone. You are highly curious: you seek out
what you don't know, you never guess - if unsure, you ask the owner. Your body is yours to use: you move your head any
time, and you drive when state.can_drive is true. Nobody drives you. When the owner is not asking anything of you, you
decide how to use the time - looking, practising, exploring or resting - and you learn from what happens. If the owner
says stop or stay, you stay.
Current state: {state}
Look at the camera image and choose ONE skill:
{skills}
Reply with ONLY JSON: {{"thought": "<one short sentence: what you notice / why>", "skill": "<name>", "args": {{...}}, "confidence": <0-1>}}"""
class Thinker:
def __init__(self):
self.lock = threading.Lock()
self.frame, self.state, self.asleep = None, {}, False
self.latest = None # (time, decision dict)
self.day, self.log, self.dir = None, None, None
self.ok_count = self.fail_count = 0
threading.Thread(target=self.run, daemon=True).start()
def feed(self, rgb, state, asleep):
with self.lock:
self.frame, self.state, self.asleep = rgb, dict(state), asleep
def take(self):
"""The newest decision not yet taken (or None)."""
with self.lock:
d, self.latest = self.latest, None
return d
def _logfile(self):
day = time.strftime('%Y%m%d', time.gmtime())
if day != self.day:
self.day = day
self.dir = os.path.join(HERE, 'think', day); os.makedirs(self.dir, exist_ok=True)
self.log = open(os.path.join(self.dir, 'thoughts.jsonl'), 'a')
return self.log
def run(self):
next_t = time.time() + 5.0
while True:
time.sleep(0.5)
with self.lock:
frame, state, asleep = self.frame, self.state, self.asleep
if frame is None or time.time() < next_t:
continue
next_t = time.time() + (SLEEP_PERIOD if asleep else PERIOD)
# efficiency (owner 2026-10-01): the same scene gave the same thought every 4 s. Think only when the view
# changed (mean abs diff of a 32x24 grey thumbnail) or MAX_QUIET s passed since the last thought.
thumb = cv2.resize(cv2.cvtColor(frame, cv2.COLOR_RGB2GRAY), (32, 24), interpolation=cv2.INTER_AREA).astype(np.int16)
last = getattr(self, 'last_thumb', None)
if last is not None and np.abs(thumb - last).mean() < SCENE_DIFF and time.time() - self.last_think < MAX_QUIET:
continue
self.last_thumb, self.last_think = thumb, time.time()
small = cv2.resize(frame, (448, int(448 * frame.shape[0] / frame.shape[1])))
jpg = cv2.imencode('.jpg', cv2.cvtColor(small, cv2.COLOR_RGB2BGR), [cv2.IMWRITE_JPEG_QUALITY, 85])[1].tobytes()
prompt = PROMPT.format(state=json.dumps(state), skills='\n'.join(f'- {k}: {v}' for k, v in SKILLS.items()))
body = {'model': MODEL, 'max_tokens': 160, 'temperature': 0.3, 'reasoning_effort': 'none', # Gemma 4: no hidden thinking
'messages': [{'role': 'user', 'content': [
{'type': 'image_url', 'image_url': {'url': 'data:image/jpeg;base64,' + base64.b64encode(jpg).decode()}},
{'type': 'text', 'text': prompt}]}]}
t0 = time.time()
try:
txt = requests.post(URL, json=body, timeout=TIMEOUT).json()['choices'][0]['message']['content'] or ''
d = json.loads(txt[txt.index('{'):txt.rindex('}') + 1])
if d.get('skill') not in SKILLS:
raise ValueError(f'unknown skill {d.get("skill")}')
d['latency_s'] = round(time.time() - t0, 2)
self.ok_count += 1
except Exception as e:
self.fail_count += 1
d = {'error': f'{type(e).__name__}: {str(e)[:120]}', 'latency_s': round(time.time() - t0, 2)}
fn = time.strftime('%H%M%S', time.gmtime()) + '.jpg'
log = self._logfile()
with open(os.path.join(self.dir, fn), 'wb') as f:
f.write(jpg)
log.write(json.dumps({'t': round(time.time(), 2), 'state': state, 'frame': fn, **d}) + '\n'); log.flush()
if 'error' in d:
continue
if d['skill'] == 'ask_owner':
with open(os.path.join(HERE, 'questions.jsonl'), 'a') as f:
f.write(json.dumps({'t': round(time.time(), 2), 'question': d.get('args', {}).get('question'),
'frame': os.path.join('think', self.day, fn)}) + '\n')
with self.lock:
self.latest = (time.time(), d)
On every processed frame the mind builds the state (lines 358-364): its mode, what it is doing (doing, the last
"why"), scene_change, I_see (the labels it detects, or nothing specific), owner_last_seen_s_ago, owner_said
(Buddy's last transcript, for 60 s), head_turn_deg, local_time, and everything from self_knowledge()
(lines 513-544):
| Field | Meaning, and how it is worked out |
|---|---|
practice_moves_done |
trials in ~/practice/body_model.json (chapter 24) |
head_directions_learned |
entries in the look learner's model |
have_room_map |
~/maps/room_live.yaml exists |
plugged, battery_v |
from the driver state and /battery (smoothed) |
can_drive |
not plugged, no e-stop, battery at least 10.6 V, not inside an owner "stay", 900 s since its last wheel skill, none running, navigation ready and fresh (under 30 s), no other wheel-skill unit active |
ready_to_explore |
practice_moves_done >= 60 |
owner_said_stay, navigation_ready, wheel_skill_running |
as named |
owner_last_seen, my_place_on_map |
from the world model and TF |
arm_servos_not_answering, arm_refused |
from the driver state and its own last refusal (within 60 s) |
navigation_ready, arm_servos_not_answering, arm_refused and the navigation condition in can_drive were added
on 2026-10-06, after the review found that the mind chose arm skills while its arm was refused and exploration while
navigation was down (findings 40-42); owner_last_seen and my_place_on_map came back with the world model the same
day.
apply_thought() (lines 587-622):
| Skill | What the mind does |
|---|---|
practice_moving, explore_room |
start_body_skill() (546-559), in any mode: refused and logged (body_skill_refused) unless can_drive (and, for explore, ready_to_explore); otherwise releases the arm and runs sudo systemctl start --no-block rosorin-practice or rosorin-explore (chapter 24) |
practice_looking |
starts a room scan, unless tracking or already scanning |
keep_watching, identify, ask_owner |
nothing: logged only |
rest, look_around, look_toward while tracking |
nothing: following Matt wins |
rest |
if curious: release the arm and sleep ("decided to rest") |
look_around |
resets the slow sweep |
look_toward |
steers the head for 4 s: 150 pulses left or right, 40 pulses up or down, or neutral |
While the mind is asleep, a look_around or look_toward thought can wake the arm (mode curious,
"decided to ...") only if the scene changed by at least 8 grey levels and an hour has passed since the last time a
thought woke it; otherwise it logs thought_wake_skipped.
Read the newest thoughts:
On the robot:
f=~/vision/think/$(date -u +%Y%m%d)/thoughts.jsonl; tail -n 3 $f | python3 -c "
import json,sys
for l in sys.stdin:
d=json.loads(l); print(d.get('latency_s'), '|', d.get('thought') or d.get('error'), '->', d.get('skill'), d.get('args'))"
Check
The first thought served by bigbuddy, 2026-09-30:On the robot:
1.37 | I see a large black speaker and some books on the shelf, but I am still looking for my owner. -> identify {'object': 'speaker', 'where': 'left side of the image'}The journal shows the same as
thinks: ... -> skill {args}. Asleep, expect about one thought a minute.
If it fails
- Every line has
errorinstead ofthought: bigbuddy is off, LM Studio is not serving, or the model name is
wrong (chapter 25). The mind keeps working without thoughts.- Under load LM Studio has answered HTTP 400 with an error body
terminatedand nochoices
(docs/capabilities.md,vision/vlm.py); the thinker logs that as aKeyErrorand tries again next turn.- Empty replies:
reasoning_effort: 'none'is missing from the request (Gemma 4 then spends its tokens thinking).
When nobody is around, the mind wakes the arm for three reasons: an idle glance, a practice look-around on the
charger, and a thought that saw the scene change. Until 2026-10-07 these came every 5, 5 and 10 minutes, and the
first ones 60 s after a restart. The owner found the head looking up and down for no reason and asked for "every
hour". Since 18:59 UTC that day, lines 64, 74 and 90:
On the robot:
PRACTICE_EVERY = 3600.0 # s: plugged in and alone, it practises looking this often (owner 2026-10-07: every hour, was 5 min)
GLANCE_EVERY, GLANCE_S = 3600.0, 20.0 # idle glance: owner 2026-10-07 every hour (was 5 min)
THINK_WAKE_EVERY = 3600.0 # a thought may wake the arm at most every hour (owner 2026-10-07; was 10 min), only if the scene changed # asleep: wake for a short look around every 5 min (one camera direction misses you)
The first glance and the first practice now also come an hour after a restart (time.time() + GLANCE_EVERY,
line 128; time.time() + PRACTICE_EVERY, line 136). Waking on seeing a person and on "yo buddy" did not change. The
trailing comment on line 90 ("every 5 min") is left over from the old value; the number is 3600.
It was deployed with scripts/ops/deploy.sh mind and read back:
On the robot:
source /opt/ros/humble/setup.bash
grep -nE "^(PRACTICE_EVERY|GLANCE_EVERY|THINK_WAKE_EVERY)" ~/vision/mind.py | cut -c1-60
echo "mind service: $(systemctl is-active rosorin-mind) since $(systemctl show rosorin-mind -p ActiveEnterTimestamp --value)"
echo "mind: $(timeout 6 ros2 topic echo --once /rosorin_mind/mode --field data 2>/dev/null | sed -n 1p | cut -c1-120)"
Check
On the robot:64:PRACTICE_EVERY = 3600.0 # s: plugged in and alone, it 74:GLANCE_EVERY, GLANCE_S = 3600.0, 20.0 # idle glance: 90:THINK_WAKE_EVERY = 3600.0 # a thought ma mind service: active since Wed 2026-10-07 18:59:27 UTC mind: {"mode": "sleep", "why": "start", "enabled": true, "have_arm": false}
docs/status.md 2026-09-29 still says "sleep + 5-min glances"; the docs differ, the code is hourly.
After one boot the camera service was "active" but published no frame for 23 minutes. Every step of the mind
returned at the "no new frame" check, so it could not look for anyone and told nobody. camera_watch(), lines
256-279, is the robot's own fix:
On the robot:
def camera_watch(self, now):
"""2026-10-07: after a boot the camera service ran but published no frame for 23 min; every step returned
silently, so it could not look for the owner and nobody knew. No new frame for CAM_STALE_S -> restart the
camera (its own heal, at most once per CAM_RETRY_S, logged); still blind after a second try -> tell the owner."""
m = self.n.last.get('rgb')
stamp = None if m is None else (m.header.stamp.sec, m.header.stamp.nanosec)
if not hasattr(self, 'cam_t') or stamp != self.cam_stamp:
if hasattr(self, 'cam_t') and getattr(self, 'cam_restarts', 0):
self.event('camera_back', after_restarts=self.cam_restarts)
self.cam_stamp, self.cam_t, self.cam_restarts = stamp, now, 0
return
if now - self.cam_t < CAM_STALE_S or now - getattr(self, 'cam_fix_t', 0.0) < CAM_RETRY_S:
return
self.cam_fix_t = now; self.cam_restarts += 1
self.event('camera_silent', secs=round(now - self.cam_t), restart=self.cam_restarts)
subprocess.run(['sudo', 'systemctl', 'restart', '--no-block', 'rosorin-camera'], capture_output=True)
if self.cam_restarts >= 2:
try:
sys.path.insert(0, os.path.expanduser('~/selfcare'))
from say import enqueue
enqueue('camera_blind', 'My camera is not sending pictures, even after restarting it. I cannot see.',
'urgent', repeat_s=1800)
except Exception as e:
self.event('say_failed', err=str(e)[:100])
No new frame for CAM_STALE_S = 20 s: restart rosorin-camera (at most once every CAM_RETRY_S = 120 s) and log
camera_silent. After two restarts with no frame, put an urgent sentence in the self-care speech queue, which Buddy
speaks (chapter 23). When frames come back it logs camera_back. It runs before the "no new frame" check in step()
so a silent camera cannot hide it.
Not seen working
The watchdog was deployed and read back on 2026-10-07 (deploy.sh mind: services active, mind running, a first
thought). A read of all event logs on the evening of 2026-10-07 found nocamera_silentorcamera_backevent,
so it has not been seen restarting a silent camera.
To find out after a boot:grep -h camera_ ~/vision/mind/*/events.jsonl.
Anything else that needs the arm (a test script, the robot API's /look, the explorer) must ask the mind to let go
first. Two ways:
From a script, the context manager in vision/mind_pause.py. It sets /rosorin_mind/enable to false, waits
(up to 8 s) until the mind reports paused with no arm, and re-enables the mind afterwards only if it was enabled
before. Every arm script in the repo uses it, for example vision/attention.py:
On the robot:
if __name__ == '__main__':
from mind_pause import mind_paused
with mind_paused('attention test'): # the robot mind hands over the arm, gets it back after
main()
By hand, with the service:
On the robot:
source /opt/ros/humble/setup.bash
timeout 15 ros2 service call /rosorin_mind/enable std_srvs/srv/SetBool "{data: false}" | tail -1
timeout 6 ros2 topic echo --once /rosorin_mind/mode --field data
# ... use the arm ...
timeout 10 ros2 service call /rosorin_mind/enable std_srvs/srv/SetBool "{data: true}" | tail -1
Check
The journal showssleep -> paused (disabled)(or fromtrack/curious) and, after re-enabling,
paused -> sleep (resume). While paused the mode message has"mode": "paused","enabled": falseand
"have_arm": false.
Read the robot back after every test
On 2026-10-05 the mind was stopped for a 15-second network test while the claw servo had silently stopped
answering. The restarted mind could not take the arm back (the driver refused while any servo was silent), the
head stayed down for hours, and nobody noticed (docs/lessons.md2026-10-05). After any pause, stop or restart,
check that the mind is back in the mode it was in and holds the arm if it held it before
(scripts/ops/readback.shprints the mode). The driver now carries the claw as optional, and the mind's state
reports silent servos.
scripts/ops/deploy.sh mind copies mind.py, world.py, reappear.py and look_learner.py to ~/vision, checks
that mind.py compiles, restarts the service and runs the read-back 20 s later:
On your laptop:
cd ~/CCode/rosorin-pro && bash scripts/ops/deploy.sh mind
Check
From the camera-watchdog deploy on 2026-10-07:On the robot:
deployed mind; reading back in 20 s services: active active active active active active active mind: {"mode": "sleep", "why": "start", "enabled": true, "have_arm": false}
If it fails
deploy.sh minddoes not copythink.py,attention.py,scan.py,scan_filter.pyormind_pause.py.
If you change one of those,scpit torosorin-wifi:vision/yourself and restart the mind.- The mind hangs if the driver restarts under it: it waits in a driver service call that never answers
(2026-10-05: silent for a minute until restarted; another time it crashed after about 4.5 minutes in the same
wait). Not fixed in the mind.deploy.sh drivertherefore stops the mind, restarts the driver, then starts the
mind. If you restartrosorin-baseby hand, restartrosorin-mindafter it.
To remove the service completely, scripts/rollback_mind_service.sh stops and disables it (the mind releases the arm
on SIGINT) and deletes the unit:
On the robot:
sudo bash ~/setup/rollback_mind_service.sh
The two timers are removed with the commands in their install scripts' headers:
sudo systemctl disable --now rosorin-owner-faces.timer && sudo rm /etc/systemd/system/rosorin-owner-faces.{service,timer} && sudo systemctl daemon-reload,
and the same for rosorin-reappear.
| What | Where |
|---|---|
| Mode changes, thoughts, errors | journalctl -u rosorin-mind |
| Event log (start, mode, heartbeat, track, found, lost, lost_look, world_look, heard, thought, room_scan_done, look_learned, arm_refused, camera_silent, owner_stop, body_skill_..., and more) | ~/vision/mind/<UTC day>/events.jsonl |
| A picture at every acquisition, and a face picture every 5 s while tracking a face (max 500 a day) | ~/vision/mind/<UTC day>/HHMMSS_<what>.jpg, HHMMSS_face.jpg |
| Every thought with its picture | ~/vision/think/<UTC day>/thoughts.jsonl, HHMMSS.jpg |
| Its questions for the owner | ~/vision/questions.jsonl |
| Room scans | ~/vision/roomscans/<date_time>/ (N.jpg, mosaic.jpg, meta.json) |
| Room memory, world model, presence map, taught words | ~/vision/room_memory.json, world.json, presence.json, learned_objects.json |
| Look learner | ~/practice/head_model.json |
| Nightly models (written by the timers) | ~/vision/reappear.json, ~/vision/owner/ |
On 2026-10-07 ~/vision/mind held 610 MB and ~/vision/think 371 MB; nothing deletes old days.
The record is explicit that the mind does not yet do what the objective needs. As of 2026-10-07 (finding numbers from
docs/research/whole_project_review.md, 2026-10-05):
identify was 78 % of 864 thoughts on 2026-10-05 (85 % on 10-01, 78 % onidentify removed from the menu, the same brain choselook_around 12 times out of 40 instead of never (findings 1, 2, 16b). It is still on the menu.~/vision/questions.jsonl, the same one up to 20 times; nothingwhole_project_review.md section 5). Work on a desk is out of view.docs/status.md 2026-10-05; not fixed).have_room_map only checks that a file exists, and on 10-05 that file was a copy of the 09-28 map (finding 41).AGENTS.md rule 8). Chapter 31 says how to run it.Fixed since the review: the claw servo no longer blocks the arm (2026-10-05), the mind knows navigation readiness,
silent servos and arm refusals and can_drive needs navigation (2026-10-06), the world model is back (2026-10-06),
the camera watchdog (2026-10-07), hourly idle wake-ups (2026-10-07).
systemctl is-active rosorin-mind prints active, and /rosorin_mind/mode reads{"mode": "sleep", "why": "start", "enabled": true, "have_arm": false} after a restart with nobody in view.track, and the head follows you.ros2 service call /rosorin_mind/enable std_srvs/srv/SetBool "{data: false}" makes it paused (disabled), andtrue brings it back to sleep (resume).scan_room event produces a new folder in ~/vision/roomscans/ with a mosaic.jpg.systemctl list-timers shows rosorin-owner-faces.timer at 03:00 and rosorin-reappear.timer at 03:20.~/vision/think/<UTC day>/thoughts.jsonl gains lines with a thought and a latency_s.Where this comes from
vision/mind.py(repo commitbfb61d8; lines cited throughout: 31-90 constants, 109-160 init, 192-253 arm
handling, 256-279 camera watchdog, 281-511step(), 513-544self_knowledge, 546-585 wheel skills and stop,
587-622apply_thought, 624-682on_heard, 694-787 room scan, 789-817lost_step, 834-855main),
vision/attention.py(24-47),vision/think.py,vision/world.py,vision/reappear.py,
vision/look_learner.py,vision/owner_faces.py,vision/mind_pause.py,vision/scan.py(113-140);
systemd/rosorin-mind.service,rosorin-reappear.{service,timer},rosorin-owner-faces.{service,timer};
scripts/install_mind_service.sh,rollback_mind_service.sh,install_reappear_timer.sh,
install_owner_faces_timer.sh,scripts/ops/deploy.sh,scripts/explore_run.sh(lines 27-28).
docs/decisions.md2026-09-30 (faces, brain on bigbuddy, nothing in the driving path depends on bigbuddy),
2026-10-01 (head motion, inquisitive look-around, presence), 2026-10-02 (world model, control delegated);
docs/lessons.md2026-09-30, 2026-10-01, 2026-10-05;docs/status.md2026-09-29, 2026-10-05, 2026-10-07 18:59;
docs/capabilities.md;docs/research/whole_project_review.md(findings 1-12, 16, 24-31, 38-42, sections 3 and
5);docs/research/doing_it_itself_reference.mdsection 1; git log ofvision/mind.py(a70d4fe, 06e493b,
60d3a0a, 7140794, 8c4d07a, 2ef8bad). Command logsources/cmdlog_robot.md: 2026-09-30 00:35-00:40 UTC (first
run, service install, glance fix), 01:53 (faces), 05:18-05:19 (brain moved to bigbuddy, first thought),
2026-10-01 19:24 (reappear timer), 2026-10-02 12:33 (look learner), 2026-10-06 10:14 and 17:42-17:44 (self
knowledge, world model restored), 2026-10-07 01:37 (camera watchdog deploy) and 18:59 (hourly change). Live
read-only check of the robot on 2026-10-07 (timers,reappear.json,owner.json, service state, events, log
folder sizes, nocamera_*events).