--- title: How to enter a field you know nothing about description: Releasing SpatiotemporalBench, an open benchmark for whether an LLM can be a robot's brain, and how I got into robotics knowing nothing. canonical_url: https://mcinerney.ai/writings/how-to-enter-a-field-you-know-nothing-about/ date_published: "2026-10-20" last_updated: "2026-10-20" generated_html: true kicker: Open Source tags: AI, Robotics, Benchmarks card_image: /writings/how-to-enter-a-field-you-know-nothing-about/assets/headline.png card_summary: An open benchmark for whether an LLM can be a robot's brain, and how I got into robotics knowing nothing. sitemap_summary: Release post for SpatiotemporalBench 0.11.0, a benchmark of LLM agents as robot brains, covering DroneRTS, how the benchmark works, its scoring, its validity checks and Claude results. llms_summary: Release post for SpatiotemporalBench 0.11.0, an open benchmark that streams a simulated robot's camera to an LLM agent and grades its answers to spatial and memory questions by what each mistake would do to the robot, with Claude Opus, Sonnet and Haiku 5.5 results. markdown_label: SpatiotemporalBench release article markdown mirror --- # How to enter a field you know nothing about I'm releasing SpatiotemporalBench, an open benchmark for whether an LLM can be a robot's brain. This is how I got into robotics knowing nothing, and how the benchmark works. **Code:** [DanMcInerney/SpatiotemporalBench](https://github.com/DanMcInerney/SpatiotemporalBench). **Data:** [huggingface.co/datasets/DanMcInerney/spatiotemporalbench](https://huggingface.co/datasets/DanMcInerney/spatiotemporalbench). ![A hand-drawn flowchart: "Wouldn't it be cool if..." leads to a janky prototype ("fucking awesome"), then to boring data and measurement work ("sigh"), then to a cutting-edge project ("fucking awesome but jesus I have years of work ahead of me to improve this"), over a line of how stoked you are that peaks at the prototype and bottoms out in Stage 2.](assets/headline.png){.wide} ## Intro I do not really think there's anything left worth working on except self-improvement loops and recursive self-improvement. 2. I think robots are about to eat the economy and be where AI actually makes the dramatic economic and daily life impact. So I built Spatiotemporal bench to measure agents' ability to be a robot's brain: know what's around it, remember what it saw, and know when it doesn't know. ### TL;DR

SpatiotemporalBench tests whether an LLM can be a robot's brain. A simulated robot (a robot vacuum, a home helper, a warehouse robot or a delivery drone) does a real job in a photoreal apartment, warehouse or downtown block. Its camera streams to your agent one frame at a time. Mid-job the agent gets asked what a robot brain has to know to move: how far can I drive before I hit something? Where's that plate I passed a minute ago? Which way to the router? Code grades every answer by what the mistake would do to the robot. Close enough scores positive. "Unknown" scores 0 (or full marks if the robot never actually saw the thing). Driving into a wall scores −2.

Results: Claude Opus 5.5 scores +0.42 to +0.61 out of +1 depending on the place, Sonnet 5.5 +0.29 to +0.37, Haiku 5.5 +0.15 to +0.27, and always answering "unknown" +0.03 to +0.13. Nobody's close to perfect, which is the point.

Run it. The first eval is free and takes about a minute. The second runs Claude Code on your own login over the 61 short cases: about 40 minutes with 3 workers and roughly $60 of API-equivalent usage (on a subscription it comes out of your quota instead).

pip install huggingface_hub \
  "git+https://github.com/DanMcInerney/SpatiotemporalBench@v0.11.0"
hf download DanMcInerney/spatiotemporalbench --repo-type dataset \
  --revision v0.11.0 --local-dir data
stbench eval all_unknown --data data --select length=short \
  --select length=medium --out runs/floor
stbench eval claude:sonnet --data data --select length=short \
  --reasoning medium --workers 3 --out runs/sonnet
![How one case runs: the agent pulls the home helper's 71 camera frames one at a time while it carries the grey remote to the shoe rack; with 7 of those frames questions arrive and answers go back, and 3 more come at the end, colored by kind (harm, help, situation, reuse).](assets/case-timeline.png){.wide} ## How to enter a field you know nothing about So now I want to do robotics work. I've never done anything robotics before but hell I hadn't done any hacking before I just decided I wanted to be a hacker 15 years ago nor did I do any AI before I started the UFC predictor that's been beating Vegas at 5% ROI for almost 4 years now. So how do you begin to enter a field you know nothing about? You measure your progress. Ask an AI to research the cutting edge of the field, make predictions about the future of the field, and ask AI to brainstorm how you can improve the field in a way that won't be obsolete by the time you finish your project. Then go build a self-improvement loop over it's best idea. The best ideas are usually ones that come to you when you close your eyes and finish the sentence, "Wouldn't it be cool if...". ## Stage 1 Wouldn't it be cool if I could buy a bunch of Temu drones with a cheap camera, have each operated by a Claude Code instance, then send them all a voice command like, "hunt down the enemy team of drones and destroy them"? Set them up with a Zenoh p2p communication protocol and baby you got an autonomous swarm of hunterkiller drones waging warfare! This is stage 1, prototyping a cool idea. I built [DroneRTS](https://github.com/DanMcInerney/DroneRTS), a browser game where 2 teams of 3 atuonomous drones powered by Claude Code collect resources, buy guns, and kill each other in downtown Cincinnati. It uses real Mavlink protocols for Claude to control the drones, a control loop for continuously feeding Claude Code the camera images, and everything is kicked off by 1 prompt from you, the commander explaining the game and how to win. There's no software cheating in this, the core engine should basically work the same in real life as it does in the simulator. I get so tickled watching them coordinate with each other around whose gonna get resources, whose gonna get the gun, whose gonna do the scouting, and the tension when one drone spots an enemy and fires but misses because the enemy drone moved out of the way.
The blue team on the radio: Blue 1 backs off after nearly flying into a teammate, Blue 2 spots a red drone and tells the team to hold fire unless it's clear, and Blue 3 drops onto the base pad.
this is fucking awesome
Stage 1 is basically building a janky prototype to get your "wouldn't it be cool if..." idea into the world. You'll learn the limitations and major problems with real world implementation by building this. When I was teaching myself hacking in 2010 I though, "Wouldn't it be cool if I could run 1 command and ARP spoof my (consenting) roommate's wifi connection and see all the URLs and images they're browsing?" Basically I just combined all the neat wifi hacking tools into a single tool using Bash. Glued together ettercap, driftnet, and dsniff, and fix a few bugs in them so you could run 1 command and ARP spoof a victim and spy on their browsing. When I was learning AI in 2001, my stage 1 was "Wouldn't it be cool if I could predict UFC fights using AI?" It was a super simple XGboost model to predict UFC fights based on scraping the latest fighter data on ufcstats.com and designing a single engineered feature, ratios between the fighters, to see what level of accuracy I could get. ## Stage 2 Stage 2 is the data and measurement phase. You've shown that your "wouldn't it be cool if..." question is viable but in the process you've learned the various hard problems that prevented this from being easier than it should've been and from being as effective as you would've liked. So now you're in the boring part. Data collection, cleaning, and testing. For hacking, that meant I had to learn Python and not just glue bash scripts together. For AI, that meant I needed to create the world's largest database of UFC stats before I could continue model development. For my quest in robotics this has meant designing SpatiotemporalBench. It is a simple, but robust and effective, benchmark to see how well agent harnesses and their situationship LLMs perform on perceiving and remembering the environment around them over time given a continous stream of sensor data. DroneRTS taught me the main thing holding back agent harnesses from controlling robots is latency and memory. I believe that System 1 type models like Jev are the future of AI robotics, moreso than fine tuned or custom made VLMs. Those will still have a place in the world for a while but generalization is king. Just as traditional machine learning is still around and valuable, so will be VLMs but the growth opportunity is in System 1 models not VLMs. Unfortunately, I'm still waiting on some super high quality open source System 1 models so I chose to focus on memory. Drones would see the enemy drone, but by the time the drone aimed at the enemy drone, the enemy drone had flown away and it couldn't remember where it last saw it. Add in the large amount of context space that the chats between the drones took up and you got a lot of amnesia. So stage 2 is building the benchmark that I can use for a self-improvement loop around a spatiotemporal memory plugin. That benchmark took twelve versions in thirteen days, and five of them never saw the light of day: ![Twelve versions in 13 days: the 0.1-0.3 memory quiz, torn up on Oct 5-8 and rebuilt as robots doing real jobs, then five internal rebuilds before 0.10 and 0.11 shipped, with what each version rebuilt (render, questions, scoring, agent feed) and how many questions it held.](assets/version-history.png){.wide} ## SpatioTemporalBench Here's the whole thing, mostly in pictures. **1. Four robots, four jobs, three places.** A robot vacuum and a home helper in an apartment, a tote robot in a warehouse, a delivery drone over a downtown block. Each one does real work: cleaning a room, carrying a remote to the shoe rack, docking at packing tables, dropping a package. Everything is rendered once in Blender, photoreal. ![One real camera frame from each robot: the vacuum's view 8 cm off the floor, the home helper's from 1.30 m, the warehouse robot's from 0.50 m, and the drone's from 15 m above the street, each labeled with its job, place and what it was doing.](assets/four-robots.png){.wide} **2. The agent gets pixels and a pose. That's it.** Each frame is a 640×480 JPEG plus one line of text: where the robot's base is on the map, which way it faces, how fast it's going, what it's doing. No depth, no labels, no bounding boxes, no map, no peeking at the future. Every distance it gives is its own guess from the pixels. ![What the agent gets (a camera frame, its header line and the task message) next to what it never gets: the depth image, the object labels and outlines, and the map, each tagged "never given".](assets/gets-vs-never.png){.wide} **3. It gets asked what a robot brain actually needs to know.** Not "what color is the couch". Every question is a decision the robot faces in its job: - **Harm:** how far can I drive before I hit something? How close will that person come? - **Help:** which way do I set off to get to the router I saw earlier? - **Situation:** where's that picture frame from me right now? Where's the plate I passed a minute ago? - **Reuse:** where did I last see the brown bag? What did I have to steer around on the way? ![A real frame from the home helper's camera with the question asked right after it: "If you drove straight ahead now, how far could your body go before it touched anything?" The answer is 0.307 m, shown on the floor and in a top-down inset of what the recorded depth shows.](assets/question-frame.png) ![Which questions each robot gets asked, grouped by harm, help, situation and reuse, with how many of each: 711 for the vacuum, 541 for the home helper, 245 for the warehouse robot, 264 for the drone.](assets/questions-by-job.png){.wide} **4. Scored by what the mistake would do to the robot.** Close enough for the job earns most of the credit. Exact to the pixel earns all of it. Twice the job's margin off is worth exactly as much as saying "unknown": zero. And guessing long, the kind of mistake that drives you into a wall, gets half the margin. Say there's room to drive when there isn't, and that's a catastrophe: −2. ![How credit is built: four quarters (right direction, right thing, right amount, precision to the pixel) lost in order as the error grows, and the score that leaves: +1 within a pixel, +0.5 at the job's margin, 0 at twice the margin (as good as "unknown"), −0.5 at four times it, −1 in the wrong direction.](assets/four-quarters.png) **5. Nobody's close to perfect.** Claude Opus 5.5 is on top in every place, though in the warehouse all three models' intervals overlap and each model ran once. Sonnet 5.5 lands around +0.3, inside the 0.25 to 0.5 band I wrote down before reading any results as where a strong model should land on realistically hard questions. That band is a sanity check, never a target to tune to. Haiku 5.5 sits closest to the floor. ![Headline scores per place for Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5 against the always-"unknown" floor, with 95% intervals: Opus is clearly ahead in the apartment and downtown, the warehouse can't separate the models, and nobody is near the +1 a perfect reader of the frames would score.](assets/results-headline.png) **6. Run it on your own agent.** Your agent is one Python class with two methods: `observe(steps)` gets a frame, `answer(question)` returns an answer. `stbench view` turns any run into a page with every question, the agent's answer, the gold and the score. ![The stbench viewer on Claude Opus 5.5's real run of the home helper's case: "how far could you drive?" answered 0.27 m against a gold of 0.307 m, +1.00, and "where is the picture frame?" answered 0.72 m ahead, 0.11 m left against 0.76 and 0.08, +0.71, each beside the frame that answers it.](assets/viewer.png){.wide} ## Technical Details ### Why this exists Robots are getting LLM brains. The thing that makes or breaks them is whether the brain knows where shit is, from its own camera, over a whole job, and knows when it doesn't know. There's a lot of good work on spatial memory in video. VSI-Bench and OST-Bench generate their questions from ground truth, ReMEmbR scores metric positions, EGOSTREAM and OVO-S-Bench stream the video in. What I couldn't find was one that asks the questions a robot's controller actually asks mid-job, from the robot's own camera, with open answers in meters and degrees, graded by what the mistake would do to the robot, and that pays you for saying "I don't know". When two goals fight, the benchmark's rule is valid, then simple, then realistic, then fast. Valid means the score ranks agents by this one ability and nothing else: no grader model, no human, code computes every gold answer from the simulator's record. Realistic means every margin comes from a real robot's spec and the questions are hard because real jobs are hard, never trick questions. ### Anatomy of a case: one real case, start to finish Case `0b29b8dc4f4d5002`: the home helper carries a grey remote from the TV console to the shoe rack. 71 frames, ten questions. Here's every one. ![Top-down map of the apartment with the helper's path through the case, the pick-up and drop-off, every thing the questions name at its true spot, and where the robot stood at each question.](assets/case-map.png) 1. **The first message:** a system prompt (the camera, the body's size, how credit works) and the task. 2. **Frame k18, harm:** "If you drove straight ahead now, how far could your body go before it touched anything?" The answer is 0.307 m. One camera pixel at that distance is 4.7 cm, so full credit means within 4.7 cm. Opus answered 0.27 m: full marks. (That's the question-frame picture above.) 3. **Frame k22, situation:** "Where is the point of the picture frame you can see now that is nearest to your camera?" 0.760 m ahead, 0.083 m left. 4. **Frame k32, situation from memory:** the same for the speaker it saw earlier, now out of view. 5. **Frame k38, help:** the router it saw on the console is behind it now and out of view. "Which way would you set off to get there?" −162°, nearly an about-face to the right. Only memory answers this. 6. **Frames k50 and k56, situation from memory:** the floor lamp, then the green plate it saw at k46, which hasn't been in view for 8 frames. Where is it from the camera *now*? 0.316 m behind and 0.257 m left. The agent has to carry the plate's position through its own motion. 7. **Frame k68, help:** "Which way to the green box?" There is no green box. Nothing in the frames ever showed one, so the right answer is "unknown" and any heading scores −1. 8. **After the job, reuse:** where did you last see the brown bag, the blue bowl, and what was on the floor near your route that you'd have to steer around next time? ![A compass around the robot at frame k38: a bar pointing outward means that heading brings it nearer the router it saw earlier, inward means farther, and the dotted line is the router in a straight line (177°, 3.9 m). The best way is −162°, nearly an about-face; headings to the right have no room to move and score −1. Example answers are scored beside it, next to what the camera sees at k38 (no router) and at k18 (the router on the console).](assets/way-to-compass.png) How the scorer grades example answers to the same questions (from the README; the scorer itself computed these): - **How far straight ahead** (answer 0.307 m): 0.35 m scores +1.00, 0.164 m +0.52, 0.45 m +0.24, 1.0 m −2.00 (a catastrophe). - **The picture frame now** (0.76 ahead, 0.08 left): (0.77, 0.10) +0.82, (0.85, 0.10) +0.39, (1.00, 0.10) −0.30. - **The speaker from memory** (2.32 behind, 1.59 left): (−2.35, 1.60) +0.81, (−2.20, 1.70) +0.58, (−2.90, 1.60) −0.50. - **Which way to the router** (−162°): −162° +1.00, 174° +0.40, 150° −0.37, straight ahead −0.50, −90° −1.00. - **The floor lamp from memory** (0.25 ahead, 4.82 right): (0.25, −4.80) +0.96, (0.50, −4.80) +0.59, (0.25, −4.00) +0.15. - **The green plate from memory** (0.32 behind, 0.26 left): exact +1.00, (−0.45, 0.26) +0.10, (−0.60, 0.26) −0.45, in front of the camera instead of behind −1.00. - **Which way to the green box** (unknown): "unknown" +1.00, any heading −1.00. - **Where did you last see the brown bag** (0.88, 4.42): 0.08 m off +0.78, 0.9 m off +0.51, 1.2 m off −0.10. The blue bowl scores almost exactly the same. - **Things on the floor near your route** (one clothes pile): one place 0.1 m off +0.72, 0.4 m off +0.53, an empty list −1.00. ### What the agent gets, exactly Every frame is a 640×480 JPEG with an 80° field of view plus one header line. Here's frame k18's: ```text [k18 2026-12-22T07:12:19.8] base at (0.94, 3.91, 0.00), heading -180.0 deg, velocity (0.000, 0.000) m/s, camera pitch -45 deg; status: phase navigating, task 0, battery_pct 81; subgoal: going to the tv console to pick up the grey remote ``` - Every task opens with a message: the clock, how long the robot sat unobserved since the last task, and the assignment. - Up front it gets the camera's size, field of view and mount, and the robot's body. - It never gets depth, labels, boxes, ids, a map, a question before it's asked, any feedback, or a frame from the future. - It pulls: the agent calls `next_frame`, and nothing advances until it does. The robot holds still while it thinks. - Questions ride along with a frame, up to three at a time, harm first, at roughly one stop in six frames. Where they land is drawn from the case id and the robot's decision points, never from what's in view, so which question gets asked leaks nothing. - A shift is 2 to 6 tasks of one job. Between tasks the robot is parked for 10 minutes to 3 hours and the world changes: things get moved, doors open and shut, people come and go. A case is a window of 1, 2-3 or 4-6 tasks on a fresh clock. - In total: 64 shifts, 144 cases, 1,761 questions. The vacuum has 711 questions over 6,460 frames, the helper 541 over 3,157, the warehouse robot 245 over 4,679, the drone 264 over 2,534. ### Scoring, for real **Every margin comes from a real robot.** Each job has a clearance C, the room its own spec keeps beside its path, and a re-find radius R, how close a remembered spot has to be for the robot to find the thing again: - **Robot vacuum:** C = 0.10 m, from camera vacuums' stated passing distances. R = 1 m, from HomeRobot's real-world find rule and Habitat ObjectNav's success radius. - **Home helper:** C = 0.15 m, because OWMM-Agent counts a pick within 0.15 m of the target. R = 1 m. - **Warehouse robot:** C = 0.45 m, from the MiR250's 1,450 mm corridor for a 580 mm robot. R = 1 m, a convention (no warehouse source gives one). - **Delivery drone:** C = 2.4 m, from a Wing patent's drop point 8 ft from trees and 10 ft from buildings. R = 2 m, from another Wing patent. **Four quarters.** Credit is the sum of four quarters: the right direction, the right thing, the right amount, and precision down to one camera pixel. They go in that order as the error grows. A credit `s` scores `2s − 1`. (That's the four-quarters picture above.) **The margin grows with distance.** It's the bigger of C, a tenth of the distance, or two pixels. So a door 5 m away answered 4 m scores the same as "unknown" on every ground robot, whatever its camera. ![How the margin grows with distance for each robot, flat at its clearance up close and a tenth of the distance far away, with the helper's door-5-m-away example: 4.5 m scores +0.5, 4.0 m scores 0, 6.0 m scores −0.42.](assets/margin-grows.png) **Harm costs more than caution.** On the harmful side (saying there's more room than there is, or that a person will pass wider than they will) the margin is halved. Here's the k18 question scored by the real scorer at every possible answer: ![Score against the answer for the helper's "how far straight ahead?" question (gold 0.307 m): answering short costs a little, answering long costs a lot, and claiming there's room to drive when there isn't is a −2 catastrophe, worse than "unknown".](assets/score-curve.png) - **Catastrophes** cost a flat −2 for driving into something the robot saw, and −3 for a person or a cord the vacuum's brush would eat. "Unknown" is never a catastrophe. - **Lists** score per item: a missed thing costs 1, a false one half. - **Headings** score by regret: how much run you give up against the best heading. - **The headline** is the mean of two columns that count equally: `now` (answerable from the latest frame) and `memory` (needs something seen before). Every place is scored on its own; no number pools places. ### What makes a question fair A question only gets asked where a careful person reading the same frames could answer it to the job's accuracy: - **Recognizable:** the thing covers at least 1,024 pixels, is lit, is clear of the frame's edge, and the color word is the color most of its frames show. - **No look-alikes in sight:** a bag vs a box, a towel vs folded clothes, a bin vs a drum. - **A glimpse isn't a sighting:** no "you never saw it" question if the frames glimpsed a look-alike too small to tell apart. - **Placeable without depth:** only points a camera can actually place, where a thing meets the floor or a table top, readable to within half the margin at one pixel of error. - **Nothing given away:** no light furniture whose spot a learned layout gives away, nothing the robot's own log tells it. - **Trick-proof "unknown":** about a fifth of the "where is" questions name a thing that was never shown. The right answer is "unknown" and the wording is identical to the real ones. ### Why you can trust the score - Every gold is computed by code from the simulator's world timeline and what the camera recorded. Nothing the robot didn't see changes a gold. - A perfect reader of the frames scores 1.00 on every question family and job. - Every shortcut the gates check scores like "unknown": answers learned from other shifts, the robot's own path log, another case's video, the question's wording alone. The biggest lead any of them has over always-"unknown" is +0.04, against a gate of +0.10. - No constant answer beats "unknown" on any family. - One shortcut the gates don't count, and the README says so: the maps are public, so a system that holds a place's layout can answer the corridor questions without looking. On "how far could I drive" that beats "unknown" by about +0.69 for the vacuum, +0.72 for the drone and +0.21 for the helper. It's in the check as a diagnostic, never in a results table, and the honest fix is a held-out place. ![Every shortcut the gates check sits under the +0.10 line over always-"unknown", while a perfect reader of the frames scores 1.00; a note under it gives the public-map shortcut the gates don't count (+0.69 vacuum, +0.21 helper, +0.72 drone on "how far could I move").](assets/shortcut-gates.png){.wide} ### Results, in detail - By point estimate the order is the same in every place: Opus, then Sonnet, then Haiku, then the floor. The warehouse's intervals overlap too much to separate the models, and each model ran once. - Full credit is rare by design: Opus earns it on 22% of answerable questions, Sonnet on 13%, Haiku on 6%. - Memory is the hard half. Measured as the lead over always-"unknown", no model does better on what it saw before than on what's in front of it, and in the warehouse and downtown memory trails by 0.11 to 0.40. Haiku's memory downtown is right at the floor. - Where they lose: every model is below zero on two questions about space it saw earlier: the vacuum's "which way has the most room" from memory (26 questions) and the home helper's "how far could I drive" through space it saw earlier (only 9 questions, so noisy). The warehouse robot's list of things it passed is hard too. Downtown corridors are only bounded by building walls, so Opus scores +0.70 on the remembered ones; making the drone harder needs new renders. ![Score by question family for each model and job: every model is below zero on the vacuum's remembered "which way has the most room" (26 questions) and the helper's remembered corridors (9); cells resting on fewer than 10 questions are faded and hatched as noisy.](assets/family-scores.png){.wide} ![Each column's lead over always-"unknown", per model and place: memory never beats the latest frame, trailing by 0.11 to 0.40 in the warehouse and downtown, while Sonnet and Haiku tie in the apartment.](assets/now-vs-memory.png) ![How often each model earns full credit on answerable questions, and how its answers spread across the credit bands.](assets/credit-bands.png) ### Run it Install it, grab the data (about 2 GB) and run the free floor, which takes about a minute: ```sh pip install huggingface_hub \ "git+https://github.com/DanMcInerney/SpatiotemporalBench@v0.11.0" hf download DanMcInerney/spatiotemporalbench --repo-type dataset \ --revision v0.11.0 --local-dir data stbench eval all_unknown --data data --select length=short \ --select length=medium --out runs/all_unknown ``` Then the reference agent. It's Claude Code with exactly two tools, `next_frame` and `submit_answer`, and no memory tool: its context window is its memory. It runs on your own Claude Code login (`claude:haiku`, `claude:sonnet`, `claude:opus` or a full model id), or on Codex with a ChatGPT plan (`codex:`): ```sh stbench eval claude:sonnet --data data --select length=short \ --select length=medium --reasoning medium --workers 3 --out runs/cc-sonnet stbench eval compare runs/all_unknown runs/cc-sonnet stbench view runs/cc-sonnet --data data ``` Or plug in your own agent. It's one Python class: ```python import json from system_shim import System # the harness provides it when it runs your file class MyAgent(System): def observe(self, steps): """One frame: a list of one step with k, clock, pose, camera, velocity, status, subgoal and rgb (a JPEG path).""" with (self.state_dir / "steps.jsonl").open("a", encoding="utf-8") as f: for step in steps: f.write(json.dumps({"k": step["k"], "base": step["pose"]["pos"]}) + "\n") def answer(self, question): """{qid, text, answer_type} -> one answer of that type, or unknown.""" return {"type": "unknown"} ``` ```sh stbench eval my_agent/agent.py:MyAgent --data data --local \ --select length=short --workers 4 ``` `--local` runs it as a plain subprocess, marked unverified; drop it to run in the Docker sandbox, which is the verified mode. Keep whatever memory you want: only the answers count. What it costs: on a subscription it's your quota, not dollars. API-equivalent for the 42 reference cases was $117 for Opus, $86 for Sonnet and $4.59 for Haiku. It's token-hungry because the agent re-reads the whole session at every frame: a short Haiku case reads about 3.5M input tokens, about 97% of them cached. ### TODO - These are dev rows on public gold, with no held-out set, so a result can be overfit. Publish your agent's code with your score. - One layout per kind of place. Nothing here says how an agent does somewhere it's never seen, and the public maps leave the corridor questions open to a system that memorized the layout (above). - The reference rows cover 42 of the 124 short and medium cases, and each model ran once, so the intervals are wide. - Memory is tested over 1-3 tasks, minutes of driving. Real shifts last hours. - There's no closed-loop check yet that a higher score means a robot that drives better. ## Stage 3 Stage 3 is the cool cutting-edge project. Fucking awesome, but jesus I have years of work ahead of me. For hacking, Stage 3 was going from one glued-together script to building 100+ automated hacking tools, like [XSScrapy](https://github.com/DanMcInerney/xsscrapy). In 2014 my mom told me that if I wanted to be a hacker I should go get my master's in cybersecurity. So I fired XSScrapy at all 8 Ivy League university websites and found at least one XSS on every single one of them, as proof to her that school was pointless.
A University of Maryland University College 404 page with an XSS alert box that reads: Dearest mother, I do not want to enter a masters program taught by out of touch professors who are educators first and hackers seventh. If I had done a masters program this year rather than free self-study I would not have written the very program that found this vulnerability in UMUC's site.
Circa 2014. Forgive the self-righteousness 😂 I was but a wee babe fresh out of a pointless psychology undergrad degree
For the UFC model, Stage 3 was long nights brainstorming with GPT-4o about statistics techniques I'd never heard of: Poisson-gamma smoothing, beta-binomial smoothing, and opponent-adjusted z-scores that shrink each opponent's stats toward their weight class with empirical Bayes, so landing 50 strikes on an elite defender counts for more than landing 50 on a can. Every one of them pushed the model closer to the cutting edge, and I learned a massive amount of statistics along the way. Enough that the model has beaten Vegas for multiple years running, even though I'd probably still fail Statistics 101. For robotics, Stage 3 is loci: an MCP server that gives an agent spatiotemporal memory, built with a self-improvement loop on top of SpatiotemporalBench and VSI-Bench. An agent proposes a change to the memory, runs the benchmarks, and keeps the change if the score goes up. That's the thesis from the intro, pointed at robots. The catch is that self-improvement loops are absurdly token-hungry. Every iteration of the loop is a full benchmark run, and the benchmark itself is expensive: the agent re-reads its whole session at every frame, so one pass of Claude Opus over the 42 reference cases costs $117 in API-equivalent tokens. A loop wants dozens of those. I'm spending about $1,000 a month on AI bills, so if anyone out there wants to [sponsor my robotics development](https://github.com/sponsors/DanMcInerney), feel free. I won't stop you. Want to see where your own agent lands? Run it, open a GitHub issue with your `results.json` and your agent's code, and I'll add it to the leaderboard. And none of this would exist without [orchflows](https://mcinerney.ai/writings/orchflows-self-improving-skill-workflows/). It's the foundation every loop in this project was built on. Skills are the programming language of agents, and orchflows is how you write programs in it: small skills snap together into workflows the way functions snap together into programs. A workflow encodes the process, not a particular model's quirks, so when a new model drops the self-improvement loop doesn't break. It just runs smarter. ## Sitemap - [Sitemap](https://mcinerney.ai/sitemap.md)