How to enter a field you know nothing about

Code: DanMcInerney/SpatiotemporalBench.

Data: huggingface.co/datasets/DanMcInerney/spatiotemporalbench.

Intro

I do not really think there's anything left worth working on except self-improvement loops and recursive self-improvement. 2. I think robots are about to eat the economy and be where AI actually makes the dramatic economic and daily life impact. So I built Spatiotemporal bench to measure agents' ability to be a robot's brain: know what's around it, remember what it saw, and know when it doesn't know.

TL;DR

pip install huggingface_hub \
  "git+https://github.com/DanMcInerney/[email protected]"
hf download DanMcInerney/spatiotemporalbench --repo-type dataset \
  --revision v0.11.0 --local-dir data
stbench eval all_unknown --data data --select length=short \
  --select length=medium --out runs/floor
stbench eval claude:sonnet --data data --select length=short \
  --reasoning medium --workers 3 --out runs/sonnet

How to enter a field you know nothing about

So now I want to do robotics work. I've never done anything robotics before but hell I hadn't done any hacking before I just decided I wanted to be a hacker 15 years ago nor did I do any AI before I started the UFC predictor that's been beating Vegas at 5% ROI for almost 4 years now. So how do you begin to enter a field you know nothing about? You measure your progress. Ask an AI to research the cutting edge of the field, make predictions about the future of the field, and ask AI to brainstorm how you can improve the field in a way that won't be obsolete by the time you finish your project. Then go build a self-improvement loop over it's best idea. The best ideas are usually ones that come to you when you close your eyes and finish the sentence, "Wouldn't it be cool if...".

Stage 1

Wouldn't it be cool if I could buy a bunch of Temu drones with a cheap camera, have each operated by a Claude Code instance, then send them all a voice command like, "hunt down the enemy team of drones and destroy them"? Set them up with a Zenoh p2p communication protocol and baby you got an autonomous swarm of hunterkiller drones waging warfare!

This is stage 1, prototyping a cool idea. I built DroneRTS, a browser game where 2 teams of 3 atuonomous drones powered by Claude Code collect resources, buy guns, and kill each other in downtown Cincinnati. It uses real Mavlink protocols for Claude to control the drones, a control loop for continuously feeding Claude Code the camera images, and everything is kicked off by 1 prompt from you, the commander explaining the game and how to win. There's no software cheating in this, the core engine should basically work the same in real life as it does in the simulator. I get so tickled watching them coordinate with each other around whose gonna get resources, whose gonna get the gun, whose gonna do the scouting, and the tension when one drone spots an enemy and fires but misses because the enemy drone moved out of the way.

Stage 1 is basically building a janky prototype to get your "wouldn't it be cool if..." idea into the world. You'll learn the limitations and major problems with real world implementation by building this.

When I was teaching myself hacking in 2010 I though, "Wouldn't it be cool if I could run 1 command and ARP spoof my (consenting) roommate's wifi connection and see all the URLs and images they're browsing?" Basically I just combined all the neat wifi hacking tools into a single tool using Bash. Glued together ettercap, driftnet, and dsniff, and fix a few bugs in them so you could run 1 command and ARP spoof a victim and spy on their browsing. When I was learning AI in 2001, my stage 1 was "Wouldn't it be cool if I could predict UFC fights using AI?" It was a super simple XGboost model to predict UFC fights based on scraping the latest fighter data on ufcstats.com and designing a single engineered feature, ratios between the fighters, to see what level of accuracy I could get.

Stage 2

Stage 2 is the data and measurement phase. You've shown that your "wouldn't it be cool if..." question is viable but in the process you've learned the various hard problems that prevented this from being easier than it should've been and from being as effective as you would've liked. So now you're in the boring part. Data collection, cleaning, and testing.

For hacking, that meant I had to learn Python and not just glue bash scripts together. For AI, that meant I needed to create the world's largest database of UFC stats before I could continue model development. For my quest in robotics this has meant designing SpatiotemporalBench. It is a simple, but robust and effective, benchmark to see how well agent harnesses and their situationship LLMs perform on perceiving and remembering the environment around them over time given a continous stream of sensor data.

DroneRTS taught me the main thing holding back agent harnesses from controlling robots is latency and memory. I believe that System 1 type models like Jev are the future of AI robotics, moreso than fine tuned or custom made VLMs. Those will still have a place in the world for a while but generalization is king. Just as traditional machine learning is still around and valuable, so will be VLMs but the growth opportunity is in System 1 models not VLMs. Unfortunately, I'm still waiting on some super high quality open source System 1 models so I chose to focus on memory. Drones would see the enemy drone, but by the time the drone aimed at the enemy drone, the enemy drone had flown away and it couldn't remember where it last saw it. Add in the large amount of context space that the chats between the drones took up and you got a lot of amnesia. So stage 2 is building the benchmark that I can use for a self-improvement loop around a spatiotemporal memory plugin.

That benchmark took twelve versions in thirteen days, and five of them never saw the light of day:

SpatioTemporalBench

Here's the whole thing, mostly in pictures.

1. Four robots, four jobs, three places. A robot vacuum and a home helper in an apartment, a tote robot in a warehouse, a delivery drone over a downtown block. Each one does real work: cleaning a room, carrying a remote to the shoe rack, docking at packing tables, dropping a package. Everything is rendered once in Blender, photoreal.

2. The agent gets pixels and a pose. That's it. Each frame is a 640×480 JPEG plus one line of text: where the robot's base is on the map, which way it faces, how fast it's going, what it's doing. No depth, no labels, no bounding boxes, no map, no peeking at the future. Every distance it gives is its own guess from the pixels.

3. It gets asked what a robot brain actually needs to know. Not "what color is the couch". Every question is a decision the robot faces in its job:

• Harm: how far can I drive before I hit something? How close will that person come?
• Help: which way do I set off to get to the router I saw earlier?
• Situation: where's that picture frame from me right now? Where's the plate I passed a minute ago?
• Reuse: where did I last see the brown bag? What did I have to steer around on the way?

4. Scored by what the mistake would do to the robot. Close enough for the job earns most of the credit. Exact to the pixel earns all of it. Twice the job's margin off is worth exactly as much as saying "unknown": zero. And guessing long, the kind of mistake that drives you into a wall, gets half the margin. Say there's room to drive when there isn't, and that's a catastrophe: −2.

5. Nobody's close to perfect. Claude Opus 5.5 is on top in every place, though in the warehouse all three models' intervals overlap and each model ran once. Sonnet 5.5 lands around +0.3, inside the 0.25 to 0.5 band I wrote down before reading any results as where a strong model should land on realistically hard questions. That band is a sanity check, never a target to tune to. Haiku 5.5 sits closest to the floor.

6. Run it on your own agent. Your agent is one Python class with two methods: observe(steps) gets a frame, answer(question) returns an answer. stbench view turns any run into a page with every question, the agent's answer, the gold and the score.

Technical Details

Why this exists

Robots are getting LLM brains. The thing that makes or breaks them is whether the brain knows where shit is, from its own camera, over a whole job, and knows when it doesn't know.

There's a lot of good work on spatial memory in video. VSI-Bench and OST-Bench generate their questions from ground truth, ReMEmbR scores metric positions, EGOSTREAM and OVO-S-Bench stream the video in. What I couldn't find was one that asks the questions a robot's controller actually asks mid-job, from the robot's own camera, with open answers in meters and degrees, graded by what the mistake would do to the robot, and that pays you for saying "I don't know".

When two goals fight, the benchmark's rule is valid, then simple, then realistic, then fast. Valid means the score ranks agents by this one ability and nothing else: no grader model, no human, code computes every gold answer from the simulator's record. Realistic means every margin comes from a real robot's spec and the questions are hard because real jobs are hard, never trick questions.

Anatomy of a case: one real case, start to finish

Case 0b29b8dc4f4d5002: the home helper carries a grey remote from the TV console to the shoe rack. 71 frames, ten questions. Here's every one.

1. The first message: a system prompt (the camera, the body's size, how credit works) and the task.
2. Frame k18, harm: "If you drove straight ahead now, how far could your body go before it touched anything?" The answer is 0.307 m. One camera pixel at that distance is 4.7 cm, so full credit means within 4.7 cm. Opus answered 0.27 m: full marks. (That's the question-frame picture above.)
3. Frame k22, situation: "Where is the point of the picture frame you can see now that is nearest to your camera?" 0.760 m ahead, 0.083 m left.
4. Frame k32, situation from memory: the same for the speaker it saw earlier, now out of view.
5. Frame k38, help: the router it saw on the console is behind it now and out of view. "Which way would you set off to get there?" −162°, nearly an about-face to the right. Only memory answers this.
6. Frames k50 and k56, situation from memory: the floor lamp, then the green plate it saw at k46, which hasn't been in view for 8 frames. Where is it from the camera *now*? 0.316 m behind and 0.257 m left. The agent has to carry the plate's position through its own motion.
7. Frame k68, help: "Which way to the green box?" There is no green box. Nothing in the frames ever showed one, so the right answer is "unknown" and any heading scores −1.
8. After the job, reuse: where did you last see the brown bag, the blue bowl, and what was on the floor near your route that you'd have to steer around next time?

How the scorer grades example answers to the same questions (from the README; the scorer itself computed these):

• How far straight ahead (answer 0.307 m): 0.35 m scores +1.00, 0.164 m +0.52, 0.45 m +0.24, 1.0 m −2.00 (a catastrophe).
• The picture frame now (0.76 ahead, 0.08 left): (0.77, 0.10) +0.82, (0.85, 0.10) +0.39, (1.00, 0.10) −0.30.
• The speaker from memory (2.32 behind, 1.59 left): (−2.35, 1.60) +0.81, (−2.20, 1.70) +0.58, (−2.90, 1.60) −0.50.
• Which way to the router (−162°): −162° +1.00, 174° +0.40, 150° −0.37, straight ahead −0.50, −90° −1.00.
• The floor lamp from memory (0.25 ahead, 4.82 right): (0.25, −4.80) +0.96, (0.50, −4.80) +0.59, (0.25, −4.00) +0.15.
• The green plate from memory (0.32 behind, 0.26 left): exact +1.00, (−0.45, 0.26) +0.10, (−0.60, 0.26) −0.45, in front of the camera instead of behind −1.00.
• Which way to the green box (unknown): "unknown" +1.00, any heading −1.00.
• Where did you last see the brown bag (0.88, 4.42): 0.08 m off +0.78, 0.9 m off +0.51, 1.2 m off −0.10. The blue bowl scores almost exactly the same.
• Things on the floor near your route (one clothes pile): one place 0.1 m off +0.72, 0.4 m off +0.53, an empty list −1.00.

What the agent gets, exactly

Every frame is a 640×480 JPEG with an 80° field of view plus one header line. Here's frame k18's:

[k18 2026-12-22T07:12:19.8] base at (0.94, 3.91, 0.00), heading -180.0 deg, velocity (0.000, 0.000) m/s, camera pitch -45 deg; status: phase navigating, task 0, battery_pct 81; subgoal: going to the tv console to pick up the grey remote

• Every task opens with a message: the clock, how long the robot sat unobserved since the last task, and the assignment.
• Up front it gets the camera's size, field of view and mount, and the robot's body.
• It never gets depth, labels, boxes, ids, a map, a question before it's asked, any feedback, or a frame from the future.
• It pulls: the agent calls next_frame, and nothing advances until it does. The robot holds still while it thinks.
• Questions ride along with a frame, up to three at a time, harm first, at roughly one stop in six frames. Where they land is drawn from the case id and the robot's decision points, never from what's in view, so which question gets asked leaks nothing.
• A shift is 2 to 6 tasks of one job. Between tasks the robot is parked for 10 minutes to 3 hours and the world changes: things get moved, doors open and shut, people come and go. A case is a window of 1, 2-3 or 4-6 tasks on a fresh clock.
• In total: 64 shifts, 144 cases, 1,761 questions. The vacuum has 711 questions over 6,460 frames, the helper 541 over 3,157, the warehouse robot 245 over 4,679, the drone 264 over 2,534.

Scoring, for real

Every margin comes from a real robot. Each job has a clearance C, the room its own spec keeps beside its path, and a re-find radius R, how close a remembered spot has to be for the robot to find the thing again:

• Robot vacuum: C = 0.10 m, from camera vacuums' stated passing distances. R = 1 m, from HomeRobot's real-world find rule and Habitat ObjectNav's success radius.
• Home helper: C = 0.15 m, because OWMM-Agent counts a pick within 0.15 m of the target. R = 1 m.
• Warehouse robot: C = 0.45 m, from the MiR250's 1,450 mm corridor for a 580 mm robot. R = 1 m, a convention (no warehouse source gives one).
• Delivery drone: C = 2.4 m, from a Wing patent's drop point 8 ft from trees and 10 ft from buildings. R = 2 m, from another Wing patent.

Four quarters. Credit is the sum of four quarters: the right direction, the right thing, the right amount, and precision down to one camera pixel. They go in that order as the error grows. A credit s scores 2s − 1. (That's the four-quarters picture above.)

The margin grows with distance. It's the bigger of C, a tenth of the distance, or two pixels. So a door 5 m away answered 4 m scores the same as "unknown" on every ground robot, whatever its camera.

Harm costs more than caution. On the harmful side (saying there's more room than there is, or that a person will pass wider than they will) the margin is halved. Here's the k18 question scored by the real scorer at every possible answer:

• Catastrophes cost a flat −2 for driving into something the robot saw, and −3 for a person or a cord the vacuum's brush would eat. "Unknown" is never a catastrophe.
• Lists score per item: a missed thing costs 1, a false one half.
• Headings score by regret: how much run you give up against the best heading.
• The headline is the mean of two columns that count equally: now (answerable from the latest frame) and memory (needs something seen before). Every place is scored on its own; no number pools places.

What makes a question fair

A question only gets asked where a careful person reading the same frames could answer it to the job's accuracy:

• Recognizable: the thing covers at least 1,024 pixels, is lit, is clear of the frame's edge, and the color word is the color most of its frames show.
• No look-alikes in sight: a bag vs a box, a towel vs folded clothes, a bin vs a drum.
• A glimpse isn't a sighting: no "you never saw it" question if the frames glimpsed a look-alike too small to tell apart.
• Placeable without depth: only points a camera can actually place, where a thing meets the floor or a table top, readable to within half the margin at one pixel of error.
• Nothing given away: no light furniture whose spot a learned layout gives away, nothing the robot's own log tells it.
• Trick-proof "unknown": about a fifth of the "where is" questions name a thing that was never shown. The right answer is "unknown" and the wording is identical to the real ones.

Why you can trust the score

• Every gold is computed by code from the simulator's world timeline and what the camera recorded. Nothing the robot didn't see changes a gold.
• A perfect reader of the frames scores 1.00 on every question family and job.
• Every shortcut the gates check scores like "unknown": answers learned from other shifts, the robot's own path log, another case's video, the question's wording alone. The biggest lead any of them has over always-"unknown" is +0.04, against a gate of +0.10.
• No constant answer beats "unknown" on any family.
• One shortcut the gates don't count, and the README says so: the maps are public, so a system that holds a place's layout can answer the corridor questions without looking. On "how far could I drive" that beats "unknown" by about +0.69 for the vacuum, +0.72 for the drone and +0.21 for the helper. It's in the check as a diagnostic, never in a results table, and the honest fix is a held-out place.

Results, in detail

• By point estimate the order is the same in every place: Opus, then Sonnet, then Haiku, then the floor. The warehouse's intervals overlap too much to separate the models, and each model ran once.
• Full credit is rare by design: Opus earns it on 22% of answerable questions, Sonnet on 13%, Haiku on 6%.
• Memory is the hard half. Measured as the lead over always-"unknown", no model does better on what it saw before than on what's in front of it, and in the warehouse and downtown memory trails by 0.11 to 0.40. Haiku's memory downtown is right at the floor.
• Where they lose: every model is below zero on two questions about space it saw earlier: the vacuum's "which way has the most room" from memory (26 questions) and the home helper's "how far could I drive" through space it saw earlier (only 9 questions, so noisy). The warehouse robot's list of things it passed is hard too. Downtown corridors are only bounded by building walls, so Opus scores +0.70 on the remembered ones; making the drone harder needs new renders.

Run it

Install it, grab the data (about 2 GB) and run the free floor, which takes about a minute:

pip install huggingface_hub \
  "git+https://github.com/DanMcInerney/[email protected]"
hf download DanMcInerney/spatiotemporalbench --repo-type dataset \
  --revision v0.11.0 --local-dir data
stbench eval all_unknown --data data --select length=short \
  --select length=medium --out runs/all_unknown

Then the reference agent. It's Claude Code with exactly two tools, next_frame and submit_answer, and no memory tool: its context window is its memory. It runs on your own Claude Code login (claude:haiku, claude:sonnet, claude:opus or a full model id), or on Codex with a ChatGPT plan (codex:<model>):

stbench eval claude:sonnet --data data --select length=short \
  --select length=medium --reasoning medium --workers 3 --out runs/cc-sonnet
stbench eval compare runs/all_unknown runs/cc-sonnet
stbench view runs/cc-sonnet --data data

Or plug in your own agent. It's one Python class:

import json

from system_shim import System  # the harness provides it when it runs your file

class MyAgent(System):
    def observe(self, steps):
        """One frame: a list of one step with k, clock, pose, camera, velocity, status, subgoal and rgb (a JPEG path)."""
        with (self.state_dir / "steps.jsonl").open("a", encoding="utf-8") as f:
            for step in steps:
                f.write(json.dumps({"k": step["k"], "base": step["pose"]["pos"]}) + "\n")

    def answer(self, question):
        """{qid, text, answer_type} -> one answer of that type, or unknown."""
        return {"type": "unknown"}

stbench eval my_agent/agent.py:MyAgent --data data --local \
  --select length=short --workers 4

--local runs it as a plain subprocess, marked unverified; drop it to run in the Docker sandbox, which is the verified mode. Keep whatever memory you want: only the answers count.

What it costs: on a subscription it's your quota, not dollars. API-equivalent for the 42 reference cases was $117 for Opus, $86 for Sonnet and $4.59 for Haiku. It's token-hungry because the agent re-reads the whole session at every frame: a short Haiku case reads about 3.5M input tokens, about 97% of them cached.

TODO

• These are dev rows on public gold, with no held-out set, so a result can be overfit. Publish your agent's code with your score.
• One layout per kind of place. Nothing here says how an agent does somewhere it's never seen, and the public maps leave the corridor questions open to a system that memorized the layout (above).
• The reference rows cover 42 of the 124 short and medium cases, and each model ran once, so the intervals are wide.
• Memory is tested over 1-3 tasks, minutes of driving. Real shifts last hours.
• There's no closed-loop check yet that a higher score means a robot that drives better.

Stage 3

Stage 3 is the cool cutting-edge project. Fucking awesome, but jesus I have years of work ahead of me.

For hacking, Stage 3 was going from one glued-together script to building 100+ automated hacking tools, like XSScrapy. In 2014 my mom told me that if I wanted to be a hacker I should go get my master's in cybersecurity. So I fired XSScrapy at all 8 Ivy League university websites and found at least one XSS on every single one of them, as proof to her that school was pointless.

For the UFC model, Stage 3 was long nights brainstorming with GPT-4o about statistics techniques I'd never heard of: Poisson-gamma smoothing, beta-binomial smoothing, and opponent-adjusted z-scores that shrink each opponent's stats toward their weight class with empirical Bayes, so landing 50 strikes on an elite defender counts for more than landing 50 on a can. Every one of them pushed the model closer to the cutting edge, and I learned a massive amount of statistics along the way. Enough that the model has beaten Vegas for multiple years running, even though I'd probably still fail Statistics 101.

For robotics, Stage 3 is loci: an MCP server that gives an agent spatiotemporal memory, built with a self-improvement loop on top of SpatiotemporalBench and VSI-Bench. An agent proposes a change to the memory, runs the benchmarks, and keeps the change if the score goes up. That's the thesis from the intro, pointed at robots.

The catch is that self-improvement loops are absurdly token-hungry. Every iteration of the loop is a full benchmark run, and the benchmark itself is expensive: the agent re-reads its whole session at every frame, so one pass of Claude Opus over the 42 reference cases costs $117 in API-equivalent tokens. A loop wants dozens of those. I'm spending about $1,000 a month on AI bills, so if anyone out there wants to sponsor my robotics development, feel free. I won't stop you.

Want to see where your own agent lands? Run it, open a GitHub issue with your results.json and your agent's code, and I'll add it to the leaderboard.

And none of this would exist without orchflows. It's the foundation every loop in this project was built on. Skills are the programming language of agents, and orchflows is how you write programs in it: small skills snap together into workflows the way functions snap together into programs. A workflow encodes the process, not a particular model's quirks, so when a new model drops the self-improvement loop doesn't break. It just runs smarter.