Vinea — a greenhouse harvesting robot, proven in simulation first

Four Venlo rows, eight metres end to end, and a trolley that rides the heating pipes instead of the floor. Dutch glasshouse heating pipe is 51 mm OD on 550 mm centres, and it is already there in every house worth selling into, already straight, already load-bearing, already used by the crop trolleys the growers push around by hand.

Putting the base on that rail makes it a 1-DOF robot. There is one number describing where it is, it comes off a wheel encoder, and it does not drift because the rail does not let it. There is no localisation stack. There is no map. It physically cannot wander into a crop row, in the sense that a bollard cannot wander into traffic. An AMR here would be solving SLAM in a repetitive, self-similar, visually hostile corridor in order to reproduce a constraint the building is giving away for free.

Why the row pitch is what forces two arms

A Venlo row pitch is 1.60 m. The trolley runs down the middle, so it sits 800 mm from each crop wall. Every clearance number from Phase II and Phase III — the 40 mm planner defence, the 15 mm guard, the approach envelopes, the staging planes — was measured at 600 mm. All of that would have needed re-measuring.

It did not, because 800 − 600 = 200, and an arm mounted 200 mm off the deck centreline toward its own crop wall sees exactly the geometry it was characterised at. Every earlier number survives unchanged. And the 200 mm on the other side of the centreline is not wasted space — it is where the second arm goes. The second arm is not an upgrade. It is what the building’s own dimensions hand you once you stop fighting them.

Play: Two-arm farm run

A shift with nothing handed to it

The run: the robot is told the house exists and which aisle it is in. It is not told where a single tomato is. It scouts, builds its own fruit list, drives, picks, crates.

The one refusal is the good kind. There was a genuine neighbour at 30 mm, inside the 40 mm the planner defends, and no route cleared it, so the planner declined rather than gambling. That is the behaviour Phase II was built for, arriving unprompted in a scene nobody hand-authored.

Ten bugs the second arm found

Fitting a second arm and then parallelising it turned up ten defects that had been shipping. This list is the most useful thing on the page, because every one of them was invisible in a working single-arm system.

1. The two arms were parked 83 mm inside each other, forearm through forearm. This is visible in every render since the arm was added. Nothing was looking.
2. Nothing checked arm against arm. The obstacle set was built from crop and structure, so the clash above was in nobody’s obstacle set — not a missed collision, an unasked question.
3. Arm B’s grasps were being scored against arm A’s gripper. B reported failure on picks that had demonstrably worked.
4. The planner previewed routes with 13 free DOF that the executor then flew with 6. Every preview was of a robot that does not exist.
5. Two arms were one rigid assembly moved by two different transports. It surfaced twice — once as a teleport, once as a pole leaving its camera behind — and was one defect.
6. Preferred posture lived in the null space.
7. So did permitted joint speed. Both were invisible with one arm: the tool arrives at the same pose either way, and the crop only cares where the tool is. It took a second arm — whose elbow is the thing you collide with — to make the null space observable at all. That is the general lesson. A redundant manipulator has degrees of freedom that no single-arm test can see, because nothing in a single-arm world is downstream of them.
8. A numpy mask was 39% of the entire simulation. It was never profiled, because a plausible story about camera cadences had been written down as a comment, and the comment was believed.

Two more are in my notes and not yet written up. They will land here rather than be quietly dropped to make the count work.

Number eight is the one to take personally. A comment is not evidence. It cost 39% of wall-clock for weeks because it read like an explanation.

The concurrency result is a limit, not a win

Both arms’ PARK postures fold the elbow across the aisle. Both elbows therefore want the shared middle of the deck, and the arms interlock there. I narrowed that interlock four times. Every narrowing found another contact at 12–15 mm.

The important part of that: the hazard is the arm that is waiting, not the one moving. A parked arm is not out of the way. It is a static obstacle sitting in the exact volume the other arm needs, and it got there by being told to relax.

So what shipped is an interlock over the whole mission: 6/7 crated, 0 guard aborts, 0% manipulation overlap. Manipulation is serialised. Everything else — mapping, planning, both deck heads scanning, travel — runs concurrent. Calling this a two-arm speedup would be a lie; the second arm currently buys reach and redundancy, not throughput.

57 logged picks across four crop densities. This is the headline result of the sprint and it is not a good one.

22.7× short. I am publishing it because it is the most valuable thing in the repository and it cost a laptop instead of an arm. A number this bad, found this early, is the entire argument for simulating first — the alternative was finding it after buying hardware.

46% clean is substantially luck

The grasp goes unstable on 34 of 57. Of those 34, 17 crate anyway. Not because anything recovers — because the fruit is flung, at up to 4.22 m/s, and happens to fly toward the crate. Half the successes are ballistics. The honest clean rate is closer to 23/57.

Play: Throughput campaign

Where the fruit is actually lost

I traced a failing pick substep by substep expecting to find a bad grasp. At the moment of detachment the fruit is centred — 2.3 mm off the tool site. The grasp is fine. The fruit is lost a second and a half later, during the carry.

It creeps out of a closed gripper under its own weight. A soft MuJoCo contact drifts under sustained tangential load: the constraint is compliant by design, gravity is a constant tangential pull along the pad face, and the contact point walks. Once the fruit centre has walked about 30 mm, the sphere is off the edge of a 37 mm pad and there is nothing left holding it.

The fix is one solref value on the pad contact — a stiffer, less damped reference that stops the drift:

<!-- before: compliant enough that the contact point walks -->
<pair geom1="pad_l" geom2="fruit" solref="0.02 1"   friction="1.0 0.005 0.0001"/>

<!-- after -->
<pair geom1="pad_l" geom2="fruit" solref="0.002 1"  friction="1.0 0.005 0.0001"/>

Carry creep: 280 mm → 34 mm. Crated: 8/8.

The 6.3 kg/hr above is now stale — it predates this fix, the deck camera, and the pick-ordering rewrite. It has not been re-measured. That is why Phase IV is still open.

Up to here the robot has been handed fruit positions. This closes the loop: an eye-in-hand RGBD camera on the wrist, and the arm picks from its own estimates.

The mount position was measured against the gripper envelope rather than guessed — swept until the camera sees the approach volume without the fingers occluding it and without the camera body entering the collision set the planner already checks. Guessing this costs you a rebuild of every clearance number downstream.

Scoring detectors on the thing that matters

Detectors were scored on recall and false positives per frame, never mAP. mAP averages precision over a sweep of confidence thresholds you will never ship, and rewards ranking quality on a task with no ranking in it. What a harvester needs to know is: did you find the fruit, and did you invent one.

HSV beating a network is not a result about HSV. It is a result about the scene: renderer-flat lighting, saturated red fruit, no specularity, no shadow. That advantage does not survive contact with a real greenhouse, and the number should not be quoted as if it would.

Deprojection, and the 33 mm that is the whole story

Pixel-to-3D deprojection was verified against known geometry to 0.39 mm. The camera model is not the problem. The problem is that a depth camera returns the front surface of the fruit and a grasp needs its centre:

# pixel + depth -> camera frame  (verified to 0.39 mm)
z = depth[v, u]
x = (u - cx) * z / fx
y = (v - cy) * z / fy
p_surface_cam = np.array([x, y, z])

# depth hits the near surface; the grasp target is the centre.
# push along the camera ray by one fruit radius.
FRUIT_RADIUS_M = 0.033
ray = p_surface_cam / np.linalg.norm(p_surface_cam)
p_centre_cam = p_surface_cam + ray * FRUIT_RADIUS_M

p_centre_world = T_world_cam @ np.append(p_centre_cam, 1.0)

That 33 mm is not a refinement, it is the estimate. Get the radius wrong, or get a partially occluded fruit whose visible surface is not its near pole, and the error goes straight into the grasp.

Play: Onboard perception

The error budget does not close

4/5 clean on its own estimates, against 5/5 on ground truth. And the budget does not close, which I am publishing rather than rounding away:

• The planner defends 40 mm of clearance.
• The runtime guard aborts at 15 mm.
• That leaves 17 mm of positional budget.
• Measured p95 estimate error is 20.9 mm.

The p95 error is larger than the budget it has to fit in. One pick in twenty is expected to be outside tolerance before it starts, and 4/5 is consistent with that rather than in tension with it. The sample is five picks; do not read it as a rate.

One fruit did not get dropped

It left the gripper and ended up six metres outside the greenhouse. In simulation that is a funny frame. In a real building it is a projectile, in a room with people in it, made mostly of glass. It is logged here because the ballistic-success behaviour in Day 9 is the same defect wearing a better outcome, and both were tolerable only because nothing here has ever had mass in a room.

A row instead of a fruit. Two things had to exist: a stem that can break, and a planner that knows a pick is not worth attempting.

Breakable welds, not FEM fracture

Proper deformable-body fracture is a research project. A weld constraint with a force threshold is an afternoon, and it reproduces the one property the rest of the system actually consumes: the fruit stays attached until a force, then it does not.

Two traps in that afternoon were worth the whole week.

Trap one: the default anchor. MuJoCo defaults a weld’s anchor to [0, 1, 0]. That is one metre along Y — a metre-long lever arm on every peduncle in the house. Fruit sagged 160 mm and I spent a while looking for a mass error that was not there.

Trap two: the weld must be compliant. Against a rigid weld, the gripper’s own closing motion loads the peduncle to 15.8 N — above the break force — so every fruit detached on grip instead of on pull. The failure looked like a grasp problem and was a constraint-stiffness problem. Nothing about the pull was ever being tested.

<!-- wrong: anchor defaults to [0 1 0] -> 1 m lever arm, 160 mm sag.
     rigid solref -> closing the gripper alone reaches 15.8 N and detaches. -->
<equality>
  <weld body1="truss_03" body2="fruit_03"/>
</equality>

<!-- right: anchor on the joint itself, and compliant enough that the
     grip transient stays under the 12 N break force. -->
<equality>
  <weld body1="truss_03" body2="fruit_03"
        anchor="0 0 0"
        solref="0.02 1" solimp="0.9 0.95 0.001"
        torquescale="0"/>
</equality>

A planner that refuses

The clearance check is deliberately unsubtle: every collision geom on the arm, against every fruit it is not currently picking, sampled along the whole route rather than at the goal pose. Checking only the final grasp pose is the usual mistake — the arm does not teleport there.

Under 40 mm, it swaps the route. There are four things it can change, tried in order of how much they cost:

1. a different pull direction off the truss
2. a rolled wrist, same approach
3. a lane under the canopy instead of through it
4. a deeper staging plane before the final approach

If none of the four clears 40 mm, it refuses the pick outright. Not a degraded attempt — no attempt. A refusal costs one tomato. A collateral detachment costs that tomato, its neighbour, and the grower’s belief that the machine is safe to leave alone in a row.

Underneath that, a runtime guard aborts the motion at 15 mm regardless of what the planner believed, because the planner reasons about a model and the guard watches the actual state.

Play: Ten-pick row

Zero collateral in both bands is the number I care about. The degradation from 10/10 to 9/10 is the planner spending its refusal budget correctly as the scene gets harder.

A Fairino FR5, imported from the manufacturer’s official URDF. That matters more than it sounds: the URDF carries the real SolidWorks masses and inertia tensors, so MuJoCo takes it with no CAD conversion step and no invented inertias.

Invented inertias are the quiet killer here. An arm with plausible-looking made-up inertia accelerates wrong, and every force number measured downstream — peduncle load, grip transient, contact impulse — is then fiction that looks like data.

Reach, measured rather than quoted

The datasheet reach is to the flange. What a planner needs is the envelope of the fingertips, with the gripper fitted. So: sample 20,000 joint configurations, forward-kinematic each one, take the hull.

reach = 0.0
for _ in range(20_000):
    data.qpos[:6] = rng.uniform(joint_lo, joint_hi)
    mujoco.mj_kinematics(model, data)
    p = data.site_xpos[FINGERTIP_SITE]
    reach = max(reach, np.linalg.norm(p - base_xy))

# -> 1.100 m to the fingertips

1.100 m. Every workspace decision after this — trolley offset, row pitch, staging planes — is derived from that number rather than from a brochure.

3.2 mm bare, 5.2 mm with the gripper

Positioning accuracy at the tool site: 3.2 mm bare, 5.2 mm with the gripper fitted. The 2 mm difference is gravity droop, and it is not a tuning failure — it is what a P+D position servo does.

A proportional-derivative controller has no term that survives a constant load. With the gripper hung off the end, each joint sees a steady gravitational torque τ_g, and the loop settles wherever k_p · e = τ_g, giving a steady-state error e_ss = τ_g / k_p that no amount of damping removes. Nulling it needs an integral term or gravity feedforward. Neither is in the demo loop, so the 2 mm is carried honestly into the error budget instead of being tuned away.

Play: First pick

42 of 42 mouse-driven picks. Drag a target, mink solves differential IK to it, the gripper closes, the fruit comes off. That is the entire Phase I deliverable.