Ready
Selected
Root tip 1
- Detected at
- Robot target
- Positioning error
Tap a root tip, or switch the controller.
One test plate, run through the full pipeline with each controller. The traced roots are the real shortest paths from shoot to tip on this plate. Each ring shows that tip's measured positioning error, drawn to scale on the 150 mm plate, with a minimum size so the smallest stay visible.
Problem
At NPEC, the Netherlands Plant Eco-phenotyping Centre, plants are grown and imaged at greenhouse scale, but inoculating their root tips is still done by hand with a pipette: slow, inconsistent and supervised. The goal was to take the human out of the loop: find every root tip in a photo of a plate, and send a robot pipette to it.
What I built
A pipeline from photo to pipette, on a simulated Opentrons OT-2 robot. Two U-Net models segment the roots and shoots, graph search traces each primary root to its tip, a calibration maps that pixel to robot coordinates, and a controller drives the pipette there. For the last step I built two controllers, a PID controller tuned by grid search and a Soft Actor-Critic reinforcement-learning agent, and benchmarked them on the same targets.
The hardest problems were not the models. Thin roots broke into gaps, roots strayed into their neighbours' columns, and a missing axis flip sent the robot to the mirror image of every target. I fixed them by bridging gaps along the growth direction, reassigning stray roots to the right plant, and calibrating millimetres per pixel from the detected dish.
- SegmentU-Net, roots and shoots
- Tracegraph search to the root tip
- Mappixels to robot millimetres
- MovePID or RL controller
Result
I chose PID for delivery. In the final evaluation it reached root tips with a mean error of 2.13 mm, against 8.89 mm for the RL agent, and in 15.7 seconds per target instead of 25.1. On the vision side, the segmentation models reached F1 scores of 0.86 for roots and 0.83 for shoots, and the pipeline found 89 of the 95 root tips in the test set.
Open space vs the real task
Mean positioning error. Open space: random targets across the robot's work envelope. Full pipeline: real root tips, through segmentation, tracing and coordinate mapping.
What I'd do now
The system works end to end; the gains now sit at the edges of each stage. Next, I would:
- Train the agent on the task it will face. It learned on random goals across the whole work envelope, while real targets cluster on a small plate. Training on plate-like targets, with randomised conditions, would likely close much of the gap it showed in the full pipeline.
- Sharpen the masks for thin roots. Most of the six missed tips came from faint roots breaking up in the mask. Higher-resolution patches, or more thin-root examples in training, would catch them earlier.
- Relax the layout assumptions. The tracer expects up to five plants in vertical columns with roots that don't overlap. Instance segmentation would handle crowded or crossing roots.
- Generalise beyond one species. The models were trained on a single plant type; NPEC grows many.
Skills gained
- Semantic segmentation: separate U-Net models for roots and shoots, trained on filtered 128-pixel patches
- Image analysis: skeletonisation, connected components and Dijkstra's algorithm to trace the primary root
- Control engineering: building a three-axis PID controller and tuning its gains by grid search
- Reinforcement learning: designing a Gymnasium environment and reward, benchmarking PPO, SAC and TD3, and tuning SAC in a team sweep
- Systems integration: mapping pixels to robot coordinates with live calibration, then benchmarking the whole loop
- Experiment tooling: Weights & Biases tracking, checkpoints and automatic kill switches for failing runs
Contact
Looking for a data science or AI intern?
I'm open to internships in machine learning, computer vision and robotics for February 2027. Email is the quickest way to reach me.