Depth-Guided Computer Vision for Precision Agricultural Fertilizer Application

Depth is estimated with MiDaS, a monocular framework that produces relative inverse depth from a single RGB image. We use the Dense Prediction Transformer (DPT) architecture, which pairs a transformer encoder with a convolutional decoder: features are extracted from four intermediate transformer layers and fused through successive Reassemble and RefineNet blocks, then progressively upsampled and passed through a convolutional head. The output is constrained non-negative by a ReLU.

Two variants were evaluated. The BEiT-Large model served as a high-accuracy reference during development; the Swin V2 Tiny model is what actually deploys to the Jetson, where its 256 × 256 square input and far smaller backbone are what make frame-rate operation possible.

Table 1. Evaluated DPT model variants.

Inference runs under torch.no_grad(). On CUDA hardware the model is cast to half precision and moved to torch.channels_last memory layout, which reduces memory bandwidth on the Jetson's shared-memory architecture. The raw prediction is upsampled back to native frame resolution with bicubic interpolation before any downstream use, so the depth map and the RGB frame remain pixel-aligned.

Table 2. Input preprocessing and inference parameters.

prediction = model.forward(sample)
prediction = (
    torch.nn.functional.interpolate(
        prediction.unsqueeze(1),
        size=target_size[::-1],
        mode="bicubic",
        align_corners=False,
    )
    .squeeze()
    .detach()
    .cpu()
    .numpy()
)

MiDaS returns relative inverse depth, not metres or feet. To obtain a usable proximity signal we calibrate against a single reference measurement: an object placed at a known 2 ft standoff, with the maximum depth value across the frame recorded at that distance. The ratio of the two yields a linear scale factor.

A purely linear model is a poor fit, because inverse depth and physical distance are related non-linearly — roughly D = 1 / (ad + b). We therefore refine the estimate with a power law anchored at the same reference point:

Table 3. Calibration constants.

A binary proximity conclusion is derived from the first-order estimate: when the reflected metric value falls below zero — indicating the object is nearer than the calibration reference — the frame is labelled “Object Detected Close to Camera.”

Field scenes contain vegetation well outside the treatment row. To stop that clutter from reaching the detector, pixels whose inverse depth falls below a fixed threshold are classified as background and replaced with solid black. The raw threshold mask is noisy, so it is cleaned with a morphological opening (removing isolated foreground speckle) followed by a closing (filling small holes inside the canopy region).

Table 4. Depth masking parameters.

background_mask = (depth_map < depth_threshold).astype(np.uint8)
kernel = np.ones((5, 5), np.uint8)
background_mask = cv2.morphologyEx(background_mask, cv2.MORPH_OPEN, kernel)
background_mask = cv2.morphologyEx(background_mask, cv2.MORPH_CLOSE, kernel)

foreground_mask = 1 - background_mask
bg = np.full_like(img_rgb, bg_color, dtype=img_rgb.dtype)
result_rgb = np.where(foreground_mask[..., None] == 1, img_rgb, bg)

Masking is deliberately decoupled from detection. Two independent flags control it: --use_depth_mask produces the masked frame, while --detect_on_depth_mask decides whether the detector consumes the masked frame or the raw one. This lets masking be evaluated as a display aid and as a detection gate separately, without rebuilding the pipeline.

Canopy localization uses a YOLOv8 detector fine-tuned on a custom canopy dataset. At load time the model is moved to GPU, its Conv+BatchNorm layers are folded together via model.fuse(), and inference is run in half precision — both throughput optimizations that matter on a power-constrained Jetson.

Table 5. Detector configuration.

Every accepted box is rasterized into a shared binary mask. This union representation is what makes the downstream logic robust: overlapping detections of the same plant contribute their area once rather than double-counting, and the zone integrator never needs to reason about individual box identity.

The frame is divided into three equal horizontal bands aligned with the physical nozzle bar. For each band we compute the fraction of its pixels that lie inside at least one detection:

Table 6. Nozzle zone definitions.

region_area = region_h * w
covered_pixels = int(det_mask[y1_area:y2_area, :].sum())
coverage_ratio = covered_pixels / max(region_area, 1)

coverage_ratios[i] = coverage_ratio
region_active[i] = coverage_ratio > coverage_threshold

Actuation is the point where perception becomes physical. Output runs on an NVIDIA Jetson through Jetson.GPIO using the BOARD pin convention, which addresses physical connector positions rather than SoC GPIO numbers. Pins initialize LOW, and a cleanup routine drives them LOW and releases the lines on exit — including on exception — so a crash cannot leave a nozzle latched open.

Bring-up progression

Before touching the detection pipeline, GPIO behaviour was validated by a series of standalone scripts, each isolating one unknown.

Table 7. GPIO validation script progression.

v1 — two-pin combinatorial encoding

The first integrated design had only two output lines available and needed to represent three zones. It used a combinatorial encoding, where zone identity is carried by the pair of pin states rather than by a dedicated line:

The encoding is unambiguous only while exactly one zone is active. With multiple zones active it degenerates: the intended resolution is a logical OR (pin1 = top OR middle, pin2 = top OR bottom), which means top alone and middle + bottom are indistinguishable at the pins. Worse, the implementation wrote pins sequentially rather than in a single pass:

if top_active:
    GPIO.output(led_pin1, GPIO.HIGH)
    GPIO.output(led_pin2, GPIO.HIGH)
if middle_active:
    GPIO.output(led_pin1, GPIO.HIGH)
if bottom_active:
    GPIO.output(led_pin2, GPIO.HIGH)
if (not middle_active):
    GPIO.output(led_pin1, GPIO.LOW)
if (not bottom_active):
    GPIO.output(led_pin2, GPIO.LOW)

Because the trailing clears are unconditional on top_active, a frame in which only the top zone is active first drives both pins HIGH and then immediately pulls both LOW — the top zone can never actually hold its output. This class of failure is intrinsic to resolving a shared line across sequential writes.

v2 — three-pin direct mapping (deployed)

The deployed system allocates one line per zone. Zone state and pin state become identical, all three pins are written in a single pass per frame, and every one of the eight multi-zone combinations is representable without conflict.

top_active, middle_active, bottom_active = region_active

GPIO.output(led_pin1, GPIO.HIGH if top_active else GPIO.LOW)
GPIO.output(led_pin2, GPIO.HIGH if middle_active else GPIO.LOW)
GPIO.output(led_pin3, GPIO.HIGH if bottom_active else GPIO.LOW)