Perception

Perception lets your skill look at the deck. Two functions live here: one captures a still image from an arm's wrist camera, and one relocalizes a known object from its AprilTags and refines its stored world pose. Both are imported with from execution.execution_functions import *.

Capturing an image

capture_image(arm, capture_name)

Takes a single still from one arm's wrist camera at the arm's current pose. It does not move the arm.

Parameters

• arm (str): which wrist camera to use. Either "left_arm" or "right_arm". Any other value fails the capture.
• capture_name (str): a name for this capture. The image is saved under this name and surfaces in the run's readouts, so use something you will recognise later.

Returns

A string locating the saved capture, or None if the capture failed. The saved capture holds the colour image and appears in the run's readouts under capture_name.

Failure behaviour

capture_image fails soft. It never raises into your skill. On an unknown arm, an unavailable camera, or any capture error it logs the problem and returns None. Check for None before you rely on the result.

path = capture_image("left_arm", "deck_overview")
if path is None:
    # capture did not succeed; decide how your skill proceeds
    ...

Relocalizing an object from AprilTags

localize_object_tags(
    object_name,
    viewpoints=None,
    *,
    arm="left_arm",
    tag_edge_m=0.020,
    use_prior=True,
    min_detections=2,
    pos_sigma_gate_mm=15.0,
    max_move_mm=None,
    max_fitness_mm=None,
    # advanced, sensible defaults below
    mode="corner",
    sigma_px=1.0,
    huber_px=2.0,
    prior_std_pos_mm=20.0,
    prior_std_rot_deg=10.0,
    use_full_theta=False,
    index=None,
)

Re-reads a known object's AprilTags and refines its stored world pose. It captures one or more camera views, solves for the object pose anchored on the pose the object already has, and writes the refined pose back only if the solve passes the acceptance gates below. It fails soft: it logs and returns a diagnostic dict, and never raises into your skill.

Parameters you will usually set

• object_name (str): the object to relocalize, by its world id or its metadata["name"].
• viewpoints (list, default None): where to look from.
  • None takes one capture from arm at the arm's current pose. No motion.
  • A list of anchor-name strings moves the arm to each named viewpoint anchor and captures there. Define these anchors on the object in its object_model.yaml.
  • A list of {"anchor": name, "arm": "left_arm"|"right_arm"} dicts lets you choose the arm per viewpoint.
    All captures in the list are fused into one solve.
• arm (str, default "left_arm"): which wrist camera to use when viewpoints is None, and the default arm for bare-string viewpoint anchors.
• tag_edge_m (float, default 0.020): the physical AprilTag edge length in metres (0.020 is 20 mm). This must match your printed tags. A wrong value mis-scales the recovered pose.
• use_prior (bool, default True): keep True to anchor the solve on the object's existing pose (relocalization). Set False for a cold start, such as a first placement or when you suspect the object moved a long way.

Acceptance gates

The refined pose is written back only if the solve is valid, saw at least min_detections tag observations, and its position uncertainty is within pos_sigma_gate_mm. Two optional guards can reject a suspect solve.

• min_detections (int, default 2): minimum number of tag observations required to accept.
• pos_sigma_gate_mm (float, mm, default 15.0): reject if the solved position 1-sigma uncertainty exceeds this.
• max_move_mm (float, mm, default None): if set, reject when the refined pose would move further than this from the stored pose. Catches a far-off prior fighting the tags or a genuine large move. None disables the guard.
• max_fitness_mm (float, mm, default None): if set, reject when the worst tag-to-surface fit residual exceeds this. None disables the guard. Note that this fit is not always available, and when it is not the guard cannot act and a warning is logged.

Advanced parameters

These have sensible defaults and are rarely changed.

• mode (str, default "corner"): the tag residual model. "corner" is the robust default. "pose" is a legacy path. Both use the same prior, gate, and write-back.
• sigma_px (float, pixels, default 1.0) and huber_px (float, pixels, default 2.0): per-corner pixel noise and outlier threshold. Used only in "corner" mode.
• prior_std_pos_mm (float, mm, default 20.0) and prior_std_rot_deg (float, deg, default 10.0): how much drift the prior expects in position and rotation.
• use_full_theta (bool, default False): use the full calibration-uncertainty prior. Only valid when the frozen calibration matches the current rig.
• index (default None): a disambiguator when several objects share the same name.

Return value

A dict. On success:

{
    "success": True,
    "views": "...",       # which arm, or "N/M anchors" reached
    "result": { ... },    # diagnostic, see below
    "before": {"xyz": [...], "wxyz": [...]},   # pose before write-back
    "after":  {"xyz": [...], "wxyz": [...]},   # pose after write-back
    "moved_mm": 3.1,      # distance the object pose moved, mm
}

The result diagnostic holds:

• object_id: the resolved world object id.
• valid (bool): whether the solve was valid.
• n_detections, n_tags (int): tag observations used, and distinct tags seen.
• pos_sigma_mm (float, mm), rot_sigma_deg (float, deg): the solved pose 1-sigma uncertainty.
• joint_angles (dict): joint name to angle for the solved configuration.
• fitness_mm (dict): per-object worst tag-to-surface residual, in mm, when available.
• prior_used (bool): whether the stored pose was used as a prior.
• mode (str), views (str): echoes of what ran.
• moved_mm (float, mm), fitness_max_mm (float, mm): the candidate move against the stored pose and the worst fit residual, as read by the guards.

before.xyz and after.xyz are position triples. before.wxyz and after.wxyz are orientation quaternions in w, x, y, z order.

When the solve is rejected or cannot run

Every failure path returns a top-level success: False with a reason string, so branch on the returned dict's top-level success, for example loc["success"]. The result sub-dict, when present, does not carry a success key, and several failure paths omit result entirely. The reasons you may see:

• "missing_uncertainty_prior": no calibration-uncertainty prior for the active setup. Also returns path.
• "object_not_found": no object matched object_name.
• "no_object_model": the object has no object_model.yaml.
• "no_captures": no usable camera views were gathered. Also returns before.
• "solve_failed": the solver raised. Also returns error and before.
• "gate_failed": the solve ran but did not pass acceptance. Returns gate (a list of the failed gate reasons), plus result, before, and after: None.

The gate failure never writes the object pose, so a rejected relocalization leaves the world unchanged.

loc = localize_object_tags(
    "my_plate",
    viewpoints=["view_a", "view_b"],
    max_move_mm=25.0,
)
if not loc["success"]:
    # inspect loc["reason"] and, on a gate failure, loc["gate"]
    ...
else:
    moved = loc["moved_mm"]

Related