Perception

Perception lets your skill look at the deck. Two functions live here: one captures a still image from an arm's wrist camera, and one relocalizes a known object from its AprilTags and refines its stored world pose. Both are imported with from execution.execution_functions import *.

Capturing an image

capture_image(arm, capture_name, save_to_project=False, export=False)

Takes a single still from one arm's wrist camera at the arm's current pose. It does not move the arm.

Parameters

• arm (str): which wrist camera to use. Either "left_arm" or "right_arm". Any other value fails the capture.
• capture_name (str): a name for this capture, and the name of the folder it lands in on the machine.

• save_to_project (bool, default False): set True to also save the capture into the project data folder at data/captures/<run>/<capture_name>/, keyed by run, so it travels with the project and syncs to the cloud. Left at False, the capture is still written on the machine under capture_name, but nothing goes into the synced project.
• export (bool, default False): set True to also upload this capture to your lab's data export bucket, under <prefix>/<project>/<run>/captures/<capture_name>/. The upload happens in the background, so it adds nothing to the time the capture itself takes. With no bucket configured, or outside a run, it does nothing and logs one warning.

Returns

A string locating the saved capture, or None if the capture failed. It points at a folder, not a single file, and it is the on-machine folder even when save_to_project=True.

The folder holds the colour image, a depth map (captures take depth by default), and a metadata file carrying the camera's intrinsics, its pose in the world frame, and the capture time. On real hardware the wrist camera writes its two infrared images alongside.

Hand the returned path to an operator prompt to show the image to a human. There is no capture browser in the app.

Failure behaviour

capture_image fails soft. It never raises into your skill. On an unknown arm, an unavailable camera, or any capture error it logs the problem and returns None. Check for None before you rely on the result.

path = capture_image("left_arm", "deck_overview")
if path is None:
    # capture did not succeed; decide how your skill proceeds
    ...

Relocalizing an object from AprilTags

localize_object_tags(
    object_name,
    viewpoints=None,
    *,
    arm="left_arm",
    tag_edge_m=None,
    collection=None,
    use_prior=True,
    min_detections=2,
    pos_sigma_gate_mm=15.0,
    max_move_mm=None,
    max_fitness_mm=None,
    # advanced, sensible defaults below
    mode="corner",
    sigma_px=1.0,
    huber_px=2.0,
    prior_std_pos_mm=20.0,
    prior_std_rot_deg=10.0,
    use_full_theta=False,
    index=None,
)

Re-reads a known object's AprilTags and refines its stored world pose. It captures one or more camera views, solves for the object pose anchored on the pose the object already has, and writes the refined pose back only if the solve passes the acceptance gates below. It fails soft: it logs and returns a diagnostic dict, and never raises into your skill.

Two consequences of an accepted solve are easy to miss:

• For an articulated object, the solved joint angles are written back too. Relocalizing a tagged object with a lid or a rotor overwrites the joint configuration the world was holding for it, not only its pose.
• The object ends up fixed in the world. If it was attached to an arm, the accepted write releases it to a fixed pose. Do not relocalize something an arm is carrying.

Parameters you will usually set

• object_name (str): the object to relocalize, by its world id or its metadata["name"].
• viewpoints (list, default None): where to look from.
  • None takes one capture from arm at the arm's current pose. No motion.
  • A list of anchor-name strings moves the arm to each named viewpoint anchor and captures there. Define these anchors on the object in its object_model.yaml.
  • A list of {"anchor": name, "arm": "left_arm"|"right_arm"} dicts lets you choose the arm per viewpoint.
    All captures in the list are fused into one solve.
• arm (str, default "left_arm"): which wrist camera to use when viewpoints is None, and the default arm for bare-string viewpoint anchors.
• tag_edge_m (float, metres, default None): the physical AprilTag edge length. Left at None it is taken from the object's tag collection, falling back to 0.020 (20 mm) when the object has none. Whatever it resolves to must match your printed tags. A wrong value mis-scales the recovered pose.
• collection (str, default None): which physical unit's tag set to solve against, for an object type that has more than one. Left at None it uses the unit this object was placed with, then the type's only collection, then the tags recorded in the object model itself. See Localizing an object for what a tagged unit is.
• use_prior (bool, default True): keep True to anchor the solve on the object's existing pose (relocalization). Set False for a cold start, such as a first placement or when you suspect the object moved a long way.

Acceptance gates

The refined pose is written back only if the solve is valid, saw at least min_detections tag observations, and its position uncertainty is within pos_sigma_gate_mm. Two optional guards can reject a suspect solve.

• min_detections (int, default 2): minimum number of tag observations required to accept.
• pos_sigma_gate_mm (float, mm, default 15.0): reject if the solved position 1-sigma uncertainty exceeds this.
• max_move_mm (float, mm, default None): if set, reject when the refined pose would move further than this from the stored pose. Catches a far-off prior fighting the tags or a genuine large move. None disables the guard.
• max_fitness_mm (float, mm, default None): if set, reject when the worst tag-to-surface fit residual exceeds this. None disables the guard.

Advanced parameters

These have sensible defaults and are rarely changed.

• mode (str, default "corner"): the tag residual model. "corner" is the robust default. "pose" is a legacy path. Both use the same prior, gate, and write-back.
• sigma_px (float, pixels, default 1.0) and huber_px (float, pixels, default 2.0): per-corner pixel noise and outlier threshold. Used only in "corner" mode.
• prior_std_pos_mm (float, mm, default 20.0) and prior_std_rot_deg (float, deg, default 10.0): how much drift the prior expects in position and rotation.
• use_full_theta (bool, default False): use the full calibration-uncertainty prior. Only valid when the frozen calibration matches the current rig.
• index (default None): a disambiguator when several objects share the same name.

Return value

A dict. On success:

{
    "success": True,
    "views": "...",       # which arm, or "N/M anchors" reached
    "result": { ... },    # diagnostic, see below
    "before": {"xyz": [...], "wxyz": [...]},   # pose before write-back
    "after":  {"xyz": [...], "wxyz": [...]},   # pose after write-back
    "moved_mm": 3.1,      # distance the object pose moved, mm
}

The result diagnostic holds:

• object_id: the resolved world object id.
• valid (bool): whether the solve was valid.
• n_detections, n_tags (int): tag observations used, and distinct tags seen.
• pos_sigma_mm (float, mm), rot_sigma_deg (float, deg): the solved pose 1-sigma uncertainty.
• joint_angles (dict): joint name to angle for the solved configuration.
• fitness_mm (dict): per-object worst tag-to-surface residual, in mm, when available.
• prior_used (bool): whether the stored pose was used as a prior.
• mode (str), views (str): echoes of what ran.
• moved_mm (float, mm), fitness_max_mm (float, mm): the candidate move against the stored pose and the worst fit residual, as read by the guards.

before.xyz and after.xyz are position triples. before.wxyz and after.wxyz are orientation quaternions in w, x, y, z order.

When the solve is rejected or cannot run

Every failure path returns a top-level success: False with a reason string, so branch on the returned dict's top-level success, for example loc["success"]. The result sub-dict, when present, does not carry a success key, and several failure paths omit result entirely. The reasons you may see:

• "missing_uncertainty_prior": no calibration-uncertainty prior for the active setup. Also returns path.
• "object_not_found": no object matched object_name.
• "no_object_model": the object has no object_model.yaml.
• "tag_collection_unresolved": the object type has several tagged units and the call could not tell which one to use, or the handle you passed in collection does not exist. Also returns collections, the list of handles it knows about, and error. Pass one of them in collection.
• "bad_tag_collection": the tagged unit's file could not be read. Also returns error.
• "no_captures": no usable camera views were gathered. Also returns before.
• "solve_failed": the solver raised. Also returns error and before.
• "gate_failed": the solve ran but did not pass acceptance. Returns gate (a list of the failed gate reasons), plus result, before, and after: None.

The gate failure never writes the object pose, so a rejected relocalization leaves the world unchanged.

Either outcome is written into the run log, so you can see what happened without reading the returned dict. An accepted solve logs Localized <name>: moved <n> mm (σ <x> mm, <n> detections), and a rejected one logs Localization of <name> rejected: followed by the gates that failed.

loc = localize_object_tags(
    "my_plate",
    viewpoints=["view_a", "view_b"],
    max_move_mm=25.0,
)
if not loc["success"]:
    # inspect loc["reason"] and, on a gate failure, loc["gate"]
    ...
else:
    moved = loc["moved_mm"]

Related