Cameras

Capture a posed still or a video from a wrist camera or another camera, and relocalize a tagged object from a wrist-camera image.

A camera is named by the arm it is mounted on ("left", "right"; "left_arm" and "right_arm" are accepted) or by the name of another camera the rig serves. list_cameras() returns the names available now.

The camera functions never raise on a camera failure: they log it and return None. Check the result before using it. A cancelled run still raises ExecutionCancelledInSkillError at their pause point.

A still image

capture_image(camera=None, name=None, *, depth=True, quality="normal", profile=None, lens_position=None,
              export=False, save_to_project=None, arm=None, capture_name=None)

Takes one image from camera and saves it under the folder name name. Returns a Capture, or None on failure.

  • depth=False saves colour only.
  • quality="best" switches the camera to its highest-quality mode for this one frame and back afterwards. Other readers of the camera see the switch.
  • profile= switches to one of the camera's advertised profiles for this one frame and back afterwards (it takes precedence over quality). lens_position (0-255, needs profile) holds the focus there for that frame, for a target nearer than the calibrated focus.
  • arm= and capture_name= are accepted for camera and name; save_to_project is ignored.
  • export=True also uploads the capture to your lab's own storage bucket, when data export is configured on the rig.

Every capture is also written into the project at data/captures/<execution_id>/<name>/, which syncs with the project.

Capture fieldValue
colorPath of the saved color.png, or None.
depthPath of the saved depth.npy, or None.
project_dirThe capture's folder in the project, or None when no project is bound.
world_P_camThe camera's pose in the world when the frame was taken (Pose), or None for a camera the world model does not place.
stampWhen the frame was exposed (s).
camera, nameThe camera and the capture name.

A Capture is also a string: the path of the capture folder. It can be passed straight to ask_user_slack(image=...), which sends every image in the folder. send_slack(files=...) takes files, so pass cap.color.

cap = capture_image("left", "deck_check")
if cap is None:
    raise RuntimeError("deck_check: the camera returned no image")
choice = ask_user_slack("Is the deck clear?", ["Yes", "No"], image=cap)

Video

rec = capture_video(cameras, name, seconds, *, fps=None, depth=False, export=False)

with record_video(cameras, name, *, fps=None, depth=False, export=False) as rec:
    move_arm(...)

capture_video records for seconds and returns when done. record_video records while the block runs, and stops at the block's end, on an exception in it, or when the run is cancelled. cameras is one camera name or a list, recorded in sync into one folder. A pause does not stop a recording.

  • fps=None keeps every frame the camera delivers; a lower fps keeps fewer.
  • depth=True also saves the full image and depth of every kept frame.

The recording folder holds recording.json and one subfolder per camera with video.mp4 and frames.jsonl (one line per frame: time, camera pose, arm joints). A camera that drops out mid-recording leaves a gap that is recorded in both files; recording resumes if it comes back.

Recording fieldValue
dirThe recording folder.
camerasThe cameras recorded (any that could not be opened are left out).
videosPer camera, its video.mp4, or None if it delivered no frame.
frames, fpsPer camera, frames written and the rate.
gapsPer camera, its outages.

Both return None (and record_video still runs its block) when no camera could be opened.

Relocalizing a tagged object

localize_object_tags(object_name, viewpoints=None, *, arm=None, use_prior=True, min_detections=2,
                     pos_sigma_gate_mm=15.0, max_move_mm=None, ...)

The options shown are the common ones; the cheat sheet lists the full signature.

For an object that carries AprilTags: takes one image with the wrist camera of arm where the arm is now, solves the object's pose from the tags it sees, and writes the refined pose into the world model only if the result passes its checks. It never raises; it returns a dict.

  • use_prior=True starts from the object's current pose in the world and treats the result as a correction of it. False solves from scratch, for an object that may have moved far.
  • The result is accepted only when it is valid, saw at least min_detections tag observations, and its position uncertainty is within pos_sigma_gate_mm. max_move_mm also rejects a result that moves the object further than that.
KeyValue
successTrue when the pose was written.
moved_mmHow far the object moved (on success).
before, afterThe pose before and after, as {"xyz", "wxyz"}.
reason, gateWhy it failed or which check rejected it.

A result that passes or fails the checks is recorded in the run log. Move the arm so the camera sees the tags before calling it. See Localizing an object for tagged objects.

res = localize_object_tags(plate_id, arm="left", max_move_mm=20.0)
if not res["success"]:
    raise RuntimeError(f"could not localize the plate: {res.get('reason')} {res.get('gate', '')}")

Did this page help you?