Data Export
Send a run's captures, API responses, and logs to a storage bucket your lab owns.
Data Export points the platform at an S3 bucket you control, so a run's output lands in your own storage instead of only travelling with the project. You configure the destination once per machine; skills then opt individual calls in with export=True.
Data Export is only offered on a local install attached to a rig. The entry does not appear in the cloud app, because the credentials live on the machine rather than in your account. A skill running in the cloud can still pass
export=True, but with nowhere configured it does nothing and logs one warning.
Opening it
The header ⋮ menu, the same one that holds Hardware Setup. The entry reads Data Export, and opens a dialog headed Data Export that names the S3 destination it is configuring.
What you need first
The bucket has to exist already. Nothing here creates one.
You also need an access key scoped to that bucket. The dialog states the minimum policy it expects: s3:PutObject, plus s3:DeleteObject so the connection test can clean up after its probe, on arn:aws:s3:::<bucket>/<prefix>/*.
Give it nothing wider. The key lives on the machine and is used by every run anyone starts there with export=True, and Test connection will exercise it against whatever bucket and prefix are typed into the dialog. A broadly scoped key is broadly scoped for everyone on the rig.
Who can change it
Anyone signed in to the machine can open the dialog and read its status: bucket, region, prefix, and the last four characters of the key. The secret is never returned.
Changing it is narrower. While no connection exists, anyone can set one up. Once one is saved, only the person who saved it, shown as Configured by, or an administrator can change or remove it. Anyone else gets Data export was configured by <owner> — only they or an admin can change it. when they press Save, and the same message ending can clear it. on Remove connection.
Test connection is not restricted, so anyone can check that the stored credentials still work.
The fields
| Field | Notes |
|---|---|
| Bucket | Required. The bucket name on its own, not a URL. |
| Region | Required. Placeholder us-east-1. |
| Prefix | Optional. A path to nest everything under, for example zeon/robot-1. Clearing it is a valid edit and drops the prefix. |
| Access key ID | Required to configure. Placeholder AKIA…. |
| Secret access key | Required to configure. |
| Session token | Optional, for temporary credentials. |
Save stays disabled until the destination is valid and, on a first setup, until both halves of the credential pair are filled in. Test connection writes and deletes a small probe object so you find out now rather than at the end of a run.
Credentials are write-once
Once saved, credentials are never sent back to the browser. In place of the fields you get a summary reading AWS key ••••<last4>, with + session token beside it when one is stored. That is enough to answer "which key is this" without handing half a credential pair to everyone who can open the dialog.
Changing them means entering a new pair. Press Replace credentials… to reopen the fields, or Keep current to back out. The key ID and the secret are always swapped together.
Editing only the bucket, region, or prefix does not require re-entering credentials.
Reading the status
With a connection configured, the dialog shows:
- Configured by whoever last saved it, with the time.
- Uploads pending, the number of queued upload jobs. A job, not a file: one exported capture folder counts as one however many files it holds, and the job currently uploading is not counted.
- Last upload error, in red. It means uploads are failing now: the first success clears it and the red block disappears. It is not a history.
Remove connection deletes the bucket and credentials from this machine after a confirmation. They cannot be recovered, and exports fail until a new connection is configured.
What a skill sends
Nothing exports automatically. Three runtime functions take an export flag:
| Call | What it uploads | Where it lands |
|---|---|---|
capture_image(..., export=True) | that capture | <prefix>/<project>/<run>/captures/<capture_name>/ |
api_request(..., save_name=..., export=True) | that response | <prefix>/<project>/<run>/api/<save_name>.json |
print_log(..., export=True) | the whole run folder, once a workflow run finishes | <prefix>/<project>/<run>/run/ |
<project> is the project's name when it has one, and its id otherwise, so a bucket full of runs is readable without a lookup. It is resolved when the data is produced, so switching project part-way through a run does not scatter that run across two prefixes.
print_log is the odd one out: its flag marks the run rather than the line. One call anywhere in the run is enough, and at the end the run's log, metadata, and ledgers upload together.
The run-folder upload happens for workflow runs only. A skill run started from the Skills Editor accepts
print_log(..., export=True)and marks itself, but nothing flushes it, so no run folder is uploaded.capture_imageandapi_requestexports are unaffected: those upload as they happen, on either kind of run.
While a destination is configured, a workflow run's log opens with an
exports → …line naming it. If someone repoints the bucket, the runs that went somewhere else say so in their own record.
When an upload fails
Uploads happen in the background, on a queue, so a slow or unreachable bucket never slows a run down or fails one.
- Each file gets three attempts in total, waiting one second and then two seconds before the retries.
- After that, the failure is recorded as
Last upload errorin this dialog and as anExport failed:line in that run's log. - Local files are never deleted. A failed upload is always recoverable from the machine.
- If uploads back up badly, the queue drops the overflow with a warning rather than growing without limit.
One cosmetic thing to expect: a capture's depth data is compressed on the way up, so it appears in your bucket as depth.npz rather than depth.npy.
Choosing between export and the project folder
export is not a replacement for save_to_project. They are two destinations, and a call can use either or both. See Two places run output can go for the comparison, but the short version is that the project folder comes down with zeon sync and works everywhere including the cloud, while export goes to storage you own and is only available here.
Related
Updated about 2 months ago