S3: create() requires s3:DeleteObject for index.lock, so an append-only writer is impossible

[khustup2]

Version: 4.7.0 · Linux aarch64, Python 3.12 · S3 storage

What happens

deeplake.create() requires s3:DeleteObject. It takes and then removes a lock object:

StorageAccessDenied: [S3] Access denied:
  _deeplake_log/d49ee0d83edd7d49ac93c13f64e2d946/index.lock
User: ... is not authorized to perform: s3:DeleteObject on
  <BUCKET>/<prefix>/tables/fresh/_deeplake_log/.../index.lock

open + append + commit on an existing dataset need no delete at all — measured below. So the delete requirement is specific to dataset creation.

Reproduce

Credential with s3:GetObject, s3:PutObject, s3:ListBucket, s3:GetBucketLocation, and KMS encrypt+decrypt. No s3:DeleteObject:

import deeplake, deeplake.types as t

# 1. append to an existing dataset — WORKS
ds = deeplake.open(url, creds=creds)
ds.append([{"event_id": "e2", "text": "second"}]); ds.commit("x")
# rows: 1 | active commits: 2

# 2. create a new dataset — FAILS on s3:DeleteObject for index.lock
deeplake.create(url2, creds=creds, schema={"event_id": t.Text(), "text": t.Text()})

Why it matters

An append-only writer that cannot destroy data is a posture people deliberately want: a process that adds rows and provably cannot remove them. Today that is achievable for open/append but not for create, so any credential that must create a dataset also carries the right to delete objects under the whole prefix.

For us this is an audit record written by a process running untrusted code. Being unable to grant "add, never remove" for the whole lifecycle is the difference between a record that can be tampered with and one that cannot.

Ask

Any of these solves it:

1. let the lock expire by S3 lifecycle rather than being deleted;
2. tolerate a stale lock at create (it is a create-time mutex, and a leftover object is not corruption);
3. make lock cleanup best-effort, so an AccessDenied on the delete does not fail the create.