Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions packages/storage/src/Storage/Device/S3.php
Original file line number Diff line number Diff line change
Expand Up @@ -747,6 +747,7 @@ protected function call(string $method, string $uri, StreamInterface|string $dat
$headers['host'] = $this->host;
$headers['date'] = gmdate('D, d M Y H:i:s T');
$headers['content-md5'] = $md5;
$headers['content-length'] = (string) $this->bodyLength($body);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Unknown uploads remain chunked

For seekable streams whose getSize() is unknown—the exact case this change intends to fix—setting this header does not give the default cURL transport the measured size. The transport calls getSize() again, leaves CURLOPT_INFILESIZE unset, and configures an HTTP/1.1 upload with an unknown length. It can therefore still use chunked transfer encoding despite the signed Content-Length, causing strict S3-compatible endpoints to continue rejecting these uploads. Pass the measured length to the transport or otherwise ensure it uses this value for request framing.

Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/storage/src/Storage/Device/S3.php
Line: 750

Comment:
**Unknown uploads remain chunked**

For seekable streams whose `getSize()` is unknown—the exact case this change intends to fix—setting this header does not give the default cURL transport the measured size. The transport calls `getSize()` again, leaves `CURLOPT_INFILESIZE` unset, and configures an HTTP/1.1 upload with an unknown length. It can therefore still use chunked transfer encoding despite the signed `Content-Length`, causing strict S3-compatible endpoints to continue rejecting these uploads. Pass the measured length to the transport or otherwise ensure it uses this value for request framing.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Claude Code Fix in Codex


$amzHeaders = array_filter($amzHeaders, fn(string $value): bool => $value !== '');
$amzHeaders['x-amz-date'] = gmdate('Ymd\THis\Z');
Expand Down Expand Up @@ -800,6 +801,30 @@ protected function call(string $method, string $uri, StreamInterface|string $dat
);
}

/**
* Length of the request body in bytes.
*
* Strict S3-compatible endpoints (for example Google Cloud Storage's XML
* API) reject requests without an explicit Content-Length with a 411, so
* every request must carry one. `StreamInterface::getSize()` may return
* null for a stream that is still seekable, in which case the size is
* measured by seeking to the end and back.
*/
private function bodyLength(StreamInterface $body): int
{
$length = $body->getSize();
if ($length !== null) {
return $length;
}

$position = $body->tell();
$body->seek(0, SEEK_END);
$length = $body->tell();
$body->seek($position);

return $length;
}

/**
* Hash a request body for signing without materializing it as a string.
*
Expand Down
195 changes: 195 additions & 0 deletions packages/storage/tests/Storage/Device/S3Test.php
Original file line number Diff line number Diff line change
Expand Up @@ -191,6 +191,136 @@ public function withFollowRedirects(bool $enabled = true): static
}
}

/**
* PSR-18 + streaming client that records every outgoing request, so tests can
* assert on the headers the real S3::call() puts on the wire.
*/
class CapturingClient implements \Psr\Http\Client\ClientInterface, \Utopia\Psr18\StreamingClientInterface
{
/**
* @var list<RequestInterface>
*/
public array $requests = [];

public function sendRequest(RequestInterface $request): ResponseInterface
{
$this->requests[] = $request;

return $this->respond($request);
}

public function stream(RequestInterface $request, callable $sink): ResponseInterface
{
$this->requests[] = $request;

return $this->respond($request);
}

private function respond(RequestInterface $request): ResponseInterface
{
if ($request->getMethod() === 'HEAD') {
return new Response(404);
}
$response = new Response(200)->withHeader('ETag', '"etag-1"');
if ($request->getMethod() === 'POST' && str_contains($request->getUri()->getQuery(), 'uploads')) {
return $response
->withHeader('content-type', 'application/xml')
->withBody(new Stream('<?xml version="1.0" encoding="UTF-8"?><InitiateMultipartUploadResult><UploadId>upload-123</UploadId></InitiateMultipartUploadResult>'));
}

return $response;
}
}

/**
* Stream whose getSize() reports null, like the decorated streams produced by
* upload pipelines, while remaining seekable.
*/
class UnknownSizeStream implements StreamInterface
{
private readonly Stream $inner;

public function __construct(string $content)
{
$this->inner = new Stream($content);
}

public function getSize(): ?int
{
return null;
}

public function __toString(): string
{
return $this->inner->__toString();
}

public function close(): void
{
$this->inner->close();
}

public function detach()
{
return $this->inner->detach();
}

public function tell(): int
{
return $this->inner->tell();
}

public function eof(): bool
{
return $this->inner->eof();
}

public function isSeekable(): bool
{
return $this->inner->isSeekable();
}

public function seek(int $offset, int $whence = SEEK_SET): void
{
$this->inner->seek($offset, $whence);
}

public function rewind(): void
{
$this->inner->rewind();
}

public function isWritable(): bool
{
return $this->inner->isWritable();
}

public function write(string $string): int
{
return $this->inner->write($string);
}

public function isReadable(): bool
{
return $this->inner->isReadable();
}

public function read(int $length): string
{
return $this->inner->read($length);
}

public function getContents(): string
{
return $this->inner->getContents();
}

public function getMetadata(?string $key = null): mixed
{
return $this->inner->getMetadata($key);
}
}

final class S3Test extends TestCase
{
private TestableS3 $s3;
Expand Down Expand Up @@ -581,4 +711,69 @@ public function testXmlListingWithSingleObjectIsDecoded(): void
$this->assertSame(11, $list->files[0]->size);
$this->assertNull($list->cursor);
}

public function testWriteSendsContentLength(): void
{
$client = new CapturingClient();
$s3 = new S3(
root: '/root',
accessKey: 'test-key',
secretKey: 'test-secret',
host: 'https://s3.example.com',
region: 'us-east-1',
client: $client,
);

$s3->write('file.txt', new Stream('hello world'), 'text/plain');

$this->assertCount(1, $client->requests);
$request = $client->requests[0];
$this->assertSame('11', $request->getHeaderLine('content-length'));
$this->assertStringContainsString('content-length', $request->getHeaderLine('authorization'));
}

public function testMultipartUploadSendsContentLengthForEveryRequest(): void
{
$client = new CapturingClient();
$s3 = new S3(
root: '/root',
accessKey: 'test-key',
secretKey: 'test-secret',
host: 'https://s3.example.com',
region: 'us-east-1',
client: $client,
);

$metadata = [];
$s3->upload(new Stream('aaaaaaaaaa'), 'file.bin', 'application/octet-stream', 1, 2, $metadata);
$s3->upload(new Stream('bbbbbb'), 'file.bin', 'application/octet-stream', 2, 2, $metadata);

// createMultipartUpload (POST, empty body), two uploadPart PUTs,
// exists() probe (HEAD), completeMultipartUpload (POST, XML body)
$this->assertCount(5, $client->requests);
$this->assertSame('0', $client->requests[0]->getHeaderLine('content-length'));
$this->assertSame('10', $client->requests[1]->getHeaderLine('content-length'));
$this->assertSame('6', $client->requests[2]->getHeaderLine('content-length'));
$this->assertSame('0', $client->requests[3]->getHeaderLine('content-length'));
$completeLength = (int) $client->requests[4]->getHeaderLine('content-length');
$this->assertGreaterThan(0, $completeLength);
$this->assertSame(\strlen((string) $client->requests[4]->getBody()), $completeLength);
Comment on lines +751 to +760

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Tests mirror request internals

This test mirrors the implementation's exact five-request sequence, ordering, and body lengths instead of testing observable endpoint behavior. That violates the repository directive against implementation-coupled tests. The same coupling appears in the authorization-string assertion and the unknown-size-stream header assertion. More importantly, the capturing fake bypasses the real cURL framing behavior, allowing the transport regression to pass. This repository requirement must be satisfied before merging; replace these checks with a protocol-level test using the actual adapter and a strict endpoint.

Context Used: Call out and harshly judge implementation-coupled ... (source)

Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/storage/tests/Storage/Device/S3Test.php
Line: 751-760

Comment:
**Tests mirror request internals**

This test mirrors the implementation's exact five-request sequence, ordering, and body lengths instead of testing observable endpoint behavior. That violates the repository directive against implementation-coupled tests. The same coupling appears in the authorization-string assertion and the unknown-size-stream header assertion. More importantly, the capturing fake bypasses the real cURL framing behavior, allowing the transport regression to pass. This repository requirement must be satisfied before merging; replace these checks with a protocol-level test using the actual adapter and a strict endpoint.

**Context Used:** Call out and harshly judge implementation-coupled ... ([source](https://app.greptile.com/review/custom-context?memory=instruction-0))

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Claude Code Fix in Codex

}

public function testContentLengthIsMeasuredForSeekableStreamWithUnknownSize(): void
{
$client = new CapturingClient();
$s3 = new S3(
root: '/root',
accessKey: 'test-key',
secretKey: 'test-secret',
host: 'https://s3.example.com',
region: 'us-east-1',
client: $client,
);

$s3->write('file.txt', new UnknownSizeStream('hello world'), 'text/plain');

$this->assertSame('11', $client->requests[0]->getHeaderLine('content-length'));
}
}