SCM_RIGHTS issue for Windows in TransferSocket class

[EdmondDantes]

It seems that socket passing algorithm is not possible for the Windows platform, as Windows does not support constants like SCM_RIGHTS. If I understand correctly, is there another approach that needs to be used here? Or perhaps it's not necessary to pass the socket in this way at all?

OS: Windows 11
PHP: PHP 8.3.4 (cli)

Log:
The transfer socket threw an exception: Undefined constant "SCM_RIGHTS"::C:\work\ct\cluster\vendor\amphp\cluster\src\Internal\StreamResourceReceivePipe.php:72
Failed sending request to bind server socket: Sending on the channel failed. Did the context die?::C:\work\ct\cluster\vendor\amphp\cluster\src\ServerSocketPipeFactory.php:65
The transfer socket threw an exception: Undefined constant "SCM_RIGHTS"::C:\work\ct\cluster\vendor\amphp\cluster\src\Internal\StreamResourceReceivePipe.php:72
The transfer socket threw an exception: Undefined constant "SCM_RIGHTS"::C:\work\ct\cluster\vendor\amphp\cluster\src\Internal\StreamResourceReceivePipe.php:72

[EdmondDantes]

I figured it out. Windows requires a different approach to implement something similar. I tried to do it using your library as a base. It turned out pretty well!!!

I had to change the architecture, and unlike UNIX, Win64 uses a process that accepts the connection and forwards it to another process, not the original socket being listened to, but the socket of the client connection.

But the overall performance of this solution is also remarkable!!! Compared to Swoole, it loses only about 15%. And this is C++ code we're talking about.

We can try it here: https://github.com/EdmondDantes/ampCluster

[kelunik]

@EdmondDantes Do you have a code sample for the original issue? IIRC, socket transfer has mainly been developed for macOS, because it doesn't support multiple processes binding to the same address / port in load balancing mode. On macOS, always the latest bind receives new connections, so it can be used for rolling restarts, but not for load balancing.

Windows does support multiple processes binding to the same address / port in load balancing mode, so socket passing should rarely be necessary.

[EdmondDantes]

@EdmondDantes Do you have a code sample for the original issue?

parent.php

<?php

include_once __DIR__ . '/vendor/autoload.php';

use Amp\CancelledException;
use Amp\Cluster\ClientSocketSendPipe;
use Amp\Cluster\ServerSocketPipeProvider;
use Amp\Parallel\Context\ProcessContextFactory;
use Amp\Parallel\Ipc\LocalIpcHub;
use Amp\SignalCancellation;
use Revolt\EventLoop;
use function Amp\async;
use function Amp\Socket\listen;

$ipcHub = new LocalIpcHub();

$serverProvider = new ServerSocketPipeProvider();

// Sharing the IpcHub instance with the context factory isn't required,
// but reduces the number of opened sockets.
$contextFactory = new ProcessContextFactory(ipcHub: $ipcHub);

$context = $contextFactory->start(__DIR__ . '/child.php');

$connectionKey = $ipcHub->generateKey();
$context->send(['uri' => $ipcHub->getUri(), 'key' => $connectionKey]);

// $socket will be a bidirectional socket to the child.
$socket = $ipcHub->accept($connectionKey);

// Listen for requests for server sockets on the given socket until cancelled by signal.
try {
    $serverProvider->provideFor($socket);
} catch (CancelledException) {
    // Signal cancellation expected.
}

child.php

declare(strict_types=1);

use Amp\Cluster\ClientSocketReceivePipe;
use Amp\Cluster\ServerSocketPipeFactory;
use Amp\Sync\Channel;

return function (Channel $channel): void {
    ['uri' => $uri, 'key' => $connectionKey] = $channel->receive();
    
    // $socket will be a bidirectional socket to the parent.
    $socket = Amp\Parallel\Ipc\connect($uri, $connectionKey);
    
    $serverFactory = new ServerSocketPipeFactory($socket);
    
    // Requests the server socket from the parent process.
    $server = $serverFactory->listen('127.0.0.1:1337');
    
    while ($client = $server->accept()) {
        // Handle client socket in a separate coroutine (fiber).
        \Amp\async(function () use ($client) { /* ... */ });
    }
};

Windows 10.

Windows does support multiple processes binding to the same address / port in load balancing mode, so socket passing should rarely be necessary.

If you try to run the code from the examples,

<?php

require __DIR__ . "/vendor/autoload.php";

use Amp\ByteStream;
use Amp\Cluster\Cluster;
use Amp\Http\Server\Driver\ConnectionLimitingServerSocketFactory;
use Amp\Http\Server\Driver\SocketClientFactory;
use Amp\Http\Server\RequestHandler\ClosureRequestHandler;
use Amp\Http\Server\SocketHttpServer;
use Amp\Log\ConsoleFormatter;
use Amp\Log\StreamHandler;
use Monolog\Logger;

$id = Cluster::getContextId() ?? getmypid();

// Creating a log handler in this way allows the script to be run in a cluster or standalone.
if (Cluster::isWorker()) {
    $handler = Cluster::createLogHandler();
} else {
    $handler = new StreamHandler(ByteStream\getStdout());
    $handler->setFormatter(new ConsoleFormatter());
}

$logger = new Logger('worker-' . $id);
$logger->pushHandler($handler);
$logger->useLoggingLoopDetection(false);

// Cluster::getServerSocketFactory() will return a factory which creates the socket
// locally or requests the server socket from the cluster watcher.
$socketFactory = Cluster::getServerSocketFactory();
$clientFactory = new SocketClientFactory($logger);

try {
    $httpServer = new SocketHttpServer($logger, $socketFactory, $clientFactory);
    $httpServer->expose('127.0.0.1:1337');

    // Start the HTTP server
    $httpServer->start(
        new ClosureRequestHandler(function (): \Amp\Http\Server\Response {
            return new \Amp\Http\Server\Response(\Amp\Http\HttpStatus::OK, [
                "content-type" => "text/plain; charset=utf-8",
            ], "Hello, World!");
        }),
        new \Amp\Http\Server\DefaultErrorHandler(),
    );
    
} catch (\Throwable $e) {
    $logger->error($e->getMessage());
}

// Stop the server when the cluster watcher is terminated.
Cluster::awaitTermination();

$httpServer->stop();

It leads to:
worker-77.error: The transfer socket closed while waiting to receive a socket [] []

Perhaps you're right that I need to create the socket in such a way that it's in SHARED mode between processes. I'll try this approach.

[EdmondDantes]

I try

    $httpServer = new SocketHttpServer($logger, $socketFactory, $clientFactory);
    $bindContext = new \Amp\Socket\BindContext();
    $httpServer->expose('127.0.0.1:1337', $bindContext->withReusePort());

And it's send requests only to first worker