backtest --data-mode=m1-csv / tick-csv never runs on Linux: CSV handle sharing P/Invokes kernel32.dll (DllNotFoundException), cBot host times out

[DaweTackos]

Summary

In the Linux console image, backtest (and optimize) with --data-mode=m1-csv or --data-mode=tick-csv never delivers a single bar to the cBot. The bot instance starts, then after 3–5 minutes the console reports process stopped responding and was terminated (sometimes aborted by timeout) and writes a report with empty values. The identical command with --data-mode=m1 or --data-mode=open finishes in seconds.

Root cause (from a runtime exception trace and the decompiled assemblies): when answering the bot host's symbol subscription, the console shares the memory-mapped CSV with the out-of-process bot host through SystemCalls.SetInheritable / SystemCalls.DuplicateHandle, which are [DllImport("kernel32.dll")]. On Linux this throws System.DllNotFoundException, the handler dies, the reply is never sent, and the bot host times out.

Environment

• Image: ghcr.io/spotware/ctrader-console tags 5.4, 5.5.17, 5.6.8, 5.7.10, 5.9.11, 5.10.1, 5.10.2-alpha and latest (all reproduce)
• Platform: linux/arm64 (Docker Desktop on macOS), also reproduced under linux/amd64
• .NET runtime in the image: 10.0.9 (5.10.1)
• Accounts: a live account and a Spotware demo account (both reproduce); symbols EURUSD and BTCUSD; periods m1 and m5
• Bots: a generated bot, a trivial "buy once in OnBar then Stop()" bot, and a minimal bot that only prints in OnStart (OnStart is never reached)

Steps to reproduce

1. Create data.csv in the documented M1 format, covering 2024-01-02 and 2024-01-03 completely (no header):

   2024.01.02,00:00,1.10366,1.10376,1.10366,1.10374,102
   2024.01.02,00:01,1.10371,1.10375,1.10368,1.10375,74
   ...
   2024.01.03,23:59,1.09253,1.09260,1.09253,1.09258,17
   

2. Run:

   docker run --rm -v $PWD:/work ghcr.io/spotware/ctrader-console:5.10.1 \
     backtest /work/AnyBot.algo --ctid=... --pwd-file=/work/pwd --account=... \
     --symbol=EURUSD --period=m5 --start=02/01/2024 --end=03/01/2024 --balance=10000 \
     --data-mode=m1-csv --data-file=/work/data.csv --report-json=/work/report.json --exit-on-stop
   

3. Compare with the same command using --data-mode=m1.

Expected

The backtest runs on the CSV bars and produces a report, as with --data-mode=m1.

Actual

Logged in.
Starting cBot...
Progress | Backtesting | 0.00 % |
Info | CBot instance [..., EURUSD, m5] started.
Error | CBot instance [..., EURUSD, m5] process stopped responding and was terminated.
{
      "Equity":,
      "NetProfit":-,
      ...

followed by System.InvalidOperationException: Message expected at ...BacktestReportSavingStateStrategy.DoEnter().

Same result with tick-csv (Date,Time,Bid,Ask file), with --spread, with --data-dir, with --cap-add SYS_PTRACE, with --security-opt seccomp=unconfined, with the CSV copied inside the image filesystem, and with a file containing a week of history before --start.

Diagnosis

Managed stacks of both processes during the stall (dotnet-stack):

• bot host (algohost.netcore), main dispatcher thread:
  SmallBacktestingRobotInstanceBase.InitializeBacktestingInstance → SmallBacktestingMarketData.GetChartSource → SmallSymbol.TrySubscribe → SmallSilentBacktestingMarketDataController.TrySubscribe → AutomateTargetRouter.SendAndWait (waiting for SilentBacktestingSubscribeSymbolResMessage)
• console: AutomateArenaThreadDispatcher.DispatcherLoop idle; no thread doing any CSV work; the CSV file is not open in either process.

Exception events of the console process recorded with
DOTNET_EnableEventPipe=1 DOTNET_EventPipeConfig=Microsoft-Windows-DotNETRuntime:0x8000:5:

System.DllNotFoundException: Unable to load shared library 'kernel32.dll' or one of its dependencies. ...
  /app/kernel32.dll: cannot open shared object file: No such file or directory
  /app/libkernel32.dll.so: cannot open shared object file: No such file or directory
  ... (2 occurrences)

Code path (decompiled from 5.10.1):

• cTrader.Automate.Instances.Backtesting.Controllers.MarketData.BigSilentBacktestingMarketDataController.HandleAsync(SilentBacktestingSubscribeSymbolReqMessage)
  → LoadAsync(...) (succeeds)
  → BacktestingCsvDataSourceMapService.MapIfNeeded(dataSource, _sharedResources)
  → MemoryMappedFile.CreateFromFile(csvFilePath) then OutProcSharedResources.ShareHandle(handle)
  → SystemCalls.SetInheritable(...) / SystemCalls.DuplicateHandle(...)
  → [DllImport("kernel32.dll")] SetHandleInformation / DuplicateHandle → DllNotFoundException
• SystemCalls is the only ISystemCalls export; cTrader.Automate.Infrastructure.Linux.dll and cTrader.Console.Infrastructure.Linux.dll do not provide a Linux implementation.
• Because the exception escapes HandleAsync after the LoadAsync try/catch, _router.Send(new SilentBacktestingSubscribeSymbolResMessage(...)) is never reached.

Second, related issue: silent hang before the bot starts

If the CSV does not cover the whole requested range by calendar day (e.g. --end=04/01/2024 with a file whose last bar is 2024.01.03,23:59), or the file cannot be parsed at all (e.g. a file containing this,is,not,a,csv), the process hangs forever after Logged in. without any message. The failed BacktestingCsvDataSourceService.GetMinMaxDateAsync / CsvBarsReadFailed result is never reported. Only a nonexistent --data-file is rejected (Can't find file specified for --data-file parameter!). Please surface these errors and exit.

Suggested fix

Provide a Linux implementation of ISystemCalls (or avoid handle duplication for the CSV mapping, e.g. let the bot host open the CSV path itself, since it already receives the path), and make CSV load failures fail fast with a message.

[amusleh-spotware-com]

Thanks a lot for the detailed report and the diagnosis — you were right on both points, and we could reproduce everything.

A fix is available for testing in a new alpha image (amd64 and arm64):

docker pull ghcr.io/spotware/ctrader-console:5.10.4-alpha

What changed:

• backtest and optimize with --data-mode=m1-csv and --data-mode=tick-csv now run on Linux. The CSV is no longer shared through the Windows-only handle calls; the bot host reads the file directly.
• A CSV that cannot be read now stops right away with Failed to load historical data: <reason> and exit code 93, instead of hanging after Logged in.
• A --start/--end outside the file's date range now prints No historical data for the specified period and exits with code 86.

Could you run your original commands against 5.10.4-alpha and let us know whether it works for you?

[DaweTackos]

Perfect thank you, confirmed on 5.10.4-alpha (linux/arm64): --data-mode=m1-csv and tick-csv now deliver every bar; the original 2-day command finishes in about a second, and a 689 MB / 12.3M-row m1 file covering 2003–2026 backtests 2004→2025 in 15 s. Exit codes 86 and 93 behave as described. Good job, thank you. :)

One remaining issue: when --end lies before the broker server's own history for the symbol (e.g. --end=04/06/2017 on an account whose history starts in 2018), the run still dies with System.InvalidOperationException: Message expected in BacktestReportSavingStateStrategy.DoEnter() — the bot never starts. It does so regardless of the CSV contents or the account's deposit currency (same for EURGBP on a GBP account). A clean error like the new 86/93 ones would help; better still, the server lookup at --end could be skipped when the CSV covers the range.

[amusleh-spotware-com]

Remainig issue with --end also resolved, it will be part of next stable release.

[DaweTackos]

Nice, thanks so much for fast reply and fixes, good job.

When i can aproxiamtly expect next stable release? Thanks