Repository navigation
feat(fs): add queueFileOperation() to bound concurrently open files (AI-assisted) - #12561
Merged
Merged
Conversation
2 of 3 tasks
slorber
added
the
Argos
✅ [V2]
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Size Change: 0 B Total Size: 12.4 MB ℹ️ View Unchanged
|
⚡️ Lighthouse report for the deploy preview of this PR
|
|
Size Change: 0 B Total Size: 12.6 MB ℹ️ View Unchanged
|
2 of 3 tasks
|
The latest updates on your projects. Learn more about Argos notifications ↗︎
|
slorber
force-pushed
the
slorber/queue-file-operation
branch
from
October 2, 2026 11:14
ed027c5 to
da09191
Compare
2 of 3 tasks
slorber
force-pushed
the
slorber/queue-file-operation
branch
2 times, most recently
from
October 8, 2026 15:52
f85e222 to
2b7cf8e
Compare
slorber
changed the title
…AI-assisted)
Large sites open a file for each of their thousands of docs/blog posts
concurrently. Bound how many files we keep open at once through a shared
queue, so that we don't depend on graceful-fs (fs-extra) retrying operations
failing with EMFILE ('too many open files').
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
slorber
force-pushed
the
slorber/queue-file-operation
branch
from
October 9, 2026 12:54
2b7cf8e to
bf2ebea
Compare
2 of 3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pre-flight checklist
Motivation
Add a small
queueFileOperation()helper to@docusaurus/fs: a sharedp-queue(concurrency 100) that bounds how many files Docusaurus keeps open at once. Use it where we open one file per content file of the site in an unboundedPromise.all:generate():createData()writes a data file per doc/blog post, plus route props modules. It's wrapped as a whole, since it's the single choke point for all plugins.write-heading-idsand translation extraction: reading every Markdown/source fileThis is the same approach as the git command queue (#10915, #10348). There's no monkey-patching and no retries.
Why
Today we're protected by
fs-extra, which usesgraceful-fs: it queues and retries operations failing withEMFILE("too many open files").node:fsdoesn't, so this PR is a prerequisite for #12558, which migrates tonode:fs/promises.Measurements:
Promise.allwith the fd limit lowered to 256: on Node 24.21,fs-extrasurvives 20,000 concurrentreadFile/writeFilecalls, whilenode:fs/promisesfails withEMFILEat 300 files (reads and writes). On Node 26, reads survive but writes still fail.ulimit -n 1024on Node 24.21:main(fs-extra): ✅ 242sEMFILE: too many open files, open '.../docs/section-2/doc-1009.md'inreadDocFileNode raises its soft fd limit to the hard limit on startup, so typical macOS/Linux machines have high limits. But low limits exist (containers, CI, Windows), and large sites (e.g. generated API docs) have thousands of files.
Stack
Removing
fs-extrafrom the monorepo, split into stacked PRs:@docusaurus/fspackage with fs helpersqueueFileOperation()to bound concurrently open filesnode:fs/promises, remove theirfs-extradependencyfs-extrafrom website, admin scripts and rootTest Plan
Test links
Deploy preview: https://deploy-preview-12561--docusaurus-2.netlify.app/
No user-facing change.
(AI-assisted)
🤖 Generated with Claude Code