Repository navigation
feat(fs): expose a queued fs API, remove queueFileOperation() (AI-assisted) - #12601
Merged
Merged
Conversation
…isted) @docusaurus/fs now exposes the subset of node:fs/promises that Docusaurus uses. Operations keeping a file open (readFile, writeFile) go through the internal queue bounding concurrently open files, other operations are called directly. Call sites don't need to queue operations themselves anymore, which also removes the risk of deadlocks with nested queueFileOperation() calls. The concurrency can be set with the secret DOCUSAURUS_FS_CONCURRENCY env var. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
✅ [V2]
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Size Change: 0 B Total Size: 12.4 MB ℹ️ View Unchanged
|
2 of 3 tasks
⚡️ Lighthouse report for the deploy preview of this PR
|
|
Size Change: 0 B Total Size: 12.6 MB ℹ️ View Unchanged
|
slorber
added
the
pr: ignore
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pre-flight checklist
Motivation
Follow-up of #12560 and #12561: move the queue inside
@docusaurus/fs, so that it bounds file system operations without call sites having to remember it.@docusaurus/fsnow exposesfs, the subset ofnode:fs/promisesthat Docusaurus uses (readFile,writeFile,access,cp,lstat,mkdir,mkdtemp,readdir,realpath,rename,rm,stat,constants).readFile(),writeFile()) go through the queue. That's what causedEMFILEerrors in my Node 24.21 measurements (see feat(fs): add queueFileOperation() to bound concurrently open files (AI-assisted) #12561). The other operations are a single system call each, with no file kept open, so they're called directly. Queueing them wouldn't preventEMFILE, and cheap calls such as the manypathExists()checks of the MDX loader would wait behind slow reads.outputFile()andreadJSON()helpers use the queuedfs.queueFileOperation()is removed, along with its 7 call sites: they now usefsfrom@docusaurus/fs.@docusaurus/fshas only been released as a canary so far. This also removes the deadlock risk of nestedqueueFileOperation()calls: only leaf operations take a queue slot.DOCUSAURUS_FS_CONCURRENCYenv variable to change the default concurrency (100), likeDOCUSAURUS_GIT_COMMAND_CONCURRENCYandDOCUSAURUS_SSG_CONCURRENCY.Next PR in the stack: migrate the remaining
node:fsimports and forbid them with a lint rule, so that all file system operations go through@docusaurus/fs.Test Plan
fsforwards arguments and results tonode:fs/promises,readFile()concurrency is bounded at 100, unqueued operations (stat(),readdir()) still complete while the queue is full, and env variable parsing.node:fs/promises:@docusaurus/fscalls it through the module object at call time.ulimit -n 1024on Node 24.21: ✅ noEMFILEerror, 55s.pnpm test,pnpm lint:knip,pnpm lint:deps,pnpm lint:spelling.Test links
Deploy preview: https://deploy-preview-12601--docusaurus-2.netlify.app/
No user-facing change.
Related issues/PRs
@docusaurus/fsand the queue), refactor: replace fs-extra with node:fs/promises and @docusaurus/fs in published packages (AI-assisted) #12558 (fs-extra migration)(AI-assisted)
🤖 Generated with Claude Code