Skip to content

Add a contents_first option #19

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
BaerLKR wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Add a contents_first option #19

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
43228fa
feat: add a content_first option
BaerLKR Sep 27, 2026
341ac10
refactor: a typestate guard to prohibit loss of filter without notice
BaerLKR Sep 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "async-walkdir"
version = "2.2.0"
version = "2.2.1"
Comment thread
BaerLKR marked this conversation as resolved.
authors = ["Ririsoft <riri@ririsoft.com>"]
edition = "2021"
description = "Asynchronous directory traversal for Rust."
Expand Down
222 changes: 196 additions & 26 deletions src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@
mod error;

use std::future::Future;
use std::marker::PhantomData;
use std::path::{Path, PathBuf};
use std::pin::Pin;
use std::task::{Context, Poll};
Expand Down Expand Up @@ -112,9 +113,22 @@ type BoxStream = futures_lite::stream::Boxed<Result<DirEntry>>;
/// # Panics
///
/// Panics if the directories depth overflows `usize`.
pub struct WalkDir {
pub struct WalkDir<State = Unfiltered> {

@ririsoft ririsoft Sep 28, 2026

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a breaking change, contrary to the PR description. Code compiling on 2.1.0 no longer compiles:

fn build(p: &str) -> WalkDir {
    WalkDir::new(p).filter(|_| async { Filtering::Continue })
}
// error[E0308]: expected `WalkDir<Unfiltered>`, found `WalkDir<Filtered>`

The same applies to struct fields typed WalkDir, wd = wd.filter(..) reassignments, and calling .filter() twice.

The typestate is only needed because the stream is built eagerly in each builder method. Suggestion: store the options and the filter in WalkDir (the filter can be boxed as Box<dyn FnMut(DirEntry) -> BoxedFut<Filtering> + Send>, a type already used in this file), and build the stream lazily on the first poll_next. Then contents_first and filter can be called in any order, and the generic parameter, the Unfiltered / Filtered markers and the State: Unpin bound can all be removed.

@BaerLKR BaerLKR Sep 28, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried using the file and it resulted in API change for me, but maybe I missed something. Will try that approach.

Regarding the breaking change: cargo-semver check said it is fine, but maybe I used it wrong. The desire to not have breaking changes was the reason for all the typestate shenanigans but oh well
thanks for the feedback

root: PathBuf,
entries: BoxStream,
opts: WalkDirOptions,
_state: PhantomData<State>,
}

/// A marker struct to signal, that the DirWalk is not being filtered
pub struct Unfiltered;
/// A marker struct to snignal, that the DirWalk has a filter applied.
Comment thread
BaerLKR marked this conversation as resolved.
/// This results in some options not being configurable after this point anymore.
pub struct Filtered;

#[derive(Default, Clone, Copy)]
struct WalkDirOptions {
contents_first: bool,
}

/// Sets the filtering behavior.
Expand All @@ -136,26 +150,58 @@ impl WalkDir {
root: root.as_ref().to_owned(),
entries: walk_dir(
root,
WalkDirOptions::default(),
None::<Box<dyn FnMut(DirEntry) -> BoxedFut<Filtering> + Send>>,
),
opts: WalkDirOptions::default(),
_state: PhantomData,
}
}
}
impl WalkDir<Unfiltered> {
/// Yield the directory's content before the directory.
///
/// When `yes` is false (the default), the directory is yielded before the contents are read.
/// This is useful when e.g. you want to skip processing some of the directories.
///
/// When `yes` is `true`, the iterator yields the contents of a directory
Comment thread
BaerLKR marked this conversation as resolved.
/// before yielding the directory itself. This is useful when, e.g. you
/// want to recursively delete a directory.
pub fn contents_first(mut self, yes: bool) -> Self {
let root = self.root.clone();
self.opts.contents_first = yes;
Self {
root: self.root,
entries: walk_dir(
root,
self.opts,
None::<Box<dyn FnMut(DirEntry) -> BoxedFut<Filtering> + Send>>,
),
opts: self.opts,
_state: PhantomData,
}
}

/// Filter entries.
pub fn filter<F, Fut>(self, f: F) -> Self
pub fn filter<F, Fut>(self, f: F) -> WalkDir<Filtered>
where
F: FnMut(DirEntry) -> Fut + Send + 'static,
Fut: Future<Output = Filtering> + Send,
{
let root = self.root.clone();
Self {
WalkDir {
root: self.root,
entries: walk_dir(root, Some(f)),
entries: walk_dir(root, self.opts, Some(f)),
opts: self.opts,
_state: PhantomData,
}
}
}

impl Stream for WalkDir {
impl<State> Stream for WalkDir<State>
where
State: Unpin,
Comment thread
BaerLKR marked this conversation as resolved.
{
type Item = Result<DirEntry>;

fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
Expand All @@ -164,7 +210,20 @@ impl Stream for WalkDir {
}
}

fn walk_dir<F, Fut>(root: impl AsRef<Path>, filter: Option<F>) -> BoxStream
struct DirStackItem {
path: PathBuf,
read_dir: ReadDir,
// holds the directory, while it is being traversed
// only relevant when `contents_first` is set to true
deferred_entry: Option<DirEntry>,
// when `contents_first` is enabled, the file tree
// is traversed in post-order, so all non-directory
// contents of a directory need to be stored in here
// until the child directories are finished traversing
deferred_files: Vec<DirEntry>,

@ririsoft ririsoft Sep 28, 2026

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Post-order only requires directories to be deferred: files have no contents, so they can be yielded as soon as they are encountered (this is what walkdir does with contents_first). Buffering all files of a directory:

  • costs O(n) memory on large flat directories and delays output until the whole directory is read,
  • reverses their order (see deferred_files.pop() below),
  • forces re-polling of the exhausted ReadDir (see below).

Also a typo in the comment above: "cild, directories".

@BaerLKR BaerLKR Sep 28, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought post-order = depth first, because for depth first I think the dirs need to be deferred as well

}

fn walk_dir<F, Fut>(root: impl AsRef<Path>, opts: WalkDirOptions, filter: Option<F>) -> BoxStream
where
F: FnMut(DirEntry) -> Fut + Send + 'static,
Fut: Future<Output = Filtering> + Send,
Expand All @@ -178,9 +237,21 @@ where
Err(InnerError::Io { path: root, source }.into()),
State::Done,
)),
Ok(rd) => walk(vec![(root, rd)], filter).await,
Ok(rd) => {
walk(
vec![DirStackItem {
path: root,
read_dir: rd,
deferred_files: Vec::new(),
deferred_entry: None,
}],
filter,
opts,
)
.await
}
},
State::Walk((dirs, filter)) => walk(dirs, filter).await,
State::Walk((dirs, filter)) => walk(dirs, filter, opts).await,
State::Done => None,
}
},
Expand All @@ -190,57 +261,122 @@ where

enum State<F> {
Start((PathBuf, Option<F>)),
Walk((Vec<(PathBuf, ReadDir)>, Option<F>)),
Walk((Vec<DirStackItem>, Option<F>)),
Done,
}

type UnfoldState<F> = (Result<DirEntry>, State<F>);

// Iterative on purpose: filtered-out entries must not grow the stack (see issue #13).
async fn walk<F, Fut>(
mut dirs: Vec<(PathBuf, ReadDir)>,
mut dirs: Vec<DirStackItem>,
mut filter: Option<F>,
opts: WalkDirOptions,
) -> Option<UnfoldState<F>>
where
F: FnMut(DirEntry) -> Fut + Send + 'static,
Fut: Future<Output = Filtering> + Send,
{
loop {
let (path, dir) = dirs.last_mut()?;
let entry = match dir.next().await {
let res = {
let item = dirs.last_mut()?;
item.read_dir.next().await
};

let entry = match res {
Some(Ok(entry)) => entry,
Some(Err(source)) => return io_error(path.to_path_buf(), source, dirs, filter),
Some(Err(source)) => {
let path = dirs.last().unwrap().path.clone();
return io_error(path, source, dirs, filter);
}
None => {
dirs.pop();
continue;
let Some(mut popped) = dirs.pop() else {
unreachable!()
};
if let Some(deferred_file) = popped.deferred_files.pop() {
dirs.push(popped);
return Some((Ok(deferred_file), State::Walk((dirs, filter))));
} else if let Some(deferred_dir) = popped.deferred_entry.take() {
return Some((Ok(deferred_dir), State::Walk((dirs, filter))));
} else {
continue;
}
}
};

let ft = match entry.file_type().await {
Ok(ft) => ft,
Err(source) => return io_error(entry.path(), source, dirs, filter),
};

let filtering = match filter.as_mut() {
Some(filter) => filter(entry.clone()).await,
None => Filtering::Continue,
};
if ft.is_dir() && filtering != Filtering::IgnoreDir {
let path = entry.path();
let rd = match read_dir(&path).await {
Ok(rd) => rd,
Err(source) => return io_error(path, source, dirs, filter),
};
dirs.push((path, rd));
}
if filtering == Filtering::Continue {
return Some((Ok(entry), State::Walk((dirs, filter))));

match filtering {
Filtering::IgnoreDir => continue,
Filtering::Ignore => {
if ft.is_dir() {
let path = entry.path();
let rd = match read_dir(&path).await {
Ok(rd) => rd,
Err(source) => return io_error(path, source, dirs, filter),
};
dirs.push(DirStackItem {
path,
read_dir: rd,
deferred_entry: None,
deferred_files: Vec::new(),
});
}
continue;
}
Filtering::Continue => {
if ft.is_dir() {
let path = entry.path();
let rd = match read_dir(&path).await {
Ok(rd) => rd,
Err(source) => return io_error(path, source, dirs, filter),
};

let deferred_entry = if opts.contents_first {
Some(entry.clone())
} else {
None
};

dirs.push(DirStackItem {
path,
read_dir: rd,
deferred_entry,
deferred_files: Vec::new(),
});

if opts.contents_first {
continue;
} else {
return Some((Ok(entry), State::Walk((dirs, filter))));
}
} else {
if opts.contents_first {
if let Some(last) = dirs.last_mut() {
last.deferred_files.push(entry);
}
continue;
} else {
return Some((Ok(entry), State::Walk((dirs, filter))));
}
}
}
}
}
}

fn io_error<F>(
path: PathBuf,
source: std::io::Error,
dirs: Vec<(PathBuf, ReadDir)>,
dirs: Vec<DirStackItem>,
filter: Option<F>,
) -> Option<UnfoldState<F>> {
let err = InnerError::Io { path, source }.into();
Expand Down Expand Up @@ -390,6 +526,40 @@ mod tests {
Ok(())
})
}
#[test]
fn contents_first() -> Result<()> {

@ririsoft ririsoft Sep 28, 2026

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: a blank line is missing before #[test].

This test only has a single file per directory, so it would not catch the reversed order. Please also add tests for:

  • contents_first(true) combined with filter, with Ignore and IgnoreDir on directories,
  • multiple files in the same directory (compare sorted or as a set, rather than relying on read_dir order),
  • contents_first(false) as a no-op.

block_on(async {
let root = tempfile::tempdir()?;
let f1 = root.path().join("f1.txt");
let d1 = root.path().join("d1");
let f2 = d1.join("f2.txt");
let d2 = d1.join("d2");
let f3 = d2.join("f3.txt");

async_fs::create_dir_all(&d2).await?;
async_fs::write(&f1, []).await?;
async_fs::write(&f2, []).await?;
async_fs::write(&f3, []).await?;

let want = vec![
f3.to_owned(),
d2.to_owned(),
f2.to_owned(),
d1.to_owned(),
f1.to_owned(),
];
let mut wd = WalkDir::new(root.path()).contents_first(true);

let mut got = Vec::new();
while let Some(entry) = wd.next().await {
let entry = entry.unwrap();
got.push(entry.path());
}
assert_eq!(got, want);

Ok(())
})
}

#[test]
fn filter_dirs_no_traverse() -> Result<()> {
Expand Down