[Bug] HadoopFileIO does not close per-instance FileSystems, causing executor thread growth with disabled Hadoop FS cache

[zhangstar333]

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

• Paimon: 1.1.1 in the affected deployment
• Hadoop FileSystem: Aliyun OSS Hadoop FileSystem
• Access pattern: many concurrent Paimon scan splits
• Hadoop FS cache: disabled to isolate credentials

Compute Engine

java

Minimal reproduce step

We observed unbounded JVM thread growth when Paimon reads object storage through HadoopFileIO in an embedded compute-engine JVM.

The issue is reproducible when all of the following are true:

1. The engine creates/deserializes a Paimon Table (and thus a HadoopFileIO) for many scan splits.

2. Hadoop FileSystem cache is disabled, for example:

   fs.oss.impl.disable.cache=true
   
   

This configuration is commonly required to avoid credential reuse between different AK/SK or STS tokens.
3. The underlying Hadoop FileSystem implementation creates executor threads. In our case it is:

org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem

Observed behavior

HadoopFileIO maintains a per-instance map:

Map<Pair<String, String>, FileSystem> fsMap;

When the Hadoop global FileSystem cache is disabled, each HadoopFileIO instance creates its own FileSystem instance through
path.getFileSystem(conf).

However, HadoopFileIO does not override close(). Therefore, calling:

table.fileIO().close();

does not close the FileSystem instances because FileIO.close() is currently a no-op by default.

For OSS, each unclosed AliyunOSSFileSystem may retain transfer/copy executor threads. With many scan splits, the JVM thread count keeps
increasing and can eventually reach thousands or more.

We checked Paimon 1.4.2, the release-1.4 branch, and current master; HadoopFileIO still has fsMap but does not close its FileSystem
instances.

What doesn't meet your expectations?

When a HadoopFileIO instance is no longer used, its owned non-shared FileSystem instances should be released so that connector-specific
resources, including executor threads, are shut down.

Question / proposal

Would the Paimon community consider implementing resource cleanup in HadoopFileIO.close()?

Conceptually, it would close and clear the FileSystem instances stored in the instance-local fsMap.

There are two design questions we would appreciate guidance on:

1. Is HadoopFileIO intended to own and close the FileSystem instances in its fsMap?
2. How should cleanup behave when Hadoop global FileSystem cache is enabled, so that closing one HadoopFileIO does not unexpectedly affect
   another user of a shared cached FileSystem?

A Paimon-level fix would help all compute engines using HadoopFileIO with disabled Hadoop FS cache, not only our embedded-engine scenario.

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!