Repository navigation
Default cache sync to the shared repo and make filters optional - #129
Conversation
| task: str | None, | ||
| model_spec: str | None, | ||
| ) -> bool: | ||
| if len(parts) != 5 or parts[0] != kind: |
There was a problem hiding this comment.
What is the purpose of the length check?
There was a problem hiding this comment.
Cache folders are always kind/task/provider/model/descriptor_hash so 5 parts means the path length is correct. I'll make it self-explanatory by unpacking the parts instead of a bare 5.
There was a problem hiding this comment.
I think this is a bit hard coded. I would prefer to write some structure that verifies this easily. Perhaps you can create some class like
class CacheSuffix:
kind : Literal["VLLM","OpenRouter"...]
task: ...
provider : ...
model : ...
...
which may or may not inherit some type of os Path class (not sure). Then we can just use this structure instead of parts. Do you think this is an over-engineering? I would love to hear your opinions about it.
I think the problem is if we want to remove provider, then we need to change this everywhere in the code; like we have to update len(parts) == 4 and so on. This way we can get rid of these hard codings
There was a problem hiding this comment.
That's a good idea. I will try it out to see if it gets complex or actually simplifies things.
There was a problem hiding this comment.
Went with a frozen CacheFolder dataclass in sqlite.py that both builds and parses the relative path, so the layout is defined once. I didn't subclass Path, since remote HF paths are PurePosixPath and local ones are Path.
Additionally, I made local and remote listing share one filter, and kind as an optional filter same as task/model. See d7a93fd.
Lmk what you think.
Define the kind/task/provider/model/descriptor_hash layout once in a CacheFolder dataclass that both builds and parses relative paths, so the HF sync no longer hard-codes path depth or segment positions. Local and remote listing share one filter, and kind becomes an optional filter like task and model, which removes the per-kind loop in the CLI.
2 participants
Description
Makes
judgearena-cacheusable without spelling out every part of the cache path.--hf_repoto the publicjudge-arena/judge-arena-cachedataset, so anyone can fetch without a token while pushing still needs write access to thejudge-arenaorg.--kind,--taskand--modeloptional filters, the omitted ones are matched from the cache folder paths.--allwhen no filter is given so a bare command does not sync everything by accident, and rejects--allcombined with filters.Tested push and fetch against the shared repo through the default, and an anonymous
fetch --all.