Skip to content

Opening a zarr behind a S3Config as YAXArray fails #69

Description

@felixcremer

When I try to open a zarr that needs credentials and a config as a YAXArray this fails because open_dataset tries to open the zarr with a store string and an empty path.
This is on current main of everything with zarr.jl at 0.11
This failed differently on previous versions but I think that worked at some point.

using Zarr
using YAXArrays
using AWSS3

cfg = AWSS3.AWS.AWSConfig(creds=nothing, region="us-west-2")
z = AWSS3.AWS.with_aws_config(cfg) do # This works
         zopen("s3://mur-sst/zarr-v1")
       end

yax = AWSS3.AWS.with_aws_config(cfg) do # This fails
         open_dataset("s3://mur-sst/zarr-v1")
       end

z = AWSS3.AWS.with_aws_config(cfg) do # This fails and is what open_dataset tries to open the zarr
         zopen("s3://mur-sst/zarr-v1", "")
       end

# This fails because of missing credentials
       zwocfg = zopen("s3://mur-sst/zarr-v1")

@time yaxz = open_dataset(z) # This works but needs 13 seconds with 13 lock conflicts
Error Message

```julia julia> yax = AWSS3.AWS.with_aws_config(cfg) do open_dataset("s3://mur-sst/zarr-v1") end ERROR: ArgumentError: Specified store S3 Object Storage with: config: AWS.AWSConfig(nothing, "us-west-2", "json", 3) bucket: mur-sst in path is neither a ZArray nor a ZGroup in a recognized zarr format. Stacktrace: [1] ZarrCore.ZarrFormat(s::S3Store, path::String) @ ZarrCore ~/Documents/scratch/zarrs3/dev/Zarr/ZarrCore/src/ZGroup.jl:45 [2] #zopen#99 @ ~/Documents/scratch/zarrs3/dev/Zarr/ZarrCore/src/ZGroup.jl:109 [inlined] [3] kwcall(::@NamedTuple{path::String, fill_as_missing::Bool}, ::typeof(zopen), s::S3Store,mode::String) @ ZarrCore ~/Documents/scratch/zarrs3/dev/Zarr/ZarrCore/src/ZGroup.jl:102 [4] zopen(s::String, mode::String; kwargs::@kwargs{fill_as_missing::Bool, path::String}) @ ZarrCore ~/Documents/scratch/zarrs3/dev/Zarr/ZarrCore/src/ZGroup.jl:129 [5] zopen @ ~/Documents/scratch/zarrs3/dev/Zarr/ZarrCore/src/ZGroup.jl:127 [inlined] [6] ZarrExt.ZarrDataset(g::String; mode::String, path::String, kwargs::@kwargs{}) @ ZarrExt ~/Documents/scratch/zarrs3/dev/YAXArrayBase/ext/ZarrExt.jl:27 [7] ZarrDataset @ ~/Documents/scratch/zarrs3/dev/YAXArrayBase/ext/ZarrExt.jl:18 [inlined] [8] to_dataset(T::Type{ZarrExt.ZarrDataset}, g::String; kwargs::@kwargs{}) @ YAXArrayBase ~/Documents/scratch/zarrs3/dev/YAXArrayBase/src/datasets/datasetinterface.jl:101 [9] to_dataset @ ~/Documents/scratch/zarrs3/dev/YAXArrayBase/src/datasets/datasetinterface.jl:98 [inlined] [10] open_dataset(g::String; skip_keys::Tuple{}, driver::Symbol, force_datetime::Bool, path::Nothing) @ YAXArrays.Datasets ~/Documents/scratch/zarrs3/dev/YAXArrays/src/DatasetAPI/Datasets.jl:464 [11] open_dataset @ ~/Documents/scratch/zarrs3/dev/YAXArrays/src/DatasetAPI/Datasets.jl:462 [inlined] [12] #41 @ ./REPL[36]:2 [inlined] [13] with_aws_config(f::var"#41#42", config::AWS.AWSConfig) @ AWS ~/.julia/packages/AWS/Igcxn/src/AWSConfig.jl:166 [14] top-level scope @ REPL[36]:1

</p>
</details> 

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions