Add .nwb to known Zarr suffixes; add tests for NWB Zarr backend

[CodyCBakerPhD]

fix #1310

This seems to have worked fine for me on a local upload attempt to staging (validation had to be suppressed until we get NeurodataWithoutBorders/nwbinspector#388 and the integration follow-up through)

At least getting the seeds of this PR started

[codecov]

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (72816ab) 88.51% compared to head (a7d16aa) 67.93%.

Files	Patch %	Lines
dandi/tests/fixtures.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #1312       +/-   ##
===========================================
- Coverage   88.51%   67.93%   -20.58%     
===========================================
  Files          77       77               
  Lines       10492    10480       -12     
===========================================
- Hits         9287     7120     -2167     
- Misses       1205     3360     +2155     

Flag	Coverage Δ
unittests	67.93% <80.00%> (-20.58%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[CodyCBakerPhD]

The current CI error

TypeError: Can't instantiate abstract class NWBZarrIO with abstract method can_read

has a known cause and will likely need to await the upcoming release of hdmf_zarr (this week sometime)

[CodyCBakerPhD]

@yarikoptic Trying the main branch (not this one) with .nwb.zarr gave the following error during dandi upload -i dandi-staging to destination https://gui-staging.dandiarchive.org/dandiset/204919/draft/files?location=test_read_nwbfile

501 Server Error: Not Implemented for url: https://dandi-api-staging-dandisets.s3.amazonaws.com/zarr/d...

Worth mentioning it began upload and proceeded find until 99% done when the error happened

log: 20230811152021Z-372.log

[yarikoptic]

eh -- I spent some time trying to see smth in the logs

• your activity came at unfortunate time when I had to relogin the client so I do not have logs for

-rw------- 1 dandi dandi-internal 112444 Aug 11 11:00 20230811-1001.log
-rw------- 1 dandi dandi-internal      0 Aug 11 11:01 20230811-1101.log
-rw------- 1 dandi dandi-internal      0 Aug 11 12:01 20230811-1201.log
-rw------- 1 dandi dandi-internal      0 Aug 11 12:20 20230811-1220.log
-rw------- 1 dandi dandi-internal  45857 Aug 11 12:37 20230811-1224.log

today

• in papertrail -- nothing informative if searched by dd868c2b-0a3b-44cb-afdb-eb62f07a701b , nothing for "implemented" or "exception" (only older ones). @dandi/archive-admin might have better skills/tools to see if anything in the logs.

or if you retry again -- I might spot smth in the fresh dumps for heroku which I should be getting now.

[yarikoptic]

The current CI error

TypeError: Can't instantiate abstract class NWBZarrIO with abstract method can_read

has a known cause and will likely need to await the upcoming release of hdmf_zarr (this week sometime)

FWIW, in dev-deps CI run we see the same Error and we do

        pip install git+https://github.com/jaraco/keyring
        pip install git+https://github.com/NeurodataWithoutBorders/nwbinspector
        pip install git+https://github.com/NeurodataWithoutBorders/pynwb

so we do not install development hdmf as well. Do you think we should?

[CodyCBakerPhD]

I just tried again (from the save environment) and did not get the previous error

Asset is now visible: https://gui-staging.dandiarchive.org/dandiset/204919/draft/files?location=test_read_nwbfile

[CodyCBakerPhD]

BTW the 'environment' I'm using right now is just the default/base on the DANDI Hub

[yarikoptic]

so we do not install development hdmf as well. Do you think we should?

trying in #1320 and it leads to an error (posted there)

I just tried again (from the save environment) and did not get the previous error

well, would need 1 more fresh .nwb.zarr I guess ;-)

[yarikoptic]

Let's make it more flexible and up to date

[yarikoptic]

I think we ended up with .nwb.zarr, right @rly ?

[yarikoptic]

Let's make it more explicit

Suggested change

	backend: Literal["hdf5", "zarr"] = "hdf5",
	backend: Literal["auto", "hdf5", "zarr"] = "auto",

with "auto" choosing based on extension.