Skip to content

added config for prepostshield experiments (Khan^MouseMIND) - #51

Open
akhanf wants to merge 18 commits into
mainfrom
prepostshield
Open

akhanf wants to merge 18 commits into
mainfrom
prepostshield

Conversation

@akhanf

@akhanf akhanf commented Jan 30, 2026

Copy link
Copy Markdown
Member
  • heuristic still needs editing..

@akhanf

akhanf commented Jan 30, 2026

Copy link
Copy Markdown
Member Author

@RicardoRios46

Copy link
Copy Markdown

Oh happy to help with heuristic. Also havent tried this repo Ill try to get involved soon

@Lucas-Arcamone

Copy link
Copy Markdown

Conversion + post-processing script for the full overnight 15T protocol

heuristics/exvivo_15T.py (new)

Naming choices

global

  • sub-_ses-exvivo15T

dwi

  • acq-multidwienc — multi dwi encoding, to avoid confusion with the encoding direction
  • acq-pgsebruker — multi-shell sequence based on Bruker's DtiEpi protocol

MT

  • acq-6p5uT5p5kHz_MTw — the MTw
  • acq-refMT_T1w — the T1w associated with the MTw

MP2RAGE

  • inv-1_part-mag_MP2RAGE
  • inv-2_part-mag_MP2RAGE
  • rec-bruker_UNI — UNI reconstruction done by the Bruker console

anat HR

  • acq-flash_T1w
  • acq-rare_T2w

MEGRE (QSM) — all volumes have acq-qsm

  • part-mag_MEGRE
  • run-01_part-imag_MEGRE
  • run-01_part-real_MEGRE

heuristics/custom/exvivo_extras.py (new)

bvec/bval creation was already handled by bruker.py. I added a layer on top for the ex vivo protocol (heuristics/custom/exvivo_extras.py). Inside it I handle three things:

  1. .bmat creation — needed for quantification of the diffusion sequences (reading the private DICOM tag (0x0177, 0x1100)).
  2. MP2RAGE — splits the two inversions and automatically handles retrieval of the parameters to put them in the JSONs (reading the private DICOM tag (0x0177, 0x1100) -- needed for T1 quantification). I noticed that part of this step was also handled by scripts in workflow/lib/bids_fixes.py. I don't think we have access to the DICOM from the fix stage, which prevents retrieving the parameters for T1 quantification. What do you think? @akhanf
  3. MEGRE relabelling — the console pushes several series and dcm2niix splits the complex data, which heudiconv names poorly (MEGRE1/MEGRE2). Relabelled to part-real/part-imag by reading ImageType from the JSON, cross-checked against an empirically observed correspondence.

Small caveat: for the MEGRE imaginary and real part : if not several reconstructions are pushed to the cfmm server, then the single volume will have run-01 even though there is no run-02. It stays BIDS compliant but it's not pretty. I haven't managed to find a viable solution. Maybe via a rule in the fix stage to remove all occurrences of run-01 if there aren't multiple runs?

Open question

@akhanf @RicardoRios46 How should we format the necessary information to maintain an accurate record of the in-house sequence parameters? aDWI-BIDS chose to create _denc.json and _denc.tsv — but which information do we actually need to store in there so we can find our way around later? Maybe we could just create a .denc file to keep things consistent with the .bval and .bvec files? any better ideas ?

@RicardoRios46

RicardoRios46 commented Sep 25, 2026 •

Copy link
Copy Markdown

There is an offitial BIDs extension for the qMRI protocols part ot the qMRI standard.

They have specifications for

  • MT
  • MTsat
  • MP2RAGE
  • Multi-parametric Mapping
  • (Among others)

I suggest we use those standard for our protocols too.

@RicardoRios46

Copy link
Copy Markdown

For the dwi:

There is no standar for the muldidiemntional enccoding stuff right now. For a previous project I was using "acq-mde" to refer to a b-tesnor encoding acquisition. It may be usefull ot hve one for OGSE only aquisition like "acq-OGSE". If there is no "acq-mde" we can assume standard diffusion adn rely on bvec and bval alone. Open to duscission.

I like including the .bmat. We should be careful in how we write it. Some other mde toolbox use the voight notation for it (so the off diagonal elements need a scale factor). I think MatMRI reads from a vectorized version of it only. Lets further discuss this.

@RicardoRios46

Copy link
Copy Markdown

An example of a previous protocol that I have that is using qht qMRI standar for MTsat and my own mde labels:

├── anat
│   ├── sub-111001_run-01_FLAIR.json
│   ├── sub-111001_run-01_FLAIR.nii.gz
│   ├── sub-111001_run-01_T1w.json
│   ├── sub-111001_run-01_T1w.nii.gz
│   ├── sub-111001_run-01_T2w.json
│   ├── sub-111001_run-01_T2w.nii.gz
│   ├── sub-111001_run-01_flip-1_mt-off_MTS.json
│   ├── sub-111001_run-01_flip-1_mt-off_MTS.nii.gz
│   ├── sub-111001_run-01_flip-1_mt-on_MTS.json
│   ├── sub-111001_run-01_flip-1_mt-on_MTS.nii.gz
│   ├── sub-111001_run-01_flip-2_mt-off_MTS.json
│   └── sub-111001_run-01_flip-2_mt-off_MTS.nii.gz
├── dwi
│   ├── sub-111001_acq-mde_dir-AP_run-01_dwi.bval
│   ├── sub-111001_acq-mde_dir-AP_run-01_dwi.bvec
│   ├── sub-111001_acq-mde_dir-AP_run-01_dwi.json
│   └── sub-111001_acq-mde_dir-AP_run-01_dwi.nii.gz
├── fmap
│   ├── sub-111001_dir-PA_run-01_epi.json
│   └── sub-111001_dir-PA_run-01_epi.nii.gz
└── sub-111001_scans.tsv

@RicardoRios46

Copy link
Copy Markdown

Additional note for dwi:

Tje bvec and bvals that come from the DICOm param may be incorrect when using our custom sequence and need to be recalculates using the bmat. There is in the plans to update the sequence to properly handle this calculations, but this is a good chance to also correct them from here.

@Lucas-Arcamone

Lucas-Arcamone commented Sep 25, 2026 •

Copy link
Copy Markdown

@RicardoRios46

Summary of what you mention:

  1. MP2RAGE: include entity mt-on / mt-off, keep inv- because equivalent with the flip- (so no need to add flip-). Change the suffix for _MTS,
  2. Could you please verify that we have the correct .bmat file and, if not, correct it? Or could you please explain to me how to create the correct version?
  3. For now, we have acq-multidwienc for the custom sequence. This file includes both PGSE-, OGSE- and STE-encoded volumes. Should we split it up? Any ideas on what additional information to include for the OGSE/STE portion, in particular? (see the following note)

Note

Previous ideas/questions: How should we format the necessary information to maintain an accurate record of the in-house DWI sequence parameters (notally the OGSE part)? aDWI-BIDS chose to create _denc.json and _denc.tsv — but which information do we actually need to store in there so we can find our way around later? Maybe we could just create a .denc file to keep things consistent with the .bval and .bvec files? any better ideas ?

@RicardoRios46

Copy link
Copy Markdown

About the qMRI BIDS standard

The example I sent uses the qMRI BIDS standard for an MTsat acquisition (MTS suffix). However, there is also one that is only for MT (MT suffix), which we can use since this protocol is MT only. Check the template in qMRI BIDS.

I also see that the template has something for MP2RAGE (for both raw data and derivatives). Let's use that one too.

DWI acquisition

What if we start using acq-FreeShapes or acq-dtiEpiFreeShapes? That's the name of the new custom sequence (dtiEpiFreeShapes), so we'll know which scans were acquired with it. (By the way, do you have access to the sequence repo?)

I think we can make acq-pgsebruker entirely optional. If it's not there, we can assume the data comes from a pgsebruker sequence, which will be the case for most bruker acquired DWI protocols.

DWI metadata

The bmat in the DICOM header should be correct; the bvec and bval are the ones that may need correction. I can share the code we have to fix them (although we eventually want to correct this in the sequence itself).

I have an idea about which metadata files to create that could help us differentiate between all these advanced acquisitions. I'd like to hear Jake and Corey's thoughts on it, so I'm going to propose it in our lab Slack channel for discussion. Once we settle on something, we can continue here.

@Lucas-Arcamone

Copy link
Copy Markdown

Thanks @RicardoRios46 — and sorry, I mixed up MP2RAGE and MTsat details in my previous message. Here's a cleaner version (checked against BIDS 1.11.1):

MP2RAGE — inv-1_part-mag_MP2RAGE and inv-2_part-mag_MP2RAGE are already valid. The only fix is UNI → UNIT1, kept in anat/ (raw):

Therefore, the UNIT1 image provided by the scanner SHOULD be stored under anat in a raw BIDS dataset along with the MP2RAGE file collection and used as the primary input for quantifying a T1map.

MT — it's 3 volumes total. What I was calling MTw is actually 2 volumes, so I'll split it into flip-1_mt-off and flip-1_mt-on, and the volume I was calling T1w becomes flip-2_mt-off. Suffix MTS for all three ([MTS glossary](https://bids-specification.readthedocs.io/en/stable/glossary.html#objects.suffixes.MTS)). Note the flip indices are pointers (both MT volumes share flip-1 = 9°, T1w is flip-2 = 20°) — the actual values go in FlipAngle in the sidecars.

DWI — you're the boss here, so let's go with acq-FreeShapes or acq-dtiEpiFreeShapes, whichever you prefer. Agreed acq-pgsebruker can be dropped (optional). bval/bvec are already being corrected on our side, so that's covered.

I don't have access to the sequence repo — could you add me? And the metadata organization proposal sounds great.

@RicardoRios46

Copy link
Copy Markdown

Nice, that is perfect for the MP2RAGE and the MTsat protocol.

Lets do acq-dtiEpiFreeShapes. Longer name but it will match the name of the sequence exactly.

Ond yeah lets work in the metadata proposal.

@Lucas-Arcamone

Lucas-Arcamone commented Sep 28, 2026 •

Copy link
Copy Markdown

Progress update — naming fixed, a few open points

Fixed

  • MP2RAGE — UNI → UNIT1, kept in anat/ (raw).
  • MT — split into flip-1_mt-off / flip-1_mt-on, plus the reference T1w as flip-2_mt-off, all with the MTS suffix (MTsat collection).
  • DWI — acq-dtiEpiFreeShapes for the in-house sequence; no acq- entity for the standard Bruker one.

Naming is now correct for all sequences. Remaining issues below.

Open points @akhanf

1. DWI custom-encoding logic not yet handled.
I haven't touched the splitting/handling of the custom-encoded DWI volumes (PGSE / OGSE / STE within acq-dtiEpiFreeShapes). Tied to the metadata-format discussion — waiting on the Slack outcome.

2. MISSING_REQUIRED_ENTITY — MEGRE echo entity missing.
I'm intentionally keeping all echoes/channels stacked within each NIfTI: we export the complex data and reconstruct the phase downstream with a separate QSM script. With 9 echoes × 2 channels × (real + imag), splitting into one file per echo-/part-/channel would explode into an unmanageable number of files. So these won't satisfy the MEGRE collection (which requires echo-). Options: add them to .bidsignore, or move them under derivatives/sourcedata. What do you prefer here?

3. SIDECAR_KEY_REQUIRED. MTS sidecars for the reference T1w (flip-2_mt-off) .
I can't patch the sidecars of flip-2_mt-off_MTS and its rec-denoised counterpart from custom_callable: heudiconv writes these files read-only, so an in-place edit fails with PermissionError. Two ways forward: chmod +w before writing in the hook (works, but feels hacky — fighting heudiconv's finalized outputs), or do it in the fix stage via an update_json action (cleaner architecturally, since metadata fixes are what that stage is for). I lean toward the fix stage — agreed?

4. SCANS_FILENAME_NOT_MATCH_DATASET.
Expected: I rename files post-conversion (MP2RAGE/MT split, MEGRE relabel) but scans.tsv is generated by heudiconv beforehand and isn't updated. Needs a scans.tsv regeneration step after all renaming — logically also a fix-stage job.

5. EXTENSION_MISMATCH on .bmat.
.bmat isn't a BIDS extension, and any future custom extension we add (e.g. for the encoding metadata) won't be either. Simplest is to add *.bmat to .bidsignore — OK to go that route, or do we want a different convention for these?

@akhanf

akhanf commented Sep 28, 2026

Copy link
Copy Markdown
Member Author

Thanks both for making progress on this!

  1. Sounds good
  2. I think adding to the bidsignore is better, so the data remain in the folder along with the other raw data (they would be hidden away if in sourcedata or derivatives)..
  3. Yes bids-fix for this makes sense.
  4. You could just delete the scans.tsv as a bids-fix, or make a new one as a bids-fix. Since this isn't a required file (and not required for pybids parsing) could just drop it. The only useful thing I find it for is to keep track of the order of acquisition and the time of each acquisition.
  5. Yep bidsignore for this too!

@Lucas-Arcamone

Copy link
Copy Markdown

prepostshield — ex vivo 15T Bruker PV360: BIDS validation clean (warnings only)

bids-validator passes with 0 errors on 3 subjects.

config/trident/exvivo_15T.yml

  • Added bidsignore: — '*_MEGRE.*' (echo-stacked, non-BIDS) and '*.bmat' → clears MISSING_REQUIRED_ENTITY and EXTENSION_MISMATCH.
  • Added rename_json_keys then regenerate_scans_tsv (last).

workflow/lib/bids_fixes.py — two new fixes:

  • rename_json_keys — value-preserving JSON key rename (idempotent). RepetitionTime → RepetitionTimeExcitation on *flip-2_mt-off_MTS → clears SIDECAR_KEY_REQUIRED.
  • regenerate_scans_tsv — rebuilds *_scans.tsv after renames/splits so filename matches the dataset; re-attaches acq_time via each sidecar's AcquisitionTime, sorted by acquisition order → clears SCANS_FILENAME_NOT_MATCH_DATASET.

Summary naming choices — ex vivo 15T Bruker PV360 (as built)

global

  • sub-<label>_ses-exvivo15T

dwi

  • acq-dtiEpiFreeShapes_dwi — free-shape encoding (OGSE/uFA)
  • dwi — multi-shell (Bruker DtiEpi protocol), no acq- label

anat — MP2RAGE

  • inv-1_part-mag_MP2RAGE
  • inv-2_part-mag_MP2RAGE
  • rec-bruker_UNIT1 — UNI reconstruction done by the Bruker console

anat — MTsat (triplet; each also produced as rec-denoised)

  • flip-1_mt-off_MTS
  • flip-1_mt-on_MTS
  • flip-2_mt-off_MTS — T1w reference of the MT protocol

anat — high-res (each also produced as rec-denoised)

  • acq-flash_T1w
  • acq-rare_T2w

MEGRE (QSM) — all volumes carry acq-qsm (.bidsignored: echo-stacked, non-BIDS)

  • part-mag_MEGRE
  • run-01_part-real_MEGRE
  • run-01_part-imag_MEGRE

@akhanf @RicardoRios46 For the custom dwi metadata organisation, I think it can be done later in another PR. If no additions, then ready for review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants