Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
a7a55e1
Initial score-based diffusion uncertainity model, based on the pop-co…
tHarvey303 May 1, 2026
9f38413
Improved noise model architecture.
tHarvey303 May 1, 2026
0c466ff
balance validation loss score
tHarvey303 May 2, 2026
fc63a26
Improving noise models
tHarvey303 May 5, 2026
d7a1e43
Performance improvements to diffusion noise model
tHarvey303 May 5, 2026
7cdc9de
More reliable training behaviour.
tHarvey303 May 6, 2026
aac1aae
Fix ValueError in _scatter when using noise models
Copilot May 6, 2026
f6e0bf6
Use direct [0] indexing for scalar extraction from apply_noise result
Copilot May 6, 2026
45702a9
Wire new noise model into feature array creation, add tests.
tHarvey303 May 6, 2026
6fab0d9
Fix more unit attributes
tHarvey303 May 6, 2026
3316f1b
Fix tests
tHarvey303 May 6, 2026
365c5dd
Merge branch 'diffusion_noise_model' of https://github.com/synthesize…
tHarvey303 May 6, 2026
4b61542
Make mock noise model for testing more stable
tHarvey303 May 7, 2026
05c0bb2
Inital docs (not finished) for diffusion noise model.
tHarvey303 May 7, 2026
56bd6ea
Add min/max magnitude clipping to diffusion noise model (in future as…
tHarvey303 May 7, 2026
d189497
Added support for asinh magnitude scaling in the diffusion noise model
tHarvey303 May 8, 2026
c8e5e7d
minor changes
tHarvey303 Jun 15, 2026
527da1a
Changes to make diffusion models work properly.
tHarvey303 Jun 15, 2026
80e501f
Merge branch 'diffusion_noise_model' of https://www.github.com/synthe…
tHarvey303 Jun 15, 2026
fa791dd
Test fixes for new noise model behaviour.
tHarvey303 Jun 15, 2026
4419b63
Add more support for ScoreNoise model in SBI_Fitter methods
tHarvey303 Jun 19, 2026
ea96db9
Merge branch 'main' into diffusion_noise_model
tHarvey303 Jun 19, 2026
a2e3dd9
Remove debug prints.
tHarvey303 Jun 23, 2026
eb72f6f
Update branch to use Syntillate noise model as well as cached SVO fil…
tHarvey303 Jul 25, 2026
61e0564
add to dependencies
tHarvey303 Jul 25, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions .github/workflows/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,17 +43,25 @@ jobs:
pip install wheel
sudo apt install pandoc
pip install "cosmos-synthesizer@git+https://github.com/synthesizer-project/synthesizer.git"
# syntillate (score-based diffusion noise model) is a hard dependency but
# is not yet on PyPI, so install it before resolving synference itself.
pip install "syntillate@git+https://github.com/synthesizer-project/syntillate.git"
pip install .[test,docs]
pip install "dense_basis@git+https://github.com/kartheikiyer/dense_basis.git"
pip install "ltu_ili@git+https://github.com/maho3/ltu-ili.git"
pip install "cosmos-synthesizer@git+https://github.com/synthesizer-project/synthesizer.git#egg=synthesizer"
pip install pytest-xdist # enable parallel pytest execution
- name: Download test data
run: |
# Download test grid data
# Download test grid data
mkdir -p data/libraries/
synthesizer-download --test-grids --dust-grid
synthesizer-download --test-grids --dust-grid --instruments JWSTNIRCamWide
python -c "from synference.utils import download_test_data; download_test_data()"
- name: Download SVO filter cache
run: |
# Pre-fetch cached SVO filter responses so the notebooks never hit the
# live SVO service, which is intermittently unavailable.
synthesizer-download --svo-filter-cache
- name: Sphinx Build
run: |
# Test sphinx build (runs all notebooks)
Expand Down
15 changes: 12 additions & 3 deletions .github/workflows/python-app.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,16 +31,25 @@ jobs:
- name: Install dependencies
run: |
python -m pip install --upgrade pip
# syntillate (score-based diffusion noise model) is a hard dependency but
# is not yet on PyPI, so install it before resolving synference itself.
pip install "syntillate@git+https://github.com/synthesizer-project/syntillate.git"
pip install .
pip install ruff pytest
pip install "ltu_ili@git+https://github.com/maho3/ltu-ili.git"
pip install "cosmos-synthesizer@git+https://github.com/synthesizer-project/synthesizer.git#egg=synthesizer"
python -c "from synference.utils import download_test_data; download_test_data()"


- name: Download SVO filter cache
run: |
# Pre-fetch cached SVO filter responses so the tests never hit the live
# SVO service, which is intermittently unavailable.
synthesizer-download --svo-filter-cache

- name: Test import
run: |
python -c "import synference"
python -c "import synference"

- name: Test with pytest
run: |
pytest
Expand Down
6 changes: 5 additions & 1 deletion docs/source/library_gen/synthesizer_crash_course.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -512,7 +512,11 @@
]
}
],
"metadata": {},
"metadata": {
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
1 change: 1 addition & 0 deletions docs/source/noise_modelling/creating_noise_model.rst
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,5 @@ Here we detail the various ways to create noise models in Synference. Noise mode
:maxdepth: 1

noise_models
diffusion_noise_models

166 changes: 166 additions & 0 deletions docs/source/noise_modelling/diffusion_noise_models.ipynb
Original file line number Diff line number Diff line change
@@ -0,0 +1,166 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "5413f0b4",
"metadata": {},
"source": [
"# Diffusion Noise Models\n",
"\n",
"This page describes the diffusion noise model available in `synference`. In contrast to the empirical noise models discussed in the previous sections, which simply model per-band uncertainties as a function of magnitude using binned interpolation, the diffusion noise model is a generative model that learns to the full latent distribution of noise in the data. This allows it to capture more complex noise properties, such as correlations between bands, and to generate more realistic noise realizations.\n",
"\n",
"This noise model is based on the diffusion probabilistic model framework, which has been successfully applied to a wide range of generative modeling tasks in machine learning. The basic idea is to model the noise as a stochastic process that gradually transforms a simple initial distribution (e.g., Gaussian noise) into the complex noise distribution observed in the data. By learning this transformation, the model can generate realistic noise samples that can be added to synthetic photometry to better match the properties of real observations.\n",
"\n",
"This is implemented as the `ScoreBasedUncertaintyModel`, which learns the conditional uncertainty distribution p(\u03c3|m), where \u03c3 is the uncertainty and m is the magnitude. It lives in the standalone [syntillate](https://github.com/synthesizer-project/syntillate) package, which Synference installs as a dependency and re-exports from `synference.noise_models`, so it can be used and serialized alongside Synference's other uncertainty models. The model is trained on a dataset of real photometric measurements, where the input is the magnitude and the target is the corresponding (log-scaled) uncertainty. Once trained, the model can be used to generate noise samples for synthetic photometry by sampling from the learned distribution.\n",
"\n",
"We will demonstate this on the standard COSMOSO2020 'Farmer' catalog (Weaver et al. 2020). If you want to train the noise model yourself, you will have to download the full catalog (~3 GB) from [here](https://cosmos2020.calet.org/).\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3fc591b0",
"metadata": {},
"outputs": [],
"source": [
"from syntillate import ScoreBasedUncertaintyModel\n",
"\n",
"?ScoreBasedUncertaintyModel"
]
},
{
"cell_type": "markdown",
"id": "ced7ee22",
"metadata": {},
"source": [
"### Training the Noise Model\n",
"\n",
"The first part is just loading and cleaning the COSMOS2020 catalog, which is fairly standard. We filter low signal to noise sources which are faint in IRAC Ch. 1. As we are interested in learning the full conditional distribution of noise across all the filters, we also remove sources with missing photometry in any of the filters."
]
},
{
"cell_type": "markdown",
"id": "cfad048c",
"metadata": {},
"source": [
"```python\n",
"\n",
"from astropy.table import Table\n",
"import numpy as np\n",
"\n",
"table = Table.read('COSMOS2020_FARMER_R1_v2.2_p3.fits', memmap=True)\n",
"\n",
"bands = ['CFHT_u', 'HSC_g', 'HSC_r', 'HSC_i', 'HSC_z',\n",
" 'HSC_y', 'UVISTA_Y', 'UVISTA_J', 'UVISTA_H', \n",
" 'UVISTA_Ks', 'IRAC_CH1', 'IRAC_CH2']\n",
"\n",
"snr_filter = table['IRAC_CH1_MAG'] < 26\n",
"table = table[snr_filter]\n",
"\n",
"for band in bands:\n",
" table = table[np.isfinite(table[band + '_MAG'])]\n"
]
},
{
"cell_type": "markdown",
"id": "f9388d26",
"metadata": {},
"source": [
"We then construct our training dataset, which consists of the magnitudes and uncertainties for each band. \n",
"\n",
"```python\n",
"\n",
"mag_array = np.array([table[band + '_MAG'] for band in bands]).T\n",
"flux_error_array = np.array([table[band + '_FLUXERR'] for band in bands]).T\n",
"\n",
"```\n",
"\n",
"Next, we can train the `ScoreBasedUncertaintyModel` on this dataset. This will learn the conditional distribution of uncertainties given magnitudes for each band. We will leave the model architecture as it's default, which is a simple MLP with 5 layers and a hidden dimension of 256. The training process can take some time, especially if you are using a CPU, so be patient!\n",
"\n",
"You can adjust the number of epochs and batch size as needed --> larger batch sizes will speed up training but require more memory, while more epochs will allow the model to learn better but will take longer.\n",
"\n",
"```python\n",
"\n",
"from syntillate import ScoreBasedUncertaintyModel\n",
"noise_model = ScoreBasedUncertaintyModel(filter_names=bands)\n",
"\n",
"# COSMOS2020 FLUXERR columns are in uJy; the model reports sigma back in\n",
"# the same units (see noise_model.sigma_units).\n",
"noise_model.fit(\n",
" mag_array,\n",
" flux_error_array,\n",
" flux_uncertainty_units='uJy',\n",
" n_epochs=200,\n",
" batch_size=1024,\n",
")\n",
"\n",
"```\n",
"\n",
"### Saving the Model\n",
"\n",
"We can save the trained model to disk for later use - the model can be serialized directly into a custom HDF5 format, and re-loaded later without needing to re-train, or rely on unstable dependencies like `pickle` or `torch.save`.\n",
"\n",
"```python\n",
"\n",
"import h5py\n",
"with h5py.File('COSMOS_noise_model.h5', 'w') as f:\n",
" noise_model.serialize_to_hdf5(f.create_group('noise_model'))\n",
"\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "46d187b0",
"metadata": {},
"source": [
"### Loading the Model\n",
"\n",
"We can load in the saved model from disk, just to prove it works! Here we will switch to our pre-trained model which you can get by running `synference-download`, but feel free to load in your own trained model if you have one.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "833cc1b5",
"metadata": {},
"outputs": [],
"source": [
"import h5py\n",
"\n",
"with h5py.File(\"COSMOS_noise_model.h5\", \"r\") as f:\n",
" noise_model = ScoreBasedUncertaintyModel._from_hdf5_group(f[\"noise_model\"])"
]
},
{
"cell_type": "markdown",
"id": "d2e11370",
"metadata": {},
"source": [
"### Validating the Noise Model"
]
},
{
"cell_type": "markdown",
"id": "827471fa",
"metadata": {},
"source": [
"### Using the Noise Model in Synference"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "synference",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.12.12"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
63 changes: 49 additions & 14 deletions examples/sbi/scripts/train_model.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
from synference import (
AsinhEmpiricalUncertaintyModel,
SBI_Fitter,
ScoreBasedUncertaintyModel,
Simformer_Fitter,
create_uncertainty_models_from_EPOCHS_cat,
)
Expand Down Expand Up @@ -74,7 +75,7 @@ class Args:
additional_model_args: tuple = ()
parameters_to_add: tuple = ()
parameters_to_remove: tuple = ()
data_err_file: str = """/home/tharvey/Downloads/JADES-Deep-GS_MASTER_Sel-f277W+f356W+f444W_v9_loc_depth_masked_10pc_EAZY_matched_selection_ext_src_UV.fits""" # noqa
data_err_file: str = None # Path to FITS file with photometric errors for empirical noise model
data_err_hdu: str = "OBJECTS" # The HDU name in the FITS file
background: bool = False
model_features: tuple = ()
Expand All @@ -87,6 +88,7 @@ class Args:
noise_model_class: str = "general" # Type of noise model to use - general, depth or asinh.
custom_config_yaml: str = None
sql_db_path: str = None # Path to SQLite database for storing results
precomputed_noise_model_path: str = None # Path saved HDF5 file with noise


parser.add_arguments(Args, dest="args")
Expand Down Expand Up @@ -122,6 +124,15 @@ def main_task(args: Args) -> None:
hst_bands = ["F435W", "F606W", "F775W", "F814W", "F850LP"]

if args.scatter_fluxes > 0:
if (args.data_err_file is None and args.precomputed_noise_model_path is None) or (
args.data_err_file is not None and args.precomputed_noise_model_path is not None
):
raise ValueError(
"When scatter_fluxes > 0, exactly one of data_err_file or "
"precomputed_noise_model_path must be provided."
)

if args.scatter_fluxes > 0 and args.data_err_file:
table = Table.read(args.data_err_file, format="fits", hdu=args.data_err_hdu)
bands = [i.split("_")[-1] for i in table.colnames if i.startswith("loc_depth")]
if len(phot_to_remove) > 0:
Expand All @@ -148,6 +159,12 @@ def main_task(args: Args) -> None:
min_flux_error=args.min_flux_error,
model_class=args.noise_model_class,
)
elif args.scatter_fluxes > 0 and args.precomputed_noise_model_path:
from synference.noise_models import load_unc_model_from_hdf5

empirical_noise_models = load_unc_model_from_hdf5(args.precomputed_noise_model_path)[
"noise_model"
]

else:
empirical_noise_models = {}
Expand Down Expand Up @@ -278,29 +295,43 @@ def log10_floor(x, floor=-6):
unused_filters.append(band_name)

print("Photometry in grid:", empirical_model_fitter.raw_observation_names, file=sys.stdout)
use_noise_models = args.scatter_fluxes > 0 and args.include_errors_in_feature_array
is_score_model = isinstance(empirical_noise_models, ScoreBasedUncertaintyModel)

for filt in empirical_model_fitter.raw_observation_names:
if (
args.scatter_fluxes > 0
and args.include_errors_in_feature_array
and filt not in empirical_noise_models
):
if not use_noise_models:
continue
if is_score_model:
# The score model conditions on all of its bands at once, so any
# filter it was not trained on cannot be used.
check_names = [i.lower() for i in empirical_noise_models.filter_names]
if filt.lower() not in check_names:
print(
f"Filter {filt} not in the score-based noise model. Adding to unused filters.",
file=sys.stdout,
)
unused_filters.append(filt)
elif filt not in empirical_noise_models:
unused_filters.append(filt)

print(f"Unused filters: {unused_filters}", file=sys.stdout)

# flux_units is "AB" unless scatter_fluxes and include_errors_in_feature_array
# and all noise models are AsinhUncertaintyModels
# and the noise model(s) work in asinh magnitudes
flux_units = "AB"
if args.scatter_fluxes > 0 and args.include_errors_in_feature_array:
if all(
isinstance(model, AsinhEmpiricalUncertaintyModel)
for model in empirical_noise_models.values()
):
if use_noise_models:
if is_score_model:
asinh = empirical_noise_models.is_asinh_scaled
else:
asinh = all(
isinstance(model, AsinhEmpiricalUncertaintyModel)
for model in empirical_noise_models.values()
)
if asinh:
flux_units = "asinh"

print(empirical_noise_models)
empirical_model_fitter.create_feature_array_from_raw_photometry(
extra_features=list(args.model_features),
extra_features=list(args.model_features[0].split(",")) if args.model_features else [],
normalize_method=args.norm_method,
include_errors_in_feature_array=args.include_errors_in_feature_array,
scatter_fluxes=args.scatter_fluxes,
Expand Down Expand Up @@ -441,3 +472,7 @@ def log10_floor(x, floor=-6):
else:
# Run in the foreground as normal
main_task(args)


# Example invocation with a pre-trained score-based noise model:
# python synference/examples/sbi/scripts/train_model.py --scatter_fluxes=1 --include_errors_in_feature_array=True --photometry_to_remove=HST/ACS_WFC.F606W,HST/ACS_WFC.F775W,HST/ACS_WFC.F814W,HST/ACS_WFC.F850LP,JWST/NIRCam.F070,HST/WFC3_IR.F105W,HST/WFC3_IR.F110W,HST/WFC3_IR.F110W,HST/WFC3_IR.F125W,HST/WFC3_IR.F140W,HST/WFC3_IR.F160W,JWST/NIRCam.F070W,JWST/NIRCam.F090W,JWST/NIRCam.F115W,JWST/NIRCam.F140M,JWST/NIRCam.F150W,JWST/NIRCam.F162M,JWST/NIRCam.F182M,JWST/NIRCam.F200W,JWST/NIRCam.F210M,JWST/NIRCam.F250M,JWST/NIRCam.F277W,JWST/NIRCam.F300M,JWST/NIRCam.F335M,JWST/NIRCam.F356W,JWST/NIRCam.F360M,JWST/NIRCam.F410M,JWST/NIRCam.F430M,JWST/NIRCam.F444W,JWST/NIRCam.F460M,JWST/NIRCam.F480M,HST/ACS_WFC/F475W,JWST/MIRI.F560W,JWST/MIRI.F770W,JWST/MIRI.F1000W,JWST/MIRI.F1130W,JWST/MIRI.F1280W,JWST/MIRI.F1500W,JWST/MIRI.F1800W,JWST/MIRI.F2100W,JWST/MIRI.F2550W,Spitzer/IRAC.I3,Spitzer/IRAC.I4,CTIO/DECam.u,CTIO/DECam.g,CTIO/DECam.r,CTIO/DECam.i,CTIO/DECam.z,CTIO/DECam.Y,LSST/LSST.u,LSST/LSST.g,LSST/LSST.r,LSST/LSST.i,LSST/LSST.z,LSST/LSST.Y,PAN-STARRS/PS1.g,PAN-STARRS/PS1.r,PAN-STARRS/PS1.i,PAN-STARRS/PS1.w,PAN-STARRS/PS1.z,PAN-STARRS/PS1.y,Paranal/VISTA.Z,CFHT/MegaCam.g,CFHT/MegaCam.r,CFHT/MegaCam.i,CFHT/MegaCam.z,HST/ACS_WFC.F435W,HST/ACS_WFC.F475W,Euclid/NISP.Y,Euclid/NISP.J/Euclid/NISP.H --grid_path=/cosma/apps/dp276/dc-harv3/synference/libraries/grid_BPASS_Chab_DenseBasis_SFH_0.01_z_14_logN_5.0_Calzetti_v4_multinode.hdf5 --model_name=BPASS_DenseBasis_COSMOS2020 --name_append=nsf --train_test_fraction=0.90 --parameter_transformations="Av=log10,sfr_10=log10_floor,mass_weighted_age=log10" --parameters_to_add="mass_weighted_age,sfr_10,log_surviving_mass,beta" --model_types=nsf --custom_config_yaml=/cosma/apps/dp276/dc-harv3/synference/examples/sbi/configs/best_params_2101.yaml --precomputed_noise_model_path=/cosma/apps/dp276/dc-harv3/synference/priv/COSMOS2020_noise_model_asinh.h5 # noqa: E501
2 changes: 2 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,8 @@ dependencies = [
"plotext",
"matplotlib",
"numba",
# Score-based diffusion noise model (ScoreBasedUncertaintyModel).
"syntillate",
]

[project.optional-dependencies]
Expand Down
4 changes: 3 additions & 1 deletion src/synference/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,8 @@
from .noise_models import (
EmpiricalUncertaintyModel, create_uncertainty_models_from_EPOCHS_cat,
DepthUncertaintyModel, UncertaintyModel, AsinhEmpiricalUncertaintyModel,
GeneralEmpiricalUncertaintyModel, save_unc_model_to_hdf5, load_unc_model_from_hdf5
GeneralEmpiricalUncertaintyModel, save_unc_model_to_hdf5, load_unc_model_from_hdf5,
ScoreBasedUncertaintyModel
)

from .custom_runner import SBICustomRunner
Expand Down Expand Up @@ -103,6 +104,7 @@
"DepthUncertaintyModel",
"UncertaintyModel",
"GeneralEmpiricalUncertaintyModel",
"ScoreBasedUncertaintyModel",
"save_unc_model_to_hdf5",
"load_unc_model_from_hdf5",
"optimize_sfh_xlimit",
Expand Down
Loading