Skip to content

Contextually type comprehension arguments during overload resolution - #4224

Open
markselby9 wants to merge 3 commits into
meta:mainfrom
markselby9:feat/4167-overload-comprehension
Open

markselby9 wants to merge 3 commits into
meta:mainfrom
markselby9:feat/4167-overload-comprehension

Conversation

@markselby9

@markselby9 markselby9 commented Jul 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes #4167.

When resolving an overloaded call, container literals (list/set/dict) are kept as expressions rather than eagerly inferred, so each candidate overload can contextually type them against its parameter (and narrow element literals). Comprehensions and generator expressions were not — they fell through to a context-free expr_infer in CallWithTypes::type_or_expr, so:

from collections.abc import Sequence
from typing import Literal, overload

Order = Literal["ascending", "descending"]

@overload
def sort_indices(sort_keys: Sequence[tuple[str, Order]]) -> str: ...
@overload
def sort_indices(sort_keys: str) -> int: ...
def sort_indices(sort_keys: Sequence[tuple[str, Order]] | str) -> str | int:
    return 0

names = ["a", "b"]
result = sort_indices([(name, "ascending") for name in names])

was inferred as list[tuple[str, str]] and reported no-matching-overload, even though mypy, pyright, and ty all accept it (reported against Narwhals).

The fix adds Expr::ListComp | Expr::SetComp | Expr::DictComp | Expr::Generator to the same un-flattened arm. These variants already support hint-directed inference via infer_with_decomposed_hint / decompose_list, so deferring their inference lets the existing per-overload contextual-typing path narrow their element/key/value types exactly as it does for container literals. When a hint doesn't decompose (e.g. a generator against a non-Sequence param), inference falls back to the previous context-free behaviour, so no path is worse than before.

Test Plan

Added test_overload_literal_narrowing_through_comprehension (the reported repro) and test_overload_literal_narrowing_through_set_dict_generator (set/dict comprehensions and generators) to pyrefly/lib/test/overload.rs. Both fail before the change (no-matching-overload, reveal_type = Unknown) and pass after (reveal_type = str).

cargo test -p pyrefly --lib passes; test.py formatting and linting are clean.

@github-actions

This comment has been minimized.

@markselby9
markselby9 force-pushed the feat/4167-overload-comprehension branch from 0a7c80a to 4973082 Compare July 24, 2026 03:12
@github-actions github-actions Bot added size/s and removed size/s labels Jul 24, 2026
@github-actions

This comment has been minimized.

@markselby9
markselby9 force-pushed the feat/4167-overload-comprehension branch from 4973082 to 2002483 Compare July 29, 2026 04:09
@github-actions github-actions Bot added size/s and removed size/s labels Jul 29, 2026
@meta-codesync

meta-codesync Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

This pull request has been imported. If you are a Meta employee, you can view this in D114002619. (Because this pull request was imported automatically, there will not be any future comments.)

@codspeed

codspeed Bot commented Jul 29, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 16 untouched benchmarks
⏩ 7 skipped benchmarks1


Comparing markselby9:feat/4167-overload-comprehension (b85bd25) with main (0ee48d3)

Open in CodSpeed

Footnotes

  1. 7 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@github-actions

This comment has been minimized.

@github-actions github-actions Bot added size/m and removed size/s labels Jul 29, 2026
@github-actions

This comment has been minimized.

@markselby9
markselby9 force-pushed the feat/4167-overload-comprehension branch from 3b9a01d to f0758f2 Compare July 29, 2026 08:59
@github-actions github-actions Bot added size/m and removed size/m labels Jul 29, 2026
@github-actions

This comment has been minimized.

@markselby9
markselby9 force-pushed the feat/4167-overload-comprehension branch from f0758f2 to 1102f6f Compare August 3, 2026 09:10
@github-actions github-actions Bot removed the size/m label Aug 3, 2026
@markselby9

Copy link
Copy Markdown
Contributor Author

Rebased onto current main (406082a6c); this is mergeable again.

callable.rs auto-merged cleanly across the call-boundary refactor (#8888e70c5 / #6ba516f98 / #e67bc8e7a) — the only conflict was an append collision in test/overload.rs, resolved keep-both. The refactor restructured the layer below argument checking, but neither seam this change uses moved: CallWithTypes::type_or_expr is byte-identical, and the post_check guard already routes through check_type_with_options(...).with_call_context(...) like its TypeForm/ShapedArray siblings, so it participates in the new boundary's residual capture the same way. Nothing here creates or consumes a CallBoundary.

I checked that the fix is still load-bearing rather than assuming it, by ablating only the type_or_expr comprehension arm on the rebased tree:

  • without it: reveal_type(sort_indices([(name, "ascending") for name in names])) → Unknown, plus no-matching-overload
  • with it: str

So the refactor fixed no part of #4167 on its own.

One thing worth reframing about the bug-marked test. While probing the guard's scope I found the widening it suppresses is not comprehension-specific — on current main, with a generic hint, list(filter(bool, {email.strip()})) and the list-literal equivalent both already give list[object], while the tuple form gives list[str]. So a provisional generic hint widening a deferred container's element type is pre-existing behaviour for list/set literals. Commit 1 makes comprehensions inherit it and commit 2 patches it back out for comprehensions, which means the guard restores comprehensions to their pre-PR result rather than introducing a new asymmetry. The bug marker is documenting a narrow symptom of that broader issue with the "don't flatten mutable builtin containers" approach.

I considered a tighter predicate — skip contextual typing only when the hint's element position is a bare unsolved quantified var, keeping narrowing for structural hints like Callable[[int], T], which would let the marker be dropped. I left it out: it makes this diff more divergent from upstream rather than less, and I can't verify it without a primer run. Happy to do it as a follow-up if you'd prefer that shape.

Also strengthened the tests: the previous generator assertion wasn't actually discriminating (a bare generator's element isn't literal-promoted, so it passed with or without the fix). Added test_overload_comprehension_hint_selects_narrower_overload, where every overload accepts the argument so it asserts which one wins; both assertions verified to fail under ablation.

Suite is 7467/0 on the new base, lint clean.

@github-actions github-actions Bot added the size/l label Aug 3, 2026
@markselby9
markselby9 force-pushed the feat/4167-overload-comprehension branch from 1102f6f to 3158a80 Compare August 11, 2026 03:38
@github-actions github-actions Bot added size/l and removed size/l labels Aug 11, 2026
@markselby9
markselby9 force-pushed the feat/4167-overload-comprehension branch from 3158a80 to db2fc5e Compare August 11, 2026 05:33
@github-actions github-actions Bot removed the size/l label Aug 11, 2026
@github-actions

This comment has been minimized.

@markselby9
markselby9 force-pushed the feat/4167-overload-comprehension branch from db2fc5e to fb0b587 Compare August 12, 2026 03:18
@github-actions github-actions Bot added size/xl and removed size/l labels Aug 12, 2026
@markselby9
markselby9 force-pushed the feat/4167-overload-comprehension branch from fb0b587 to 1613409 Compare August 12, 2026 03:31
@github-actions github-actions Bot added size/xl and removed size/xl labels Aug 12, 2026
@github-actions

This comment has been minimized.

When resolving an overloaded call, container literals (list/set/dict) are kept
as expressions rather than eagerly inferred, so each candidate overload can
contextually type them against its parameter and narrow element literals.
Comprehensions and generator expressions were not: they fell through to a
context-free `expr_infer`, so `[(name, "ascending") for name in names]` was
inferred as `list[tuple[str, str]]` and failed to match a
`Sequence[tuple[str, Literal["ascending", "descending"]]]` overload, even
though mypy, pyright, and ty all accept it.

Include comprehension and generator expressions in the un-flattened arm of
`CallWithTypes::type_or_expr`. They already support hint-directed inference via
`infer_with_decomposed_hint` / `decompose_list`, so deferring their inference
lets the existing per-overload contextual-typing path narrow their
element/key/value types exactly as it does for container literals.

Closes meta#4167
Contextually typing a comprehension against a not-yet-solved generic parameter
can widen its already-known element type (for example, `str` to `object` in
`filter`). Guard against that, but only for arguments `CallWithTypes`
deliberately deferred — the overload and union-callee resolution paths that
evaluate an argument more than once.

Recording the deferral on `TypeOrExpr` scopes the guard away from ordinary
generic calls (which must keep typing the comprehension contextually, exactly
like a list literal) and reaches keyword arguments, which are checked on a
separate path. The guard skips contextual typing only when context-free
inference is already fully resolved, so hint-dependent elements — a bare
lambda's parameter, or a variable an earlier argument already pinned — still
narrow. Generators in starred position are flattened rather than deferred,
since deferral there yields no narrowing and re-infers per candidate.
@markselby9
markselby9 force-pushed the feat/4167-overload-comprehension branch from 1613409 to 92a69eb Compare August 13, 2026 08:41
@github-actions github-actions Bot added size/xl and removed size/xl labels Aug 13, 2026
@github-actions

This comment has been minimized.

The widening guard inferred the comprehension into the shared `arg_errors`
only to check whether its context-free type was fully resolved. On the fallback
path the caller re-infers the same expression, so body diagnostics landed in
`arg_errors` twice and only the collector's per-range dedup hid the duplicates.
Probe into a throwaway collector, commit its diagnostics into `arg_errors` only
on the path that keeps the element type (which does not re-infer), and discard
them on the fallback. The success path still surfaces body errors, so
correctness no longer leans on dedup.
@github-actions github-actions Bot added size/xl and removed size/xl labels Aug 13, 2026
@github-actions

Copy link
Copy Markdown

Diff from mypy_primer, showing the effect of this PR on open source code:

============================================================
SUMMARY
============================================================
Total: +28 new errors, -33 fixed errors

Projects with changes (6):
  rotki: +0 -1
  jax: +0 -2
  scikit-learn: +24 -27
  core: +1 -0
  xarray: +2 -2
  optuna: +1 -1
============================================================

FULL DIFF DETAILS
------------------------------------------------------------

rotki (https://github.com/rotki/rotki)
- ERROR rotkehlchen/api/services/assets.py:554:32-557:15: No matching overload found for function `typing.MutableMapping.update` called with arguments: (dict[Asset, list[Price | int]]) [no-matching-overload]

jax (https://github.com/google/jax)
- ERROR jax/experimental/mosaic/gpu/utils.py:858:23-69: The type of `known_factor_prod` is unknown; it is inferred as an implicit `Any` [unknown-variable-type]
- ERROR jax/experimental/roofline/rooflines.py:455:15-63: The type of `axes_size` is unknown; it is inferred as an implicit `Any` [unknown-variable-type]

scikit-learn (https://github.com/scikit-learn/scikit-learn)
- ERROR sklearn/compose/tests/test_column_transformer.py:876:9-35: Result of call expression is of type `ndarray | ndarray[tuple[Any, ...], dtype[Unknown]]` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/compose/tests/test_column_transformer.py:876:9-35: Result of call expression is of type `ndarray[tuple[Any, ...], dtype[Any | Unknown]] | ndarray` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/compose/tests/test_column_transformer.py:883:9-35: Result of call expression is of type `ndarray | ndarray[tuple[Any, ...], dtype[Unknown]]` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/compose/tests/test_column_transformer.py:883:9-35: Result of call expression is of type `ndarray[tuple[Any, ...], dtype[Any | Unknown]] | ndarray` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/compose/tests/test_column_transformer.py:2219:9-35: Result of call expression is of type `ndarray | ndarray[tuple[Any, ...], dtype[Unknown]]` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/compose/tests/test_column_transformer.py:2219:9-35: Result of call expression is of type `ndarray[tuple[Any, ...], dtype[Any | Unknown]] | ndarray` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/compose/tests/test_column_transformer.py:2400:9-35: Result of call expression is of type `ndarray | ndarray[tuple[Any, ...], dtype[Unknown]]` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/compose/tests/test_column_transformer.py:2400:9-35: Result of call expression is of type `ndarray[tuple[Any, ...], dtype[Any | Unknown]] | ndarray` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/ensemble/_voting.py:430:38-434:14: No matching overload found for function `numpy.lib._shape_base_impl.apply_along_axis` called with arguments: ((x: Unknown) -> signedinteger[_NBitIntP], axis=Literal[1], arr=ndarray[tuple[Any, ...], dtype[Unknown]]) [no-matching-overload]
+ ERROR sklearn/ensemble/_voting.py:430:38-434:14: No matching overload found for function `numpy.lib._shape_base_impl.apply_along_axis` called with arguments: ((x: Unknown) -> signedinteger[_NBitIntP], axis=Literal[1], arr=ndarray[tuple[Any, ...], dtype[Any | Unknown]]) [no-matching-overload]
- ERROR sklearn/ensemble/_voting.py:498:29-37: No matching overload found for function `numpy._core.shape_base.hstack` called with arguments: (ndarray[tuple[Any, ...], dtype[Unknown]]) [no-matching-overload]
+ ERROR sklearn/ensemble/_voting.py:498:29-37: No matching overload found for function `numpy._core.shape_base.hstack` called with arguments: (ndarray[tuple[Any, ...], dtype[Any | Unknown]]) [no-matching-overload]
- ERROR sklearn/ensemble/tests/test_voting.py:110:9-26: Result of call expression is of type `ndarray[tuple[Any, ...], dtype[Unknown]] | Unknown` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/ensemble/tests/test_voting.py:110:9-26: Result of call expression is of type `ndarray[tuple[Any, ...], dtype[Any | Unknown]] | Unknown` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/ensemble/tests/test_voting.py:112:9-26: Result of call expression is of type `ndarray[tuple[Any, ...], dtype[Unknown]]` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/ensemble/tests/test_voting.py:112:9-26: Result of call expression is of type `ndarray[tuple[Any, ...], dtype[Any | Unknown]]` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/ensemble/tests/test_voting.py:114:9-28: Result of call expression is of type `ndarray[tuple[Any, ...], dtype[Unknown]]` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/ensemble/tests/test_voting.py:114:9-28: Result of call expression is of type `ndarray[tuple[Any, ...], dtype[Any | Unknown]]` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/inspection/_permutation_importance.py:72:24-42: Object of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Unknown]] | Unknown` has no attribute `columns` [missing-attribute]
+ ERROR sklearn/inspection/_permutation_importance.py:72:24-42: Object of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Any | Unknown]] | Unknown` has no attribute `columns` [missing-attribute]
- ERROR sklearn/metrics/tests/test_classification.py:1038:16-1044:6: The type of `cov_ypyp` is unknown; it is inferred as an implicit `Any` [unknown-variable-type]
- ERROR sklearn/metrics/tests/test_common.py:1638:13-1643:14: The type of this argument is unknown [unknown-argument-type]
- ERROR sklearn/model_selection/_validation.py:414:26-28: Cannot set item in `dict[str, list[Unknown] | ndarray[tuple[Any, ...], dtype[Unknown]]]` [unsupported-operation]
+ ERROR sklearn/model_selection/_validation.py:414:26-28: Cannot set item in `dict[str, list[Unknown] | ndarray[tuple[Any, ...], dtype[Any | Unknown]]]` [unsupported-operation]
- ERROR sklearn/tests/test_pipeline.py:1111:9-35: Result of call expression is of type `ndarray | ndarray[tuple[Any, ...], dtype[Unknown]]` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/tests/test_pipeline.py:1111:9-35: Result of call expression is of type `ndarray[tuple[Any, ...], dtype[Any | Unknown]] | ndarray` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/tests/test_pipeline.py:1369:9-38: Result of call expression is of type `ndarray | ndarray[tuple[Any, ...], dtype[Unknown]]` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/tests/test_pipeline.py:1369:9-38: Result of call expression is of type `ndarray[tuple[Any, ...], dtype[Any | Unknown]] | ndarray` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/tests/test_pipeline.py:1391:9-38: Result of call expression is of type `ndarray | ndarray[tuple[Any, ...], dtype[Unknown]]` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/tests/test_pipeline.py:1391:9-38: Result of call expression is of type `ndarray[tuple[Any, ...], dtype[Any | Unknown]] | ndarray` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/utils/_param_validation.py:883:47-51: Type of lambda parameter `self` is unknown [implicit-any-lambda]
- ERROR sklearn/utils/tests/test_indexing.py:227:13-61: Result of call expression is of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Unknown]] | Unknown` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/utils/tests/test_indexing.py:227:13-61: Result of call expression is of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Any | Unknown]] | Unknown` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/utils/tests/test_indexing.py:344:13-51: Result of call expression is of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Unknown]] | Unknown` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/utils/tests/test_indexing.py:344:13-51: Result of call expression is of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Any | Unknown]] | Unknown` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/utils/tests/test_indexing.py:367:9-41: Result of call expression is of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Unknown]] | Unknown` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/utils/tests/test_indexing.py:367:9-41: Result of call expression is of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Any | Unknown]] | Unknown` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/utils/tests/test_indexing.py:373:9-49: Result of call expression is of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Unknown]] | Unknown` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/utils/tests/test_indexing.py:373:9-49: Result of call expression is of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Any | Unknown]] | Unknown` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/utils/tests/test_indexing.py:397:9-54: Result of call expression is of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Unknown]] | Unknown` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/utils/tests/test_indexing.py:397:9-54: Result of call expression is of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Any | Unknown]] | Unknown` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/utils/tests/test_indexing.py:405:9-47: Result of call expression is of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Unknown]] | Unknown` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/utils/tests/test_indexing.py:405:9-47: Result of call expression is of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Any | Unknown]] | Unknown` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/utils/tests/test_indexing.py:428:9-20: Object of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Unknown]] | Unknown` has no attribute `iloc` [missing-attribute]
+ ERROR sklearn/utils/tests/test_indexing.py:428:9-20: Object of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Any | Unknown]] | Unknown` has no attribute `iloc` [missing-attribute]
- ERROR sklearn/utils/tests/test_indexing.py:439:9-43: Result of call expression is of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Unknown]] | Unknown` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/utils/tests/test_indexing.py:439:9-43: Result of call expression is of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Any | Unknown]] | Unknown` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/utils/validation.py:2580:19-41: No matching overload found for function `numpy._core.fromnumeric.amax` called with arguments: (ndarray[tuple[int], dtype[numpy.bool]] | ndarray[Unknown, Unknown] | None) [no-matching-overload]
+ ERROR sklearn/utils/validation.py:2580:19-41: No matching overload found for function `numpy._core.fromnumeric.amax` called with arguments: (ndarray[tuple[int], dtype[float64]] | ndarray[Unknown, Unknown] | None) [no-matching-overload]
- ERROR sklearn/utils/validation.py:2581:22-44: No matching overload found for function `numpy._core.fromnumeric.amin` called with arguments: (ndarray[tuple[int], dtype[numpy.bool]] | ndarray[Unknown, Unknown] | None) [no-matching-overload]
+ ERROR sklearn/utils/validation.py:2581:22-44: No matching overload found for function `numpy._core.fromnumeric.amin` called with arguments: (ndarray[tuple[int], dtype[float64]] | ndarray[Unknown, Unknown] | None) [no-matching-overload]

core (https://github.com/home-assistant/core)
+ ERROR homeassistant/components/soundtouch/media_player.py:280:53-64: The type of this argument is unknown [unknown-argument-type]

xarray (https://github.com/pydata/xarray)
- ERROR xarray/core/dataset.py:2719:20-26: Object of type `_arrayapi[Any, Any] | _arrayfunction[Any, Any] | ndarray | ndarray[Unknown, dtype[Unknown]] | Unknown` has no attribute `ndim` [missing-attribute]
+ ERROR xarray/core/dataset.py:2719:20-26: Object of type `_arrayapi[Any, Any] | _arrayfunction[Any, Any] | ndarray | ndarray[Unknown, dtype[Any | Unknown]] | ndarray[Unknown, dtype[Unknown]] | Unknown` has no attribute `ndim` [missing-attribute]
- ERROR xarray/core/dataset.py:2724:23-27: Yielded type `tuple[Hashable, _arrayapi[Any, Any] | _arrayfunction[Any, Any] | ndarray | ndarray[Unknown, dtype[Unknown]] | Unknown]` is not assignable to declared yield type `tuple[Hashable, Variable | int | ndarray | slice[Any, Any, Any]]` [invalid-yield]
+ ERROR xarray/core/dataset.py:2724:23-27: Yielded type `tuple[Hashable, _arrayapi[Any, Any] | _arrayfunction[Any, Any] | ndarray | ndarray[Unknown, dtype[Any | Unknown]] | ndarray[Unknown, dtype[Unknown]] | Unknown]` is not assignable to declared yield type `tuple[Hashable, Variable | int | ndarray | slice[Any, Any, Any]]` [invalid-yield]

optuna (https://github.com/optuna/optuna)
- ERROR optuna/visualization/_hypervolume_history.py:114:30-50: `*` is not supported between `ndarray[tuple[int], dtype[signedinteger[_NBitIntP]]]` and `None` [unsupported-operation]
+ ERROR optuna/visualization/_hypervolume_history.py:114:30-50: `*` is not supported between `ndarray[tuple[int], dtype[float64]]` and `None` [unsupported-operation]

@stroxler stroxler left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

I think there are a set of issues remaining to work through (besides needing a rebase) before we can merge this:

  1. Nested comprehensions bypass the exponential-work guard ([location](

    | Expr::ListComp(_)
    | Expr::SetComp(_)
    | Expr::DictComp(_)
    | Expr::Generator(_)),
    _,
    ) if !nests_calls_to_depth(e, MIN_FLATTEN_CALL_DEPTH) => {
    // Hack: keep mutable builtin containers as expressions, since they often need to be
    // contextually typed against the function's parameter types, unless nesting depth
    // reaches or exceeds `MIN_FLATTEN_CALL_DEPTH` to avoid exponential blowup.
    // Comprehensions and generators are included for the same reason: their
    // element/key/value types must be inferred against the parameter hint (e.g. to
    // narrow a literal element) rather than eagerly here.
    //
    // Mark the expression as deliberately deferred: this call is evaluated more than
    // once (overload / union-callee resolution), so argument checking must apply the
    // guard that keeps a provisional generic hint from widening it.
    TypeOrExpr::Expr(e, true)
    )). The depth detector still counts only container literals. At 11 nested overload calls, the base finishes in 0.082 s; the PR exceeds 15 s. Constructor nesting also regresses.

  2. Nested Any loses known element structure ([location](

    let probe = self.error_collector();
    let ty = self.expr_infer(x, &probe);
    if ty.any(|t| t.is_any()) || !ty.collect_maybe_placeholder_vars().is_empty() {
    return None;
    )). Filtering a comprehension over list[list[Any]] yields list[object] on the PR, while the base preserves list[list[Any]].

  3. Failed overload constraints leak between candidates ([location](

    | Expr::ListComp(_)
    | Expr::SetComp(_)
    | Expr::DictComp(_)
    | Expr::Generator(_)),
    _,
    ) if !nests_calls_to_depth(e, MIN_FLATTEN_CALL_DEPTH) => {
    // Hack: keep mutable builtin containers as expressions, since they often need to be
    // contextually typed against the function's parameter types, unless nesting depth
    // reaches or exceeds `MIN_FLATTEN_CALL_DEPTH` to avoid exponential blowup.
    // Comprehensions and generators are included for the same reason: their
    // element/key/value types must be inferred against the parameter hint (e.g. to
    // narrow a literal element) rather than eagerly here.
    //
    // Mark the expression as deliberately deferred: this call is evaluated more than
    // once (overload / union-callee resolution), so argument checking must apply the
    // guard that keeps a provisional generic hint from widening it.
    TypeOrExpr::Expr(e, true)
    )). A referenced empty list hidden inside a deferred comprehension is pinned to int by a rejected candidate, causing the valid str overload and subsequent append to fail.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Overload resolution doesn't do "expected type" literal narrowing through a comprehension

4 participants