Repository navigation
Contextually type comprehension arguments during overload resolution - #4224
markselby9 wants to merge 3 commits into
Conversation
This comment has been minimized.
This comment has been minimized.
0a7c80a to
4973082
Compare
This comment has been minimized.
This comment has been minimized.
4973082 to
2002483
Compare
|
This pull request has been imported. If you are a Meta employee, you can view this in D114002619. (Because this pull request was imported automatically, there will not be any future comments.) |
Merging this PR will not alter performance
Comparing Footnotes
|
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
3b9a01d to
f0758f2
Compare
This comment has been minimized.
This comment has been minimized.
f0758f2 to
1102f6f
Compare
|
Rebased onto current
I checked that the fix is still load-bearing rather than assuming it, by ablating only the
So the refactor fixed no part of #4167 on its own. One thing worth reframing about the I considered a tighter predicate — skip contextual typing only when the hint's element position is a bare unsolved quantified var, keeping narrowing for structural hints like Also strengthened the tests: the previous generator assertion wasn't actually discriminating (a bare generator's element isn't literal-promoted, so it passed with or without the fix). Added Suite is 7467/0 on the new base, lint clean. |
1102f6f to
3158a80
Compare
3158a80 to
db2fc5e
Compare
This comment has been minimized.
This comment has been minimized.
db2fc5e to
fb0b587
Compare
fb0b587 to
1613409
Compare
This comment has been minimized.
This comment has been minimized.
When resolving an overloaded call, container literals (list/set/dict) are kept as expressions rather than eagerly inferred, so each candidate overload can contextually type them against its parameter and narrow element literals. Comprehensions and generator expressions were not: they fell through to a context-free `expr_infer`, so `[(name, "ascending") for name in names]` was inferred as `list[tuple[str, str]]` and failed to match a `Sequence[tuple[str, Literal["ascending", "descending"]]]` overload, even though mypy, pyright, and ty all accept it. Include comprehension and generator expressions in the un-flattened arm of `CallWithTypes::type_or_expr`. They already support hint-directed inference via `infer_with_decomposed_hint` / `decompose_list`, so deferring their inference lets the existing per-overload contextual-typing path narrow their element/key/value types exactly as it does for container literals. Closes meta#4167
Contextually typing a comprehension against a not-yet-solved generic parameter can widen its already-known element type (for example, `str` to `object` in `filter`). Guard against that, but only for arguments `CallWithTypes` deliberately deferred — the overload and union-callee resolution paths that evaluate an argument more than once. Recording the deferral on `TypeOrExpr` scopes the guard away from ordinary generic calls (which must keep typing the comprehension contextually, exactly like a list literal) and reaches keyword arguments, which are checked on a separate path. The guard skips contextual typing only when context-free inference is already fully resolved, so hint-dependent elements — a bare lambda's parameter, or a variable an earlier argument already pinned — still narrow. Generators in starred position are flattened rather than deferred, since deferral there yields no narrowing and re-infers per candidate.
1613409 to
92a69eb
Compare
This comment has been minimized.
This comment has been minimized.
The widening guard inferred the comprehension into the shared `arg_errors` only to check whether its context-free type was fully resolved. On the fallback path the caller re-infers the same expression, so body diagnostics landed in `arg_errors` twice and only the collector's per-range dedup hid the duplicates. Probe into a throwaway collector, commit its diagnostics into `arg_errors` only on the path that keeps the element type (which does not re-infer), and discard them on the fallback. The success path still surfaces body errors, so correctness no longer leans on dedup.
|
Diff from mypy_primer, showing the effect of this PR on open source code: ============================================================
SUMMARY
============================================================
Total: +28 new errors, -33 fixed errors
Projects with changes (6):
rotki: +0 -1
jax: +0 -2
scikit-learn: +24 -27
core: +1 -0
xarray: +2 -2
optuna: +1 -1
============================================================
FULL DIFF DETAILS
------------------------------------------------------------
rotki (https://github.com/rotki/rotki)
- ERROR rotkehlchen/api/services/assets.py:554:32-557:15: No matching overload found for function `typing.MutableMapping.update` called with arguments: (dict[Asset, list[Price | int]]) [no-matching-overload]
jax (https://github.com/google/jax)
- ERROR jax/experimental/mosaic/gpu/utils.py:858:23-69: The type of `known_factor_prod` is unknown; it is inferred as an implicit `Any` [unknown-variable-type]
- ERROR jax/experimental/roofline/rooflines.py:455:15-63: The type of `axes_size` is unknown; it is inferred as an implicit `Any` [unknown-variable-type]
scikit-learn (https://github.com/scikit-learn/scikit-learn)
- ERROR sklearn/compose/tests/test_column_transformer.py:876:9-35: Result of call expression is of type `ndarray | ndarray[tuple[Any, ...], dtype[Unknown]]` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/compose/tests/test_column_transformer.py:876:9-35: Result of call expression is of type `ndarray[tuple[Any, ...], dtype[Any | Unknown]] | ndarray` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/compose/tests/test_column_transformer.py:883:9-35: Result of call expression is of type `ndarray | ndarray[tuple[Any, ...], dtype[Unknown]]` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/compose/tests/test_column_transformer.py:883:9-35: Result of call expression is of type `ndarray[tuple[Any, ...], dtype[Any | Unknown]] | ndarray` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/compose/tests/test_column_transformer.py:2219:9-35: Result of call expression is of type `ndarray | ndarray[tuple[Any, ...], dtype[Unknown]]` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/compose/tests/test_column_transformer.py:2219:9-35: Result of call expression is of type `ndarray[tuple[Any, ...], dtype[Any | Unknown]] | ndarray` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/compose/tests/test_column_transformer.py:2400:9-35: Result of call expression is of type `ndarray | ndarray[tuple[Any, ...], dtype[Unknown]]` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/compose/tests/test_column_transformer.py:2400:9-35: Result of call expression is of type `ndarray[tuple[Any, ...], dtype[Any | Unknown]] | ndarray` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/ensemble/_voting.py:430:38-434:14: No matching overload found for function `numpy.lib._shape_base_impl.apply_along_axis` called with arguments: ((x: Unknown) -> signedinteger[_NBitIntP], axis=Literal[1], arr=ndarray[tuple[Any, ...], dtype[Unknown]]) [no-matching-overload]
+ ERROR sklearn/ensemble/_voting.py:430:38-434:14: No matching overload found for function `numpy.lib._shape_base_impl.apply_along_axis` called with arguments: ((x: Unknown) -> signedinteger[_NBitIntP], axis=Literal[1], arr=ndarray[tuple[Any, ...], dtype[Any | Unknown]]) [no-matching-overload]
- ERROR sklearn/ensemble/_voting.py:498:29-37: No matching overload found for function `numpy._core.shape_base.hstack` called with arguments: (ndarray[tuple[Any, ...], dtype[Unknown]]) [no-matching-overload]
+ ERROR sklearn/ensemble/_voting.py:498:29-37: No matching overload found for function `numpy._core.shape_base.hstack` called with arguments: (ndarray[tuple[Any, ...], dtype[Any | Unknown]]) [no-matching-overload]
- ERROR sklearn/ensemble/tests/test_voting.py:110:9-26: Result of call expression is of type `ndarray[tuple[Any, ...], dtype[Unknown]] | Unknown` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/ensemble/tests/test_voting.py:110:9-26: Result of call expression is of type `ndarray[tuple[Any, ...], dtype[Any | Unknown]] | Unknown` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/ensemble/tests/test_voting.py:112:9-26: Result of call expression is of type `ndarray[tuple[Any, ...], dtype[Unknown]]` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/ensemble/tests/test_voting.py:112:9-26: Result of call expression is of type `ndarray[tuple[Any, ...], dtype[Any | Unknown]]` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/ensemble/tests/test_voting.py:114:9-28: Result of call expression is of type `ndarray[tuple[Any, ...], dtype[Unknown]]` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/ensemble/tests/test_voting.py:114:9-28: Result of call expression is of type `ndarray[tuple[Any, ...], dtype[Any | Unknown]]` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/inspection/_permutation_importance.py:72:24-42: Object of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Unknown]] | Unknown` has no attribute `columns` [missing-attribute]
+ ERROR sklearn/inspection/_permutation_importance.py:72:24-42: Object of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Any | Unknown]] | Unknown` has no attribute `columns` [missing-attribute]
- ERROR sklearn/metrics/tests/test_classification.py:1038:16-1044:6: The type of `cov_ypyp` is unknown; it is inferred as an implicit `Any` [unknown-variable-type]
- ERROR sklearn/metrics/tests/test_common.py:1638:13-1643:14: The type of this argument is unknown [unknown-argument-type]
- ERROR sklearn/model_selection/_validation.py:414:26-28: Cannot set item in `dict[str, list[Unknown] | ndarray[tuple[Any, ...], dtype[Unknown]]]` [unsupported-operation]
+ ERROR sklearn/model_selection/_validation.py:414:26-28: Cannot set item in `dict[str, list[Unknown] | ndarray[tuple[Any, ...], dtype[Any | Unknown]]]` [unsupported-operation]
- ERROR sklearn/tests/test_pipeline.py:1111:9-35: Result of call expression is of type `ndarray | ndarray[tuple[Any, ...], dtype[Unknown]]` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/tests/test_pipeline.py:1111:9-35: Result of call expression is of type `ndarray[tuple[Any, ...], dtype[Any | Unknown]] | ndarray` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/tests/test_pipeline.py:1369:9-38: Result of call expression is of type `ndarray | ndarray[tuple[Any, ...], dtype[Unknown]]` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/tests/test_pipeline.py:1369:9-38: Result of call expression is of type `ndarray[tuple[Any, ...], dtype[Any | Unknown]] | ndarray` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/tests/test_pipeline.py:1391:9-38: Result of call expression is of type `ndarray | ndarray[tuple[Any, ...], dtype[Unknown]]` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/tests/test_pipeline.py:1391:9-38: Result of call expression is of type `ndarray[tuple[Any, ...], dtype[Any | Unknown]] | ndarray` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/utils/_param_validation.py:883:47-51: Type of lambda parameter `self` is unknown [implicit-any-lambda]
- ERROR sklearn/utils/tests/test_indexing.py:227:13-61: Result of call expression is of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Unknown]] | Unknown` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/utils/tests/test_indexing.py:227:13-61: Result of call expression is of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Any | Unknown]] | Unknown` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/utils/tests/test_indexing.py:344:13-51: Result of call expression is of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Unknown]] | Unknown` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/utils/tests/test_indexing.py:344:13-51: Result of call expression is of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Any | Unknown]] | Unknown` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/utils/tests/test_indexing.py:367:9-41: Result of call expression is of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Unknown]] | Unknown` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/utils/tests/test_indexing.py:367:9-41: Result of call expression is of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Any | Unknown]] | Unknown` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/utils/tests/test_indexing.py:373:9-49: Result of call expression is of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Unknown]] | Unknown` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/utils/tests/test_indexing.py:373:9-49: Result of call expression is of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Any | Unknown]] | Unknown` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/utils/tests/test_indexing.py:397:9-54: Result of call expression is of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Unknown]] | Unknown` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/utils/tests/test_indexing.py:397:9-54: Result of call expression is of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Any | Unknown]] | Unknown` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/utils/tests/test_indexing.py:405:9-47: Result of call expression is of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Unknown]] | Unknown` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/utils/tests/test_indexing.py:405:9-47: Result of call expression is of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Any | Unknown]] | Unknown` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/utils/tests/test_indexing.py:428:9-20: Object of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Unknown]] | Unknown` has no attribute `iloc` [missing-attribute]
+ ERROR sklearn/utils/tests/test_indexing.py:428:9-20: Object of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Any | Unknown]] | Unknown` has no attribute `iloc` [missing-attribute]
- ERROR sklearn/utils/tests/test_indexing.py:439:9-43: Result of call expression is of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Unknown]] | Unknown` and is not used; assign to `_` if this is intentional [unused-call-result]
+ ERROR sklearn/utils/tests/test_indexing.py:439:9-43: Result of call expression is of type `list[Unknown] | ndarray[tuple[Any, ...], dtype[Any | Unknown]] | Unknown` and is not used; assign to `_` if this is intentional [unused-call-result]
- ERROR sklearn/utils/validation.py:2580:19-41: No matching overload found for function `numpy._core.fromnumeric.amax` called with arguments: (ndarray[tuple[int], dtype[numpy.bool]] | ndarray[Unknown, Unknown] | None) [no-matching-overload]
+ ERROR sklearn/utils/validation.py:2580:19-41: No matching overload found for function `numpy._core.fromnumeric.amax` called with arguments: (ndarray[tuple[int], dtype[float64]] | ndarray[Unknown, Unknown] | None) [no-matching-overload]
- ERROR sklearn/utils/validation.py:2581:22-44: No matching overload found for function `numpy._core.fromnumeric.amin` called with arguments: (ndarray[tuple[int], dtype[numpy.bool]] | ndarray[Unknown, Unknown] | None) [no-matching-overload]
+ ERROR sklearn/utils/validation.py:2581:22-44: No matching overload found for function `numpy._core.fromnumeric.amin` called with arguments: (ndarray[tuple[int], dtype[float64]] | ndarray[Unknown, Unknown] | None) [no-matching-overload]
core (https://github.com/home-assistant/core)
+ ERROR homeassistant/components/soundtouch/media_player.py:280:53-64: The type of this argument is unknown [unknown-argument-type]
xarray (https://github.com/pydata/xarray)
- ERROR xarray/core/dataset.py:2719:20-26: Object of type `_arrayapi[Any, Any] | _arrayfunction[Any, Any] | ndarray | ndarray[Unknown, dtype[Unknown]] | Unknown` has no attribute `ndim` [missing-attribute]
+ ERROR xarray/core/dataset.py:2719:20-26: Object of type `_arrayapi[Any, Any] | _arrayfunction[Any, Any] | ndarray | ndarray[Unknown, dtype[Any | Unknown]] | ndarray[Unknown, dtype[Unknown]] | Unknown` has no attribute `ndim` [missing-attribute]
- ERROR xarray/core/dataset.py:2724:23-27: Yielded type `tuple[Hashable, _arrayapi[Any, Any] | _arrayfunction[Any, Any] | ndarray | ndarray[Unknown, dtype[Unknown]] | Unknown]` is not assignable to declared yield type `tuple[Hashable, Variable | int | ndarray | slice[Any, Any, Any]]` [invalid-yield]
+ ERROR xarray/core/dataset.py:2724:23-27: Yielded type `tuple[Hashable, _arrayapi[Any, Any] | _arrayfunction[Any, Any] | ndarray | ndarray[Unknown, dtype[Any | Unknown]] | ndarray[Unknown, dtype[Unknown]] | Unknown]` is not assignable to declared yield type `tuple[Hashable, Variable | int | ndarray | slice[Any, Any, Any]]` [invalid-yield]
optuna (https://github.com/optuna/optuna)
- ERROR optuna/visualization/_hypervolume_history.py:114:30-50: `*` is not supported between `ndarray[tuple[int], dtype[signedinteger[_NBitIntP]]]` and `None` [unsupported-operation]
+ ERROR optuna/visualization/_hypervolume_history.py:114:30-50: `*` is not supported between `ndarray[tuple[int], dtype[float64]]` and `None` [unsupported-operation] |
stroxler
left a comment
There was a problem hiding this comment.
Thanks!
I think there are a set of issues remaining to work through (besides needing a rebase) before we can merge this:
-
Nested comprehensions bypass the exponential-work guard ([location](
)). The depth detector still counts only container literals. At 11 nested overload calls, the base finishes in 0.082 s; the PR exceeds 15 s. Constructor nesting also regresses.pyrefly/pyrefly/lib/alt/callable.rs
Lines 116 to 132 in b85bd25
-
Nested
Anyloses known element structure ([location]()). Filtering a comprehension overpyrefly/pyrefly/lib/alt/callable.rs
Lines 814 to 817 in b85bd25
list[list[Any]]yieldslist[object]on the PR, while the base preserveslist[list[Any]]. -
Failed overload constraints leak between candidates ([location](
)). A referenced empty list hidden inside a deferred comprehension is pinned topyrefly/pyrefly/lib/alt/callable.rs
Lines 116 to 132 in b85bd25
intby a rejected candidate, causing the validstroverload and subsequent append to fail.
Summary
Fixes #4167.
When resolving an overloaded call, container literals (
list/set/dict) are kept as expressions rather than eagerly inferred, so each candidate overload can contextually type them against its parameter (and narrow element literals). Comprehensions and generator expressions were not — they fell through to a context-freeexpr_inferinCallWithTypes::type_or_expr, so:was inferred as
list[tuple[str, str]]and reportedno-matching-overload, even though mypy, pyright, and ty all accept it (reported against Narwhals).The fix adds
Expr::ListComp | Expr::SetComp | Expr::DictComp | Expr::Generatorto the same un-flattened arm. These variants already support hint-directed inference viainfer_with_decomposed_hint/decompose_list, so deferring their inference lets the existing per-overload contextual-typing path narrow their element/key/value types exactly as it does for container literals. When a hint doesn't decompose (e.g. a generator against a non-Sequenceparam), inference falls back to the previous context-free behaviour, so no path is worse than before.Test Plan
Added
test_overload_literal_narrowing_through_comprehension(the reported repro) andtest_overload_literal_narrowing_through_set_dict_generator(set/dict comprehensions and generators) topyrefly/lib/test/overload.rs. Both fail before the change (no-matching-overload,reveal_type=Unknown) and pass after (reveal_type=str).cargo test -p pyrefly --libpasses;test.pyformatting and linting are clean.