Skip to content

fix: exclude descendants when finding a clear tap point (#199) - #435

Open
8crsk wants to merge 1 commit into
droidrun:mainfrom
8crsk:fix/clear-point-excludes-descendants
Open

fix: exclude descendants when finding a clear tap point (#199)#435
8crsk wants to merge 1 commit into
droidrun:mainfrom
8crsk:fix/clear-point-excludes-descendants

Conversation

@8crsk

@8crsk 8crsk commented Sep 3, 2026

Copy link
Copy Markdown

Closes #199.

The report says overlapping elements cause incorrect clicking and that the tree
needs to respect hierarchy. Here's the specific mechanism, and it's a bit worse
than mis-clicking: any container whose children fill it can't be tapped at all.

Why it happens

IndexedFormatter._flatten_with_index assigns indices with a pre-order walk, so
every descendant ends up with a higher index than its ancestor.

UIState.get_clear_point treats any higher-indexed overlapping element as
something drawn on top of the target:

if el_idx is not None and el_idx > index and el_bounds_str:
    if rects_overlap(target_bounds, el_bounds):
        blockers.append(el_bounds)

Descendants satisfy that test. But a descendant is painted inside its ancestor,
not over the top of it, so an element's own children get treated as obstructions
and find_clear_point runs out of room.

Reproduction

An ordinary clickable row with an icon and a label, not a corner case:

LinearLayout  index=5  bounds=0,100,1000,300     <- the tappable row
  ImageView   index=6  bounds=0,100,200,300
  TextView    index=7  bounds=200,100,1000,300

The two children exactly cover the parent.

Before: get_clear_point(5) raises Element 5 is fully obscured by overlapping elements
After: get_clear_point(5) returns (500, 200), the centre of the row

Tapping the leaf children (6, 7) worked before and still works, which is probably
why this survived. It only bites when the agent targets the container, and the
container is often the only node carrying the click handler.

The fix

Skip descendants when building the blocker list. That's enough:

  • Ancestors are already excluded, since an ancestor always has a lower index.
  • Genuine overlays (a snackbar, dialog or FAB drawn after the target) keep their
    higher index and are still treated as blockers.

I also corrected the docstring, which promised a fallback to the centre that the
implementation has never done. It raises instead. Raising seems right for an
element that really is covered, so I assumed the docstring was the wrong half.
Happy to flip it if you'd rather have the documented behaviour.

Tests

tests/test_clear_point_nesting.py, 10 cases: nested children, deeply nested
descendants, a partial overlay the point has to dodge, a full overlay that should
still raise, an overlay over a container that also has children, ancestor
exclusion, and elements missing an index or bounds.

I checked the tests actually catch the bug. Three of them fail against current
main, and the seven covering overlay behaviour pass both before and after, so
existing behaviour is unchanged.

Full suite: 653 passed. Four tests fail on my machine
(test_grok_oauth, test_manager_prompt_selection,
test_visual_remote_connection) but they're a pre-existing Windows cp1252
decode issue and fail identically on a clean checkout of main. ruff and
black are clean.

Element indices are assigned by a pre-order walk of the accessibility tree
(IndexedFormatter._flatten_with_index), so every descendant of an element has a
higher index than the element itself.

get_clear_point treats any higher-indexed overlapping element as something
drawn on top of the target. Descendants match that test, but a descendant is
painted *inside* its ancestor rather than over the top of it. The result is
that a container whose children happen to fill it is reported as obscured and
cannot be tapped at all.

This is an ordinary Android layout, not a corner case -- a clickable row
holding an icon and a label:

    LinearLayout  index=5  bounds=0,100,1000,300     <- the tappable row
      ImageView   index=6  bounds=0,100,200,300
      TextView    index=7  bounds=200,100,1000,300

Before: get_clear_point(5) raised
        "Element 5 is fully obscured by overlapping elements"
After:  get_clear_point(5) returns (500, 200), the centre of the row.

Skipping descendants is enough to fix it. Ancestors are already excluded by the
existing index comparison, since an ancestor always has a lower index, and
genuine overlays -- a snackbar, dialog or FAB drawn after the target -- keep
their higher index and are still treated as blockers.

Also corrects the docstring, which promised a fallback to the centre that the
implementation has never done; it raises instead. Raising is the right
behaviour for an element that really is covered, so the docstring was the part
that was wrong.

Adds tests/test_clear_point_nesting.py covering nested children, deeply nested
descendants, partial and full overlays, ancestor exclusion, and elements
missing an index or bounds.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Accessibility Tree Overlapping Elements Issue in Android App Control

1 participant