Beyond the Current Scene:
Event-Referential Grasping with Active View Selection

Hyunjoon Lee^1,* · Haebeom Jung^1,* · Eunsung Cha¹ · Daeun Lee¹ · Yu-Chiang Frank Wang²
Jaesung Choe² · Jaesik Park^1,†

¹Seoul National University  ·  ²NVIDIA
^*Equal contribution  ·  ^†Corresponding author

Code coming soon!

TL;DR

An instruction such as “Pick up the object moved last” refers to a past event, but the target may be hidden when the robot is asked to act. As the figure shows, identifying the object alone does not help a robot grasp what it cannot see. BeyondCSe uses the event history to determine both which object is meant and where to look, moving the camera to reveal the target before grasping it.