[johnbowen42]

Currently if we have LLVM IR shapes like

// Assume %0 is a closure pointer
store int32 %1, %0
store int32 %2, %0
call registered_lambda(%0)

our analysis of the lambda callsite cannot automatically tell which store to trust as the definitive source of truth for the first slot of the lambda at the callsite. Fortunately, LLVM has O(N) (N is size of the function) built-in analysis called MemorySSA, which can answer, at the callsite call registered_lambda(%0), which store is definitive. In the parlance of MemorySSA, we say the instruction store int32 %2, %0 "clobbers", where clobbering basically means is the source of truth for that memory value and the callsite.

Fortunately, this LLVM IR shape is relatively exotic for lambda closure pointer. Nonetheless, we want Proteus to conservatively and correctly deal with simple cases of memory clobbering, explicitly reject complex LLVM IR shapes, and above all not trust the def-use graph traversal ordering as a source of ground truth for memory locations. This PR introduces a MemorySSA oracle model, that the def-use walk uses as a guide when analyzing instructions that write to lambda closure pointers. The oracle allows the analysis to ask, "which instruction clobbers the lambda closure pointer at a given lambda callsite??" Of course, an adversarial user can cook up examples where lambda closures pass through phi nodes, or a lambda is re-assembled piecemeal from disparate memory locations. These are all cases we currently conservatively reject, and update the clang frontend to do help users understand why.

Here are some of the edge cases where we fail to specialize
(1) What if the analysis finds that different slots in a lambda are stored from different pointers? -- fail to specialize
(2) What if we find that multiple user-defined function calls determine the value of different slots of the lambda closure -- fail to specialize
(3) What if we find multiple memcpys that copy different lambda slots from different source pointers? -- fail to specialize

Here are some cases where we do specialize
✅ the lambda closure pointer is clobbered at the callsite by a collection of stores, all from the same base ptr
✅ the lambda closure pointer is stored to multiple times, only one store clobbers
✅ the lambda closure pointer is clobbered at the callsite by a memory intrinsic
✅ the lambda closure pointer is clobbered at the callsite by a function call, which writes to the closure pointer using one of the above patterns

We try to handle all unsupported LLVM IR patterns from the Clang frontend which is also updated here. I also have added a flag to weaken the Clang frontend so that we can test that the LLVM IR analysis also knows about the bad shapes I describe above.

[johnbowen42]

marking as draft temporarily while I push up several commits