Reconcile desired VM state with observed agent state

[halfcyan]

Summary

Recover VM inventory and runtime state after manager, agent, or cluster restarts by reconciling etcd records with agent observations.

Scope

• Load desired VM state from etcd.
• Discover observed VMs from agents/runtime directories.
• Recreate missing actors for desired VMs.
• Represent desired-but-unavailable VMs as pending/error.
• Define policy for unknown runtime VMs.
• Make reconciliation idempotent.

Tasks

• Define desired, observed, pending, error, and unknown VM reconciliation states.
• Load desired VM records from etcd during manager startup.
• Discover observed VMs from agents and local runtime directories.
• Recreate missing VM actors for desired VMs.
• Mark desired-but-unavailable VMs as pending or error with a reason.
• Define and implement handling for unknown runtime VMs.
• Make reconciliation safe to run repeatedly.
• Add manager-restart and agent-restart tests with mocked agents.
• Expose desired and observed state distinctly through the API.

What this should do

• Manager restart restores VM inventory.
• Agent restart does not permanently hide healthy local VMs.
• Repeated reconciliation does not create duplicate VMs.
• Desired and observed state are distinguishable in API responses.
• Recovery behavior has mocked-agent tests.

Dependencies

• Depends on draft issue 03.

Related issues

• Related to VM Agents: Live Migration #15 (live migration), VM Agents: Configuration #16 (agent configuration), and Placement groups #28 (placement groups).

Issue links

• Depends on #100.
• Related to existing #15, #16, and #28.