feat(GSP1314): Aeron cluster failover lab solution and full Hashnode guide

labs/GSP1314/HASHNODE.md

@@ -0,0 +1,175 @@
+# Enable Native Exchange Messaging and Clustering with Aeron - GSP1314
+
+## Overview
+
+This lab demonstrates a replicated Aeron Cluster for an auction workload. Three preprovisioned Compute Engine VMs run Aeron members, elect a leader, process bids, recover from leader failure, and synchronize a restarted member. This guide is independently written from the public official task list.
+
+### What you'll do
+
+- Start the three Aeron members and confirm leader election.
+- Send 20 sample auction bids.
+- Inspect Aeron statistics, membership, and process IDs.
+- Stop the elected leader, observe failover, then rejoin it while processing 300 bids.
+
+## Setup and requirements
+
+### Before you click the Start Lab button
+
+Open the [official GSP1314 lab](https://www.cloudskillsboost.google/focuses/124773?parent=catalog). Start it in a private browser and use **only** the temporary student account supplied in the lab panel. Never publish passwords, usernames, project IDs, or access tokens. Wait for the three supplied VMs named aeron-cluster-node1, aeron-cluster-node2, and aeron-cluster-node3 to be running. The Aeron sample software is already installed; do not create new VMs.
+
+### Activate Cloud Shell
+
+Open and authorize Cloud Shell, then identify the active project, zones, and internal IPs:
+
+```bash
+gcloud config get-value project
+gcloud compute instances list --format='table(name,zone,status,networkInterfaces[0].networkIP)'
+for n in 1 2 3; do
+  VM="aeron-cluster-node$n"
+  ZONE="$(gcloud compute instances list --filter="name=$VM" --format='value(zone.basename())' | head -n 1)"
+  IP="$(gcloud compute instances describe "$VM" --zone="$ZONE" --format='value(networkInterfaces[0].networkIP)')"
+  printf '%s | %s | %s\n' "$VM" "$ZONE" "$IP"
+done
+```
+
+Record the private addresses in node1,node2,node3 order. The script automatically discovers project and zone instead of hardcoding them.
+
+## Task 1. Start the Aeron Cluster
+
+Open SSH terminals to all three VMs. Map node1 to member 0, node2 to member 1, and node3 to member 2. On each VM, set THIS_HOST to that VM's internal IP, and HOST_NAMES to the comma-separated IPs of all three VMs in order. For node1, use:
+
+```bash
+export THIS_HOST='<NODE1_PRIVATE_IP>'
+export HOST_NAMES='<NODE1_PRIVATE_IP>,<NODE2_PRIVATE_IP>,<NODE3_PRIVATE_IP>'
+export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
+cd "$HOME/aeron/aeron-samples/scripts/cluster"
+if ! grep -q 'java.base/sun.nio.ch=ALL-UNNAMED' script-common; then
+  sed -i '/^ADD_OPENS=/ s|"$| --add-opens java.base/sun.nio.ch=ALL-UNNAMED"|' script-common
+fi
+./basic-auction-cluster 0
+```
+
+Repeat on node2 with THIS_HOST set to its IP and the final argument changed to 1; repeat on node3 with argument 2. Leave the three terminals running. Confirm that leader election and follower readiness appear in the logs.
+
+### Test completed task
+
+Click **Check my progress** for Start the Aeron Cluster (30 points). The script cannot confirm grader success.
+
+## Task 2. Start the Sample Auction Client
+
+Open another SSH terminal on node1, keeping its cluster member running. Set the same IP variables and send 20 bids from client ID 101:
+
+```bash
+export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
+export THIS_HOST='<NODE1_PRIVATE_IP>'
+export HOST_NAMES='<NODE1_PRIVATE_IP>,<NODE2_PRIVATE_IP>,<NODE3_PRIVATE_IP>'
+cd "$HOME/aeron/aeron-samples/scripts/cluster"
+./basic-auction-client 101 20
+```
+
+Review the client output and cluster logs for the sent bids and updated auction state. Actual values and timestamps may vary.
+
+### Test completed task
+
+Click **Check my progress** for Start the Auction Client (30 points).
+
+## Task 3. Run Administrative Tools For the Cluster
+
+From node1, run the Aeron statistics command:
+
+```bash
+cd "$HOME/aeron/aeron-samples/scripts"
+./aeron-stat watch=false
+```
+
+Inspect the cluster status, leader member ID, and PID:
+
+```bash
+cd "$HOME/aeron/aeron-samples/scripts/cluster"
+JAR="$(find "$HOME/aeron/aeron-all/build/libs" -maxdepth 1 -name 'aeron-all-*.jar' -print -quit)"
+java -cp "$JAR" io.aeron.cluster.ClusterTool node0/cluster list-members
+java -cp "$JAR" io.aeron.cluster.ClusterTool node0/cluster describe
+java -cp "$JAR" io.aeron.cluster.ClusterTool node0/cluster pid
+```
+
+Read the leaderMemberId value; do not assume member 0 is always the leader. This inspection has no separate grader checkpoint.
+
+## Task 4. Simulate Failover of the Cluster
+
+Identify the current leader from Task 3. On that member's VM, query its PID using the appropriate node0/cluster, node1/cluster, or node2/cluster path:
+
+```bash
+cd "$HOME/aeron/aeron-samples/scripts/cluster"
+JAR="$(find "$HOME/aeron/aeron-all/build/libs" -maxdepth 1 -name 'aeron-all-*.jar' -print -quit)"
+java -cp "$JAR" io.aeron.cluster.ClusterTool node0/cluster pid
+```
+
+The example targets member 0; replace it with the observed leader. Verify the returned PID belongs to the Aeron cluster, then stop only that process:
+
+```bash
+ps -p '<AERON_PID>' -o pid,args
+kill -9 '<AERON_PID>'
+```
+
+From a surviving VM, inspect list-members again and wait until another member is elected. Do not stop the VMs or unrelated processes.
+
+### Test completed task
+
+Click **Check my progress** for Simulate Failover by Stopping the Leader Node (30 points).
+
+## Task 5. Restart the Previous Leader Node
+
+On a surviving node, verify the new leader and launch a 300-bid client session, using the correct private IP for THIS_HOST:
+
+```bash
+export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
+export THIS_HOST='<CURRENT_NODE_PRIVATE_IP>'
+export HOST_NAMES='<NODE1_PRIVATE_IP>,<NODE2_PRIVATE_IP>,<NODE3_PRIVATE_IP>'
+cd "$HOME/aeron/aeron-samples/scripts/cluster"
+./basic-auction-client 101 300
+```
+
+While the bids are being sent, return to the stopped member's VM. Restart its original cluster member ID:
+
+```bash
+export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
+export THIS_HOST='<PREVIOUS_LEADER_PRIVATE_IP>'
+export HOST_NAMES='<NODE1_PRIVATE_IP>,<NODE2_PRIVATE_IP>,<NODE3_PRIVATE_IP>'
+cd "$HOME/aeron/aeron-samples/scripts/cluster"
+./basic-auction-cluster '<PREVIOUS_LEADER_MEMBER_NUMBER>'
+```
+
+Check membership and recovery logs for catch-up activity. If the client finishes before recovery is observable, repeat the batch as the official lab permits.
+
+### Test completed task
+
+Click **Check my progress** for Start the Previous Leader Node (10 points). The checkpoints total 100 points, but this guide does not claim grader PASS.
+
+---
+
+## Solution of Lab
+
+### Quick
+
+After the Draft PR is merged into main, run exactly these two commands in an authorized lab Cloud Shell. The URL deliberately points to main and cannot retrieve the new script before merge.
+
+```bash
+curl -LO https://raw.githubusercontent.com/ePlus-DEV/storage/refs/heads/main/labs/GSP1314/lab.sh
+source lab.sh
+```
+
+The script detects the project, VM zones, and private IPs; starts the cluster; submits bids; inspects membership; performs the required failover; and attempts to restore the previous leader. FORCE_COLOR=1 enables colors through tee, while NO_COLOR disables them. Task 4 intentionally stops the elected leader's lab process.
+
+### Manual
+
+Chưa tìm thấy video YouTube hướng dẫn thủ công đã xác minh cho lab này.
+
+The five manual Task sections above are the full walkthrough; a video is not necessary.
+
+## Sources and verification
+
+- [Official GSP1314 instructions, five Tasks, and four checkpoints](https://www.cloudskillsboost.google/focuses/124773?parent=catalog)
+- [Aeron open-source project](https://github.com/real-logic/aeron)
+- [Compute Engine SSH CLI documentation](https://cloud.google.com/sdk/gcloud/reference/compute/ssh)
+
+**Status:** static validated only; runtime/grader unverified. Google Skills screenshots are not embedded because permission to reuse the original images has not been verified. This file is ready for Hashnode review, not published.

labs/GSP1314/lab.sh

@@ -0,0 +1,109 @@
+#!/usr/bin/env bash
+# Copyright (c) 2026 ePlus.DEV | Unlicense
+set -Eeuo pipefail
+
+if [[ -v NO_COLOR ]]; then C1='' C2='' C3='' RESET='';
+elif [[ -t 1 || "${FORCE_COLOR:-0}" == 1 ]]; then
+  C1=$'\033[1;36m'; C2=$'\033[1;32m'; C3=$'\033[1;33m'; RESET=$'\033[0m'
+else C1='' C2='' C3='' RESET=''; fi
+log() { printf '%s%s%s\n' "$C1" "$*" "$RESET"; }
+ok() { printf '%s%s%s\n' "$C2" "$*" "$RESET"; }
+warn() { printf '%s%s%s\n' "$C3" "$*" "$RESET"; }
+trap 'warn "Execution failed at line $LINENO (exit $?)."' ERR
+printf '%s\n' "${C1}ePlus.DEV – Starting Execution${RESET}" 'GSP1314 — Aeron Cluster (static validated only; runtime/grader unverified)'
+for cmd in gcloud awk grep sed java; do
+  if [[ "$cmd" == java ]]; then continue; fi
+  command -v "$cmd" >/dev/null || { warn "Missing required command: $cmd"; exit 1; }
+done
+PROJECT_ID="${PROJECT_ID:-$(gcloud config get-value project 2>/dev/null)}"
+[[ -n "$PROJECT_ID" && "$PROJECT_ID" != '(unset)' ]] || { warn 'Select a Google Cloud lab project first.'; exit 1; }
+declare -A ZONES IPS
+VMS=(aeron-cluster-node1 aeron-cluster-node2 aeron-cluster-node3)
+for vm in "${VMS[@]}"; do
+  zone="$(gcloud compute instances list --project="$PROJECT_ID" --filter="name=$vm" --format='value(zone.basename())' | head -n 1)"
+  [[ -n "$zone" ]] || { warn "Preprovisioned VM $vm not found"; exit 1; }
+  ZONES["$vm"]="$zone"
+  ip="$(gcloud compute instances describe "$vm" --project="$PROJECT_ID" --zone="$zone" --format='value(networkInterfaces[0].networkIP)')"
+  [[ "$ip" =~ ^[0-9.]+$ ]] || { warn "No private IP for $vm"; exit 1; }
+  IPS["$vm"]="$ip"
+done
+HOST_NAMES="${IPS[${VMS[0]}]},${IPS[${VMS[1]}]},${IPS[${VMS[2]}]}"
+remote() {
+  local vm="$1" script="$2" attempt
+  for attempt in 1 2 3; do
+    if gcloud compute ssh "$vm" --project="$PROJECT_ID" --zone="${ZONES[$vm]}" --quiet --command="bash -lc $(printf '%q' "$script")"; then return 0; fi
+    warn "SSH retry $attempt/3 for $vm"; sleep 8
+  done
+  return 1
+}
+start_node() {
+  local idx="$1" vm="${VMS[$1]}" script
+  script="set -e
+export THIS_HOST='${IPS[$vm]}' HOST_NAMES='$HOST_NAMES' JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
+cd \"\$HOME/aeron/aeron-samples/scripts/cluster\"
+if ! grep -q 'java.base/sun.nio.ch=ALL-UNNAMED' script-common; then
+  sed -i '/^ADD_OPENS=/ s|\"\$| --add-opens java.base/sun.nio.ch=ALL-UNNAMED\"|' script-common
+fi
+pidfile=\"\$HOME/.eplus-aeron-${idx}.pid\"
+if [[ -f \"\$pidfile\" ]] && kill -0 \"\$(cat \"\$pidfile\")\" 2>/dev/null && [[ \"\$(ps -o stat= -p \"\$(cat \"\$pidfile\")\" 2>/dev/null)\" != Z* ]]; then
+  echo 'Aeron node ${idx} already running'; exit 0
+fi
+nohup ./basic-auction-cluster ${idx} > \"\$HOME/aeron-node-${idx}.log\" 2>&1 < /dev/null &
+echo \"\$!\" > \"\$pidfile\"
+sleep 2
+kill -0 \"\$(cat \"\$pidfile\")\" 2>/dev/null || { tail -n 30 \"\$HOME/aeron-node-${idx}.log\"; exit 1; }
+echo 'Aeron node ${idx} started'"
+  remote "$vm" "$script"
+}
+cluster_tool() {
+  local idx="$1" vm="${VMS[$1]}" action="$2" script
+  script="set -e; cd \"\$HOME/aeron/aeron-samples/scripts/cluster\"; jar=\$(find \"\$HOME/aeron/aeron-all/build/libs\" -maxdepth 1 -name 'aeron-all-*.jar' -print -quit); [[ -n \"\$jar\" ]]; java -cp \"\$jar\" io.aeron.cluster.ClusterTool node${idx}/cluster ${action}"
+  remote "$vm" "$script"
+}
+get_leader() {
+  local idx="$1" result
+  result="$(cluster_tool "$idx" list-members 2>/dev/null || true)"
+  if [[ "$result" =~ leaderMemberId=([012]) ]]; then printf '%s' "${BASH_REMATCH[1]}"; else return 1; fi
+}
+log '== Task 1/5 — Start the three preprovisioned Aeron nodes =='
+for idx in 0 1 2; do start_node "$idx"; done
+leader=''
+for attempt in {1..12}; do
+  leader="$(get_leader 0 || true)"
+  [[ "$leader" =~ ^[012]$ ]] && break
+  sleep 5
+done
+[[ "$leader" =~ ^[012]$ ]] || { warn 'Leader election not confirmed'; exit 1; }
+ok "Cluster leader: member $leader"
+log '== Task 2/5 — Submit 20 auction bids =='
+remote "${VMS[0]}" "set -e; export THIS_HOST='${IPS[${VMS[0]}]}' HOST_NAMES='$HOST_NAMES' JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64; cd \"\$HOME/aeron/aeron-samples/scripts/cluster\"; ./basic-auction-client 101 20"
+ok '20-bid client completed'
+log '== Task 3/5 — Inspect cluster statistics and membership =='
+remote "${VMS[0]}" "cd \"\$HOME/aeron/aeron-samples/scripts\"; ./aeron-stat watch=false | head -n 25" || warn 'AeronStat unavailable; continuing with cluster tools'
+cluster_tool 0 describe
+cluster_tool 0 list-members
+log '== Task 4/5 — Stop current leader and verify failover =='
+leader="$(get_leader 0)"
+[[ "$leader" =~ ^[012]$ ]] || exit 1
+leader_vm="${VMS[$leader]}"
+remote "$leader_vm" "set -e; cd \"\$HOME/aeron/aeron-samples/scripts/cluster\"; jar=\$(find \"\$HOME/aeron/aeron-all/build/libs\" -maxdepth 1 -name 'aeron-all-*.jar' -print -quit); pid=\$(java -cp \"\$jar\" io.aeron.cluster.ClusterTool node${leader}/cluster pid | tr -dc '0-9'); [[ \"\$pid\" =~ ^[0-9]+\$ ]]; ps -p \"\$pid\" -o args= | grep -qi aeron; kill -9 \"\$pid\"; echo \"Stopped Aeron member ${leader}\""
+new_leader=''
+probe=$(( (leader + 1) % 3 ))
+for attempt in {1..18}; do
+  new_leader="$(get_leader "$probe" || true)"
+  [[ "$new_leader" =~ ^[012]$ && "$new_leader" != "$leader" ]] && break
+  sleep 5
+done
+[[ "$new_leader" =~ ^[012]$ && "$new_leader" != "$leader" ]] || { warn 'New leader not confirmed'; exit 1; }
+ok "New leader: member $new_leader"
+log '== Task 5/5 — Submit 300 bids and rejoin previous leader =='
+new_vm="${VMS[$new_leader]}"
+remote "$new_vm" "set -e; export THIS_HOST='${IPS[$new_vm]}' HOST_NAMES='$HOST_NAMES' JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64; cd \"\$HOME/aeron/aeron-samples/scripts/cluster\"; nohup ./basic-auction-client 101 300 > \"\$HOME/aeron-bids-300.log\" 2>&1 < /dev/null & echo '300-bid client launched'"
+start_node "$leader"
+for attempt in {1..12}; do
+  if cluster_tool "$leader" list-members >/dev/null 2>&1; then break; fi
+  sleep 5
+done
+cluster_tool "$new_leader" list-members
+ok 'Aeron cluster workflow complete. Check all four Google Skills checkpoints manually.'
+warn 'static validated only; runtime/grader unverified'