diff --git a/labs/GSP1288/HASHNODE.md b/labs/GSP1288/HASHNODE.md new file mode 100644 index 0000000..250f22a --- /dev/null +++ b/labs/GSP1288/HASHNODE.md @@ -0,0 +1,206 @@ +# Configuring Vector Search in Spanner - GSP1288 + +## Overview + +This Google Skills lab uses an existing Spanner instance and a Google-hosted embedding model to demonstrate semantic product search. The exercise creates a remote model, loads ten product records, generates vectors, and ranks available products by cosine distance. This guide is independently written from the publicly accessible task requirements. + +### What you'll do + +- Configure a remote embeddings model in a lab-provisioned Spanner database. +- Create a product schema and insert ten product records. +- Authorize the Spanner service agent to call the model and generate embeddings. +- Run a similarity query and verify the four graded checkpoints. + +## Setup and requirements + +### Before you click the Start Lab button + +Open the [official GSP1288 lab](https://www.skills.google/focuses/119411). Use a private browsing window and sign in with the temporary lab account shown in the lab panel. Do not use your personal Google Cloud account, and do not publish the temporary username, password, project ID, or any access token. The lab is timed and resources are temporary. + +The lab provides a Spanner instance named `cymbal-retail` and a GoogleSQL database named `cymbal-bicycles`. Wait for these resources to become ready. The model name and region must match your lab's current configuration. + +### Activate Cloud Shell + +Open Cloud Shell in the Google Cloud console, authorize it using the lab account, and confirm the active project: + +```bash +gcloud config get-value project +gcloud spanner instances describe cymbal-retail --format='value(config)' +gcloud spanner databases describe cymbal-bicycles --instance=cymbal-retail +``` + +Set variables from your active project and lab configuration. The sample model identifier below can be overridden if the lab displays another value: + +```bash +export PROJECT_ID="$(gcloud config get-value project)" +export INSTANCE=cymbal-retail +export DATABASE=cymbal-bicycles +export REGION="$(gcloud spanner instances describe "$INSTANCE" --format='value(config)' | sed -nE 's/.*regional-([a-z]+-[a-z0-9]+[0-9]).*/\1/p')" +export MODEL_ID="${MODEL_ID:-gemini-embedding-001}" +printf 'Project: %s | Region: %s | Model: %s\n' "$PROJECT_ID" "$REGION" "$MODEL_ID" +``` + +If the instance is multi-regional or the displayed region differs, set `REGION` to the lab-provided Agent Platform endpoint location. Do not guess the region. + +## Task 1. Create an embeddings model in Spanner and configure it to an Agent Platform model endpoint + +In the Google Cloud console, open Spanner, select the `cymbal-retail` instance and `cymbal-bicycles` database, then open Spanner Studio. Create a new SQL tab. The remote model passes text to the Google model and returns a vector and statistics. + +From Cloud Shell, the equivalent DDL is: + +```bash +gcloud spanner databases ddl update "$DATABASE" --instance="$INSTANCE" --project="$PROJECT_ID" --ddl=" +CREATE MODEL EmbeddingsModel +INPUT(content STRING(MAX),) +OUTPUT(embeddings STRUCT, values ARRAY>,) +REMOTE OPTIONS(endpoint='//aiplatform.googleapis.com/projects/$PROJECT_ID/locations/$REGION/publishers/google/models/$MODEL_ID')" +``` + +Only create the model if it does not already exist; rerunning the same `CREATE MODEL` statement produces an already-exists error. + +### Test completed task + +Use the first Google Skills **Check my progress** checkpoint to verify creation of `EmbeddingsModel`. Check its DDL using: + +```bash +gcloud spanner databases ddl describe "$DATABASE" --instance="$INSTANCE" --project="$PROJECT_ID" +``` + +## Task 2. Create a new table and load products data + +Create the products table in Spanner Studio or with the CLI. The `ARRAY` column will hold the model output; the timestamp column allows commit timestamps. + +```sql +CREATE TABLE products ( + categoryId INT64 NOT NULL, + productId INT64 NOT NULL, + productName STRING(MAX) NOT NULL, + productDescription STRING(MAX) NOT NULL, + productDescriptionEmbedding ARRAY, + createTime TIMESTAMP NOT NULL OPTIONS (allow_commit_timestamp = true), + inventoryCount INT64 NOT NULL, + priceInCents INT64 +) PRIMARY KEY(categoryId, productId); +``` + +To run this from Cloud Shell, save the DDL as `products.sql` and use `gcloud spanner databases ddl update "$DATABASE" --instance="$INSTANCE" --ddl-file=products.sql`. + +Load ten bicycle-related products. The independently authored sample below uses the same ten product identities and inventory fields as the lab, with shorter original descriptions rather than reproducing copyrighted source prose. For exact reference data, consult the official lab. Execute in Spanner Studio: + +```sql +INSERT INTO products (categoryId,productId,productName,productDescription,createTime,inventoryCount,priceInCents) +VALUES +(1,1,'Cymbal Helios Helmet','Adjustable youth helmet with vents and a comfortable protective fit.',PENDING_COMMIT_TIMESTAMP(),100,10999), +(1,2,'Cymbal Sprout','Small balance bicycle designed for toddlers aged two to four.',PENDING_COMMIT_TIMESTAMP(),10,13999), +(1,3,'Cymbal Spark Jr.','First pedal bicycle for young riders learning confident handling.',PENDING_COMMIT_TIMESTAMP(),34,13900), +(1,4,'Cymbal Summit','Mountain bicycle with suspension for rugged off road trails.',PENDING_COMMIT_TIMESTAMP(),0,79999), +(1,5,'Cymbal Breeze','Electric commuter bicycle with assisted pedaling and battery power.',PENDING_COMMIT_TIMESTAMP(),72,129999), +(1,6,'Cymbal Trailblazer Backpack','Weather resistant backpack for cycling tools and supplies.',PENDING_COMMIT_TIMESTAMP(),24,7999), +(1,7,'Cymbal Phoenix Lights','Front and rear bicycle lights for visibility after sunset.',PENDING_COMMIT_TIMESTAMP(),87,3999), +(1,8,'Cymbal Windstar Pump','Compact portable bicycle tire pump for quick inflation.',PENDING_COMMIT_TIMESTAMP(),36,24999), +(1,9,'Cymbal Odyssey Multi-Tool','Pocket tool kit with screwdrivers and bicycle repair keys.',PENDING_COMMIT_TIMESTAMP(),52,999), +(1,10,'Cymbal Nomad Water Bottle','Reusable leak resistant bottle for hydration on rides.',PENDING_COMMIT_TIMESTAMP(),42,1299); +``` + +Confirm the table contains ten rows: + +```sql +SELECT COUNT(*) AS product_count FROM products WHERE categoryId=1; +``` + +### Test completed task + +Use the second Google Skills checkpoint after the table and ten records exist. Do not insert duplicates on a rerun; the automated script uses per-product existence checks. + +## Task 3. Generate and store text embeddings for the products data + +Find the Google Cloud project number and grant the lab-specified **Cloud Spanner API Service Agent** role to the Spanner-managed service agent. This permission is specifically required for the model endpoint integration. + +```bash +PROJECT_NUMBER="$(gcloud projects describe "$PROJECT_ID" --format='value(projectNumber)')" +SPANNER_AGENT="service-${PROJECT_NUMBER}@gcp-sa-spanner.iam.gserviceaccount.com" +gcloud projects add-iam-policy-binding "$PROJECT_ID" \ + --member="serviceAccount:$SPANNER_AGENT" \ + --role='roles/spanner.serviceAgent' --condition=None +``` + +After IAM propagation, run the following SQL in Spanner Studio: + +```sql +UPDATE products p1 +SET productDescriptionEmbedding = ( + SELECT embeddings.values FROM ML.PREDICT( + MODEL EmbeddingsModel, + (SELECT productDescription AS content FROM products p2 WHERE p2.productId=p1.productId) + ) +) +WHERE categoryId=1 AND productDescriptionEmbedding IS NULL; +``` + +Check for ten populated vectors: + +```sql +SELECT COUNT(*) AS embedded_products +FROM products +WHERE categoryId=1 AND productDescriptionEmbedding IS NOT NULL; +``` + +If model invocation returns a permission error, wait briefly for IAM propagation and retry. Verify that the model ID and region are correct before changing permissions. + +### Test completed task + +Run the third Google Skills checkpoint after the product embeddings have been stored. + +## Task 4. Perform vector search using text embeddings in Spanner + +Generate an embedding for a search phrase and rank products by cosine distance. Filter out unavailable inventory and return the five closest matches: + +```sql +SELECT productName, productDescription, inventoryCount, + COSINE_DISTANCE(productDescriptionEmbedding, ( + SELECT embeddings.values + FROM ML.PREDICT(MODEL EmbeddingsModel, + (SELECT 'starter bicycle for a three year old child' AS content)) + )) AS distance +FROM products +WHERE inventoryCount > 0 AND productDescriptionEmbedding IS NOT NULL +ORDER BY distance +LIMIT 5; +``` + +Smaller distances indicate closer semantic similarity. The balance bicycle should be relevant to a toddler search, although the exact ranking may vary with the embedding model and sample descriptions. + +### Test completed task + +Run the fourth Google Skills checkpoint. The automated script performs a final count of total rows and non-null vectors, but cannot invoke the grader itself. + +--- + +## Solution of Lab + +### Quick + +Run these two commands **only after the Draft PR has been merged into `main`**. Until then the main-branch URL may return 404. + +```bash +curl -LO https://raw.githubusercontent.com/ePlus-DEV/storage/refs/heads/main/labs/GSP1288/lab.sh +source lab.sh +``` + +The script auto-detects the active project and Spanner region, honors `NO_COLOR` and `FORCE_COLOR=1`, checks for existing schema/data, and retries transient failures. Override `MODEL_ID` or `REGION` only when your lab configuration requires it. + +### Manual + +Chưa tìm thấy video YouTube hướng dẫn thủ công đã xác minh cho lab này. + +The full manual task sequence and SQL commands are provided above; a video is optional and does not replace the lab instructions. + +## Sources and verification + +- [Official Google Skills GSP1288](https://www.skills.google/focuses/119411) — official task and checkpoint order. +- [Google Codelab: Getting started with Spanner Vector Search](https://codelabs.developers.google.com/spanner-getting-started-vector-search) — related technical background, not a substitute for the lab. +- [Spanner DML via gcloud](https://docs.cloud.google.com/spanner/docs/dml-tasks) and [Spanner DDL CLI](https://docs.cloud.google.com/sdk/gcloud/reference/spanner/databases/ddl/update). + +Image assets from the Google Skills page have not been embedded because reuse and public hotlink permissions have not been verified. Consult the linked official page for original screenshots. + +**Verification status:** static validated only; runtime/grader unverified. No Google Cloud lab credentials were used, and no grader result is claimed. diff --git a/labs/GSP1288/lab.sh b/labs/GSP1288/lab.sh new file mode 100644 index 0000000..4da12cb --- /dev/null +++ b/labs/GSP1288/lab.sh @@ -0,0 +1,115 @@ +#!/usr/bin/env bash +# Copyright (c) 2026 ePlus.DEV | Unlicense +# GSP1288 - Configuring Vector Search in Spanner +# Static validation only; requires an authorized, active Google Skills lab. +set -Eeuo pipefail + +if [[ -z "${NO_COLOR:-}" ]] && { [[ -t 1 ]] || [[ "${FORCE_COLOR:-0}" == "1" ]]; }; then + GREEN=$'\033[0;32m'; CYAN=$'\033[0;36m'; YELLOW=$'\033[0;33m'; RED=$'\033[0;31m'; RESET=$'\033[0m' +else + GREEN=''; CYAN=''; YELLOW=''; RED=''; RESET='' +fi +info() { printf '%s▶ %s%s\n' "$CYAN" "$*" "$RESET"; } +ok() { printf '%s✔ %s%s\n' "$GREEN" "$*" "$RESET"; } +warn() { printf '%s! %s%s\n' "$YELLOW" "$*" "$RESET" >&2; } +die() { printf '%s✖ %s%s\n' "$RED" "$*" "$RESET" >&2; exit 1; } +trap 'die "Stopped near line $LINENO. Inspect the preceding command."' ERR +printf '%s\n' "${CYAN}ePlus.DEV – Starting Execution${RESET}" +printf '%s\n' "Copyright (c) 2026 ePlus.DEV | Unlicense" +command -v gcloud >/dev/null || die 'Google Cloud CLI is required.' +command -v python3 >/dev/null || die 'Python 3 is required.' + +PROJECT_ID="${PROJECT_ID:-${GOOGLE_CLOUD_PROJECT:-$(gcloud config get-value project 2>/dev/null || true)}}" +[[ -n "$PROJECT_ID" && "$PROJECT_ID" != "(unset)" ]] || die 'No active project. Set PROJECT_ID.' +INSTANCE="${INSTANCE:-cymbal-retail}" +DATABASE="${DATABASE:-cymbal-bicycles}" +gcloud spanner instances describe "$INSTANCE" --project="$PROJECT_ID" --format='value(name)' >/dev/null || + die "Lab-provisioned Spanner instance $INSTANCE not found in $PROJECT_ID." +gcloud spanner databases describe "$DATABASE" --instance="$INSTANCE" --project="$PROJECT_ID" --format='value(name)' >/dev/null || + die "Lab-provisioned database $DATABASE not found." +CONFIG="$(gcloud spanner instances describe "$INSTANCE" --project="$PROJECT_ID" --format='value(config)' 2>/dev/null || true)" +SPANNER_REGION="$(printf '%s' "$CONFIG" | sed -nE 's/.*regional-([a-z]+-[a-z0-9]+[0-9]).*/\1/p')" +META_REGION="$(gcloud compute project-info describe --project="$PROJECT_ID" --format=json 2>/dev/null | python3 -c 'import json,sys; d=json.load(sys.stdin); print(next((i.get("value","") for i in d.get("commonInstanceMetadata",{}).get("items",[]) if i.get("key")=="google-compute-default-region"),""))' 2>/dev/null || true)" +REGION="${REGION:-${META_REGION:-${SPANNER_REGION:-$(gcloud config get-value compute/region 2>/dev/null || true)}}}" +[[ -n "$REGION" && "$REGION" != "(unset)" ]] || die 'Unable to determine region; set REGION from the lab details.' +META_ZONE="$(gcloud compute project-info describe --project="$PROJECT_ID" --format=json 2>/dev/null | python3 -c 'import json,sys; d=json.load(sys.stdin); print(next((i.get("value","") for i in d.get("commonInstanceMetadata",{}).get("items",[]) if i.get("key")=="google-compute-default-zone"),""))' 2>/dev/null || true)" +ZONE="${ZONE:-${META_ZONE:-$(gcloud config get-value compute/zone 2>/dev/null || true)}}" +if [[ "$ZONE" == "(unset)" ]]; then ZONE=''; fi +# Spanner is regional/multi-regional and does not consume ZONE; it is logged for diagnostics. +MODEL_ID="${MODEL_ID:-gemini-embedding-001}" +[[ "$MODEL_ID" =~ ^[a-zA-Z0-9._-]+$ ]] || die 'Invalid MODEL_ID.' +[[ "$REGION" =~ ^[a-z]+-[a-z0-9]+[0-9]$ ]] || die 'Invalid REGION.' +info "Project: $PROJECT_ID | Spanner: $INSTANCE/$DATABASE | Region: $REGION | Zone: ${ZONE:-n/a} | Model: $MODEL_ID" + +run_sql() { + gcloud spanner databases execute-sql "$DATABASE" --instance="$INSTANCE" --project="$PROJECT_ID" --sql="$1" +} +run_ddl() { + gcloud spanner databases ddl update "$DATABASE" --instance="$INSTANCE" --project="$PROJECT_ID" --ddl="$1" --quiet +} +retry() { + local attempt + for attempt in 1 2 3 4; do + if "$@"; then return 0; fi + if (( attempt == 4 )); then return 1; fi + warn "Attempt $attempt failed; retrying in 12 seconds..." + sleep 12 + done +} +ddl_contains() { + gcloud spanner databases ddl describe "$DATABASE" --instance="$INSTANCE" --project="$PROJECT_ID" --format=json | + grep -Eiq "$1" +} + +info 'Task 1/4 — Create the remote embeddings model.' +if ddl_contains 'CREATE MODEL EmbeddingsModel'; then + ok 'EmbeddingsModel already exists.' +else + retry run_ddl "CREATE MODEL EmbeddingsModel INPUT(content STRING(MAX),) OUTPUT(embeddings STRUCT, values ARRAY>,) REMOTE OPTIONS(endpoint = '//aiplatform.googleapis.com/projects/$PROJECT_ID/locations/$REGION/publishers/google/models/$MODEL_ID')" || + die 'Failed to create EmbeddingsModel.' +fi + +info 'Task 2/4 — Create the products table and load 10 original sample records.' +if ddl_contains 'CREATE TABLE products'; then + ok 'Products table already exists.' +else + retry run_ddl 'CREATE TABLE products (categoryId INT64 NOT NULL, productId INT64 NOT NULL, productName STRING(MAX) NOT NULL, productDescription STRING(MAX) NOT NULL, productDescriptionEmbedding ARRAY, createTime TIMESTAMP NOT NULL OPTIONS (allow_commit_timestamp = true), inventoryCount INT64 NOT NULL, priceInCents INT64) PRIMARY KEY(categoryId, productId)' || + die 'Failed to create products table.' +fi +# Independently written descriptions, not copied from the lab or codelab. +rows=( + '1|Cymbal Helios Helmet|Adjustable youth helmet with vents and a comfortable protective fit.|100|10999' + '2|Cymbal Sprout|Small balance bicycle designed for toddlers aged two to four.|10|13999' + '3|Cymbal Spark Jr.|First pedal bicycle for young riders learning confident handling.|34|13900' + '4|Cymbal Summit|Mountain bicycle with suspension for rugged off road trails.|0|79999' + '5|Cymbal Breeze|Electric commuter bicycle with assisted pedaling and battery power.|72|129999' + '6|Cymbal Trailblazer Backpack|Weather resistant backpack for cycling tools and supplies.|24|7999' + '7|Cymbal Phoenix Lights|Front and rear bicycle lights for visibility after sunset.|87|3999' + '8|Cymbal Windstar Pump|Compact portable bicycle tire pump for quick inflation.|36|24999' + '9|Cymbal Odyssey Multi-Tool|Pocket tool kit with screwdrivers and bicycle repair keys.|52|999' + '10|Cymbal Nomad Water Bottle|Reusable leak resistant bottle for hydration on rides.|42|1299' +) +for record in "${rows[@]}"; do + IFS='|' read -r product_id name description inventory price <<< "$record" + sql="INSERT INTO products (categoryId,productId,productName,productDescription,createTime,inventoryCount,priceInCents) SELECT 1,$product_id,'$name','$description',PENDING_COMMIT_TIMESTAMP(),$inventory,$price WHERE NOT EXISTS (SELECT 1 FROM products WHERE categoryId=1 AND productId=$product_id)" + retry run_sql "$sql" >/dev/null || die "Failed to load product $product_id." +done +ok '10 product keys inserted or already present.' + +info 'Task 3/4 — Grant Spanner service-agent role and generate embeddings.' +PROJECT_NUMBER="$(gcloud projects describe "$PROJECT_ID" --format='value(projectNumber)')" +[[ "$PROJECT_NUMBER" =~ ^[0-9]+$ ]] || die 'Could not resolve project number.' +SERVICE_AGENT="service-$PROJECT_NUMBER@gcp-sa-spanner.iam.gserviceaccount.com" +retry gcloud projects add-iam-policy-binding "$PROJECT_ID" \ + --member="serviceAccount:$SERVICE_AGENT" --role='roles/spanner.serviceAgent' --condition=None --quiet >/dev/null || + die 'Cannot grant the documented Spanner service-agent role.' +retry run_sql 'UPDATE products p1 SET productDescriptionEmbedding = (SELECT embeddings.values FROM ML.PREDICT(MODEL EmbeddingsModel, (SELECT productDescription AS content FROM products p2 WHERE p2.productId=p1.productId))) WHERE categoryId=1 AND productDescriptionEmbedding IS NULL' || + die 'Embedding update failed; verify model ID, IAM propagation and region.' + +info 'Task 4/4 — Run semantic similarity search.' +retry run_sql "SELECT productName,inventoryCount,COSINE_DISTANCE(productDescriptionEmbedding,(SELECT embeddings.values FROM ML.PREDICT(MODEL EmbeddingsModel,(SELECT 'starter bicycle for a three year old child' AS content)))) AS distance FROM products WHERE inventoryCount>0 AND productDescriptionEmbedding IS NOT NULL ORDER BY distance LIMIT 5" || + die 'Similarity query failed.' +info 'Read-back: check the number of products and non-null embeddings.' +run_sql 'SELECT COUNT(*) AS products,COUNTIF(productDescriptionEmbedding IS NOT NULL) AS embedded FROM products WHERE categoryId=1' +ok 'Finished SQL workflow. Check each Google Skills grader checkpoint manually.' +warn 'static validated only; runtime/grader unverified.'