Sep22 sao workshop changes for o11y cloud

content/en/ninja-workshops/ai/3-splunk-agent-observability/4-instrument-the-application/3-generate-traffic.md

@@ -132,14 +132,14 @@ Then rebuild the Docker image:
 
 ```bash
 cd ~/workshop/healthcare-assistant
-docker build -f 2-app-with-instrumentation/Dockerfile -t localhost:9999/healthcare-assistant:app-with-instrumentation-v2 .
-docker push localhost:9999/healthcare-assistant:app-with-instrumentation-v2
+docker build -f 2-app-with-instrumentation/Dockerfile -t localhost:9999/healthcare-assistant:app-with-instrumentation-v3 .
+docker push localhost:9999/healthcare-assistant:app-with-instrumentation-v3
 ```
 
 Update the `~/workshop/healthcare-assistant/2-app-with-instrumentation/k8s.yaml` file to reference the local image instead: 
 
 ````
-image: localhost:9999/healthcare-assistant:app-with-instrumentation-v2
+image: localhost:9999/healthcare-assistant:app-with-instrumentation-v3
 ````
 
 And redeploy the application: 

content/en/ninja-workshops/ai/3-splunk-agent-observability/8-apply-guardrails/1-add-agent-control.md

@@ -33,19 +33,20 @@ kubectl create configmap splunk-agent-control-config \
   --from-literal=AGENT_CONTROL_AGENT_NAME="agent-control-example" \
   --from-literal=AGENT_CONTROL_API_KEY_HEADER="Splunk-AO-API-Key" \
   --from-literal=AGENT_CONTROL_RUNTIME_AUTH_MODE="jwt" \
-  --from-literal=AGENT_CONTROL_TARGET_TYPE="agent_stream"
+  --from-literal=AGENT_CONTROL_TARGET_TYPE="log_stream"
 ```
 
 Alternatively, if you're using Splunk Agent Observability within Splunk Observability Cloud for this workshop,
 please use the following command instead:
 
 ```bash
 kubectl create configmap splunk-agent-control-config \
-  --from-literal=AGENT_CONTROL_URL="https://app.$REALM.observability.splunkcloud.com/api/agent-control" \
+  --from-literal=AGENT_CONTROL_URL="https://app.$REALM.observability.splunkcloud.com/ao/agent-control" \
   --from-literal=AGENT_CONTROL_AGENT_NAME="agent-control-example" \
-  --from-literal=AGENT_CONTROL_API_KEY_HEADER="Splunk-AO-API-Key" \
+  --from-literal=AGENT_CONTROL_API_KEY_HEADER="X-SF-Token" \
+  --from-literal=AGENT_CONTROL_RUNTIME_TOKEN_HEADER="X-Agent-Control-Runtime-Token" \
   --from-literal=AGENT_CONTROL_RUNTIME_AUTH_MODE="jwt" \
-  --from-literal=AGENT_CONTROL_TARGET_TYPE="agent_stream"
+  --from-literal=AGENT_CONTROL_TARGET_TYPE="log_stream"
 ```
 
 {{< /step >}}

content/en/ninja-workshops/ai/3-splunk-agent-observability/8-apply-guardrails/3-test-controls.md

@@ -55,36 +55,8 @@ section of Splunk Show event:
 
 {{% notice title="Troubleshooting" style="tip" icon="exclamation-triangle" %}}
 
-To see what Agent Control is doing, enable console logging in `~/workshop/healthcare-assistant/4-app-with-controls/agent.py`:
-
-```python
-from splunk_ao.utils.log_config import enable_console_logging
-
-enable_console_logging()
-```
-
-Then rebuild the Docker image:
-
-```bash
-cd ~/workshop/healthcare-assistant
-docker build -f 4-app-with-controls/Dockerfile -t localhost:9999/healthcare-assistant:app-with-controls-v2 .
-docker push localhost:9999/healthcare-assistant:app-with-controls-v2
-```
-
-Update the `~/workshop/healthcare-assistant/4-app-with-controls/k8s.yaml` file to reference the local image instead:
-
-````
-image: localhost:9999/healthcare-assistant:app-with-controls-v2
-````
-
-And redeploy the application:
-
-```bash
-cd ~/workshop/healthcare-assistant/4-app-with-controls
-kubectl apply -f k8s.yaml
-```
-
-Use the following command to view the application logs:
+The app already logs to stdout, so you can see what Agent Control is doing directly with
+`kubectl logs` — no code change or image rebuild required. View the application logs:
 
 {{< tabs id="healthcare-app-logs" >}}
 {{% tab title="Script" %}}
@@ -115,6 +87,27 @@ INFO - splunk_ao.logger - Session started with ID: ec03c538-cf9e-4bed-b97e-4b3c2
 Watch the terminal for `Agent Control initialized` on startup and `BLOCKED` / `STEERED`
 messages when a control fires.
 
+If you need to dig deeper — for example, to see exactly which environment variables the app
+reads and the full HTTP request/response for every Agent Control call (agent registration,
+runtime token exchange, and control evaluation) — set `AGENT_CONTROL_DEBUG=true` in the
+`splunk-agent-control-config` ConfigMap and restart the app:
+
+```bash
+kubectl patch configmap splunk-agent-control-config --type merge -p '{"data":{"AGENT_CONTROL_DEBUG":"true"}}'
+kubectl rollout restart deployment/healthcare-assistant
+```
+
+Credential headers (such as `X-SF-Token`) are masked in these logs by default. If you need to
+verify the exact token value, add `AGENT_CONTROL_DEBUG_UNMASK=true` the same way:
+
+```bash
+kubectl patch configmap splunk-agent-control-config --type merge -p '{"data":{"AGENT_CONTROL_DEBUG_UNMASK":"true"}}'
+kubectl rollout restart deployment/healthcare-assistant
+```
+
+Leave both unset for normal use — the app only logs a concise `Agent Control initialized`
+line and any configuration warnings.
+
 {{% /notice %}}
 
 {{< /step >}}

workshop/demos/healthcare-assistant/k8s-demo-o11y-eu-splunk-show.yaml

@@ -16,7 +16,7 @@ spec:
     spec:
       containers:
         - name: healthcare-assistant-eu
-          image: ghcr.io/splunk/healthcare-assistant:app-with-controls-v2
+          image: ghcr.io/splunk/healthcare-assistant:app-with-controls-v3
           imagePullPolicy: Always
           ports:
             - containerPort: 8501

workshop/demos/healthcare-assistant/k8s-demo-o11y-us-splunk-show.yaml

@@ -16,7 +16,7 @@ spec:
     spec:
       containers:
         - name: healthcare-assistant-us
-          image: ghcr.io/splunk/healthcare-assistant:app-with-controls-v2
+          image: ghcr.io/splunk/healthcare-assistant:app-with-controls-v3
           imagePullPolicy: Always
           ports:
             - containerPort: 8501

workshop/demos/healthcare-assistant/k8s-demo-standalone.yaml

@@ -16,7 +16,7 @@ spec:
     spec:
       containers:
         - name: healthcare-assistant
-          image: ghcr.io/splunk/healthcare-assistant:app-with-controls-v2
+          image: ghcr.io/splunk/healthcare-assistant:app-with-controls-v3
           imagePullPolicy: Always
           ports:
             - containerPort: 8501

workshop/healthcare-assistant/.dockerignore

@@ -1,8 +1,8 @@
 **/__pycache__
 **/*.pyc
 **/*.pyo
-1-base-app/.streamlit/secrets.toml
-1-base-app/k8s.yaml
-1-base-app/postgres.yaml
-1-base-app/setup-job.yaml
-1-base-app/start_vectordb.sh
+**/.streamlit/secrets.toml
+**/k8s*.yaml
+**/postgres.yaml
+**/setup-job.yaml
+**/start_vectordb.sh

workshop/healthcare-assistant/1-base-app/agent.py

@@ -2,8 +2,8 @@
 import asyncio
 import inspect
 import json
+import threading
 import uuid
-from concurrent.futures import ThreadPoolExecutor
 from typing import Annotated, List, Dict, Optional, TypedDict
 
 from langchain_core.messages import AIMessage, BaseMessage, HumanMessage, SystemMessage
@@ -23,15 +23,40 @@ class State(TypedDict):
     messages: Annotated[list, add_messages]
 
 
+_bg_loop: Optional[asyncio.AbstractEventLoop] = None
+_bg_loop_lock = threading.Lock()
+
+
+def _get_background_loop() -> asyncio.AbstractEventLoop:
+    """Return a process-wide event loop running on a dedicated daemon thread.
+
+    Streamlit calls process_query() synchronously on every interaction. Using
+    asyncio.run() per call creates and then *closes* a new loop each time, but
+    modern langchain-openai/openai reuse a cached async HTTP client across calls,
+    so its connection pool outlives the loop that created it. When a later request
+    reaps a keep-alive connection bound to that closed loop, httpx raises
+    "RuntimeError: Event loop is closed". Running every coroutine on one stable,
+    long-lived loop keeps the shared client and its pool on a single loop.
+    """
+    global _bg_loop
+    if _bg_loop is not None and not _bg_loop.is_closed():
+        return _bg_loop
+    with _bg_loop_lock:
+        if _bg_loop is None or _bg_loop.is_closed():
+            loop = asyncio.new_event_loop()
+            threading.Thread(
+                target=loop.run_forever,
+                name="agent-async-loop",
+                daemon=True,
+            ).start()
+            _bg_loop = loop
+    return _bg_loop
+
+
 def _run_async(coro):
-    """Run an async coroutine from sync code (e.g. Streamlit)."""
-    try:
-        asyncio.get_running_loop()
-    except RuntimeError:
-        return asyncio.run(coro)
-
-    with ThreadPoolExecutor(max_workers=1) as executor:
-        return executor.submit(asyncio.run, coro).result()
+    """Run an async coroutine from sync code (e.g. Streamlit) on the shared loop."""
+    loop = _get_background_loop()
+    return asyncio.run_coroutine_threadsafe(coro, loop).result()
 
 
 class HealthcareAgent:

workshop/healthcare-assistant/2-app-with-instrumentation/agent.py

@@ -2,8 +2,8 @@
 import asyncio
 import inspect
 import json
+import threading
 import uuid
-from concurrent.futures import ThreadPoolExecutor
 from typing import Annotated, List, Dict, Optional, TypedDict
 
 from langchain_core.messages import AIMessage, BaseMessage, HumanMessage, SystemMessage
@@ -31,15 +31,40 @@ class State(TypedDict):
     messages: Annotated[list, add_messages]
 
 
+_bg_loop: Optional[asyncio.AbstractEventLoop] = None
+_bg_loop_lock = threading.Lock()
+
+
+def _get_background_loop() -> asyncio.AbstractEventLoop:
+    """Return a process-wide event loop running on a dedicated daemon thread.
+
+    Streamlit calls process_query() synchronously on every interaction. Using
+    asyncio.run() per call creates and then *closes* a new loop each time, but
+    modern langchain-openai/openai reuse a cached async HTTP client across calls,
+    so its connection pool outlives the loop that created it. When a later request
+    reaps a keep-alive connection bound to that closed loop, httpx raises
+    "RuntimeError: Event loop is closed". Running every coroutine on one stable,
+    long-lived loop keeps the shared client and its pool on a single loop.
+    """
+    global _bg_loop
+    if _bg_loop is not None and not _bg_loop.is_closed():
+        return _bg_loop
+    with _bg_loop_lock:
+        if _bg_loop is None or _bg_loop.is_closed():
+            loop = asyncio.new_event_loop()
+            threading.Thread(
+                target=loop.run_forever,
+                name="agent-async-loop",
+                daemon=True,
+            ).start()
+            _bg_loop = loop
+    return _bg_loop
+
+
 def _run_async(coro):
-    """Run an async coroutine from sync code (e.g. Streamlit)."""
-    try:
-        asyncio.get_running_loop()
-    except RuntimeError:
-        return asyncio.run(coro)
-
-    with ThreadPoolExecutor(max_workers=1) as executor:
-        return executor.submit(asyncio.run, coro).result()
+    """Run an async coroutine from sync code (e.g. Streamlit) on the shared loop."""
+    loop = _get_background_loop()
+    return asyncio.run_coroutine_threadsafe(coro, loop).result()
 
 
 class HealthcareAgent:

workshop/healthcare-assistant/2-app-with-instrumentation/k8s-o11y.yaml

@@ -16,7 +16,7 @@ spec:
     spec:
       containers:
         - name: healthcare-assistant
-          image: ghcr.io/splunk/healthcare-assistant:app-with-instrumentation-v2
+          image: ghcr.io/splunk/healthcare-assistant:app-with-instrumentation-v3
           imagePullPolicy: Always
           ports:
             - containerPort: 8501

workshop/healthcare-assistant/2-app-with-instrumentation/k8s.yaml

@@ -16,7 +16,7 @@ spec:
     spec:
       containers:
         - name: healthcare-assistant
-          image: ghcr.io/splunk/healthcare-assistant:app-with-instrumentation-v2
+          image: ghcr.io/splunk/healthcare-assistant:app-with-instrumentation-v3
           imagePullPolicy: Always
           ports:
             - containerPort: 8501

workshop/healthcare-assistant/3-app-with-experiments/agent.py

@@ -2,8 +2,8 @@
 import asyncio
 import inspect
 import json
+import threading
 import uuid
-from concurrent.futures import ThreadPoolExecutor
 from typing import Annotated, List, Dict, Optional, TypedDict
 
 from langchain_core.messages import AIMessage, BaseMessage, HumanMessage, SystemMessage
@@ -30,15 +30,40 @@ class State(TypedDict):
     messages: Annotated[list, add_messages]
 
 
+_bg_loop: Optional[asyncio.AbstractEventLoop] = None
+_bg_loop_lock = threading.Lock()
+
+
+def _get_background_loop() -> asyncio.AbstractEventLoop:
+    """Return a process-wide event loop running on a dedicated daemon thread.
+
+    Streamlit calls process_query() synchronously on every interaction. Using
+    asyncio.run() per call creates and then *closes* a new loop each time, but
+    modern langchain-openai/openai reuse a cached async HTTP client across calls,
+    so its connection pool outlives the loop that created it. When a later request
+    reaps a keep-alive connection bound to that closed loop, httpx raises
+    "RuntimeError: Event loop is closed". Running every coroutine on one stable,
+    long-lived loop keeps the shared client and its pool on a single loop.
+    """
+    global _bg_loop
+    if _bg_loop is not None and not _bg_loop.is_closed():
+        return _bg_loop
+    with _bg_loop_lock:
+        if _bg_loop is None or _bg_loop.is_closed():
+            loop = asyncio.new_event_loop()
+            threading.Thread(
+                target=loop.run_forever,
+                name="agent-async-loop",
+                daemon=True,
+            ).start()
+            _bg_loop = loop
+    return _bg_loop
+
+
 def _run_async(coro):
-    """Run an async coroutine from sync code (e.g. Streamlit)."""
-    try:
-        asyncio.get_running_loop()
-    except RuntimeError:
-        return asyncio.run(coro)
-
-    with ThreadPoolExecutor(max_workers=1) as executor:
-        return executor.submit(asyncio.run, coro).result()
+    """Run an async coroutine from sync code (e.g. Streamlit) on the shared loop."""
+    loop = _get_background_loop()
+    return asyncio.run_coroutine_threadsafe(coro, loop).result()
 
 
 class HealthcareAgent:

workshop/healthcare-assistant/4-app-with-controls/README.md

@@ -9,11 +9,14 @@ Edit the `.streamlit/secrets.toml` file and add the following environment variab
 ```yaml
 # Agent Controls
 # -----------------------------------------------------------------------------
-agent_control_url = "https://console.multitenant.galileocloud.io/api/agent-control" 
+# Standalone:
+agent_control_url = "https://console.multitenant.sao.splunkcloud.com/api/agent-control"
+# Observability Cloud:
+# agent_control_url = "https://app.<realm>.observability.splunkcloud.com/ao/agent-control"
 agent_control_agent_name = "agent-control-example"
 agent_control_runtime_auth_mode="jwt"
-agent_control_api_key_header = "Splunk-AO-Key"
-agent_control_target_type="log_stream"
+agent_control_api_key_header = "Splunk-AO-API-Key"
+agent_control_target_type="agent_stream"
 ```
 
 ## Read New Config