Exercises

1. Px-sock-shop — Bug Detection

Setup

1. Go to the sock-shop demo application URL.
2. Browse the page, click around.

Trigger the Error

1. Go to the Catalogue tab.
2. Select at least two tags from the left Filter panel, for example geek and formal.
3. Click Apply.
4. Notice that no socks appear when two or more filters are selected.
5. Click Clear to clear filters between attempts.
6. You can repeat this any number of times.

Investigate with Pixie

1. Open the Pixie Live UI and log in.
2. Select namespace px-sock-shop
3. Select the px/cluster script and see what the view presents.
4. Select the px/namespace script. Enter the namespace we’re operating in.
5. Review what is displayed on the dashboard.
6. Change the time range to -15m.
7. Go into the details of the px-sock-shop/catalogue service and see what they contain.
   • At least one error should be reported. If not, repeat the steps from the “Trigger the Error” section.
8. Change the script to px/http_data
9. Open the script editor (press Ctrl+E) or click the editor icon in the UI.
10. Replace the line:

    df.node = df.ctx['node']
    

    with

    # Access the service name.
    df.service = df.ctx['service']
    
    # Filter to only catalogue service.
    df = df[df.service == 'px-sock-shop/catalogue']
    
    # Filter to errors greater or equal to 400.
    df = df[df.resp_status >= 400]
    

11. Execute the script. Errors should be visible. See what is causing the errors.

Database Investigation

1. Change the script to px/mysql_data
2. Add the following line at row 34:

    df = df[df.resp_status == 3]
   

3. Run the script and see what caused the error.

Explore what other Pixie scripts have to offer

2. Endpoint Deprecation

We want to decide whether we can deprecate one of our application endpoints. To do this, we must first check:

• whether it is being used
• who is using it

We’ll use the Pixie script editor for this:

1. Open the Scratch Pad.
2. Open the script editor.
3. In the PxL Script and Vis Spec tabs, paste the contents of the corresponding files from the endpoint-deprecation/service_endpoints_summary directory.
4. Run the script for the otel-demo-quoteservice service. Do you see the requests the service handles?
5. Let’s see exactly what the requests look like. Load the files from the service_requests directory.
6. Now we want to see who is making requests and for what. Load the files from the service_endpoint_requests directory.

3. Cost Estimation

Load the files from the cost-estimation directory into Scratch Pad. See how the values for estimation are calculated.

4. Redis Usage Discovery

Find which service uses Redis in the default namespace and for what purpose.

5. Why So Slow

In the otel namespace, find the service with the largest response time variance and see in the Flame Graph what it mainly spends execution time on.

Guided Exercises

Service Performance Monitoring

1. Service Graph

1. Open the Pixie Live View and select the px/cluster script.
   • View the graph of HTTP traffic between services, with latency, error, and throughput per service.
   • Hover over edges for stats; thicker lines indicate more traffic.
2. Scroll down to the Services table.
   • Latency, error, and throughput rates for all HTTP traffic.
3. Click the LATENCY column title to sort services by latency.
   • Click vertical quantile lines on the box plot to switch between P50, P90, and P99 latency values.

2. Service Performance

1. From the SERVICE column, click on a service (e.g., px-sock-shop/front-end) to open the px/service script.
   • Latency, error, and throughput over time for all HTTP requests.
   • Modify start_time to change the time window (e.g., -30m, -1h).
2. Scroll down to the Sample of Slow Requests table and expand the REQ_PATH column.

3. Endpoint Performance

1. Select pxbeta/service_endpoints from the script drop-down menu.
2. Set the service argument to the desired service (e.g., px-sock-shop/catalogue).
   • Latency, error, and throughput per logical endpoint (wildcards for URL parameters).
3. Click on an endpoint in the Endpoints table to see an overview and sample of slow requests.

Request Tracing

1. Full-Body HTTP Requests

1. Select the px/http_data_filtered script.
   • Shows recent HTTP requests filtered by service, pod, request path, and response status code.
2. Filter by status code: Set status_code to 500 and re-run.
3. Filter by service: Set svc to px-sock-shop/carts and re-run.
4. Click a table row to view the data in JSON format — scroll to resp_body for error details.

2. Service Errors

1. From the SVC column, click on the px-sock-shop/carts service name.
   • Opens the px/service script — shows error rate over time.
2. Scroll down to the Inbound Traffic by Requesting Service table.

3. Pod Errors

1. Click on the pod name to open the px/pod script.
   • Shows HTTP error rate and high-level resource metrics.

Network Monitoring

1. Network Traffic

1. Select the px/net_flow_graph script. Enter a namespace.
   • Pan, zoom, and rearrange nodes.
   • Grey hexagons = pods; blue circles = remote endpoints.
2. Filter by setting the to_entity_filter argument.

2. DNS Requests

1. Select the px/dns_flow_graph script.
2. Sort by LATENCY_AVG to find highest latency requests.

3. TCP Drops

1. Select bpftrace/tcp_drops and press RUN.
   • Hover over edges to see TCP drops between pod pairs.
   • Enable Hierarchy View for a different perspective.

Infrastructure Health

1. Resource Usage by Node

1. Select the px/nodes script.
   • CPU usage, memory consumption, and network traffic stats.
2. Click a node name to open px/node for detailed stats.
3. Change groupby to “pod” to group by pod.

2. Resource Usage by Pod

1. Click a pod name to open px/pod.
   • High-level HTTP metrics, resource usage, containers, processes.
2. Scroll down for the CPU flamegraph.
   • Dark blue = K8s metadata, light blue = user space app code, light green = kernel code.

Database Query Profiling

1. MySQL Stats

1. Select px/mysql_stats.
   • Latency, error, and throughput for all MySQL requests.
2. Filter by pod (e.g., px-sock-shop/catalogue).

2. Normalized SQL Queries

1. Select px/sql_queries.
   • Latency per normalized SQL query (constants replaced with ?).
2. Click a query to see latency per individual parameter.

3. Full Body Requests

1. Select px/mysql_data.
   • Full request and response bodies.
   • Sort by descending latency to find slow queries.