What is SLI? Meaning, Architecture, Examples, Use Cases, and How to Measure It (2026 Guide)

---

Quick Definition (30–60 words)

An SLI (Service Level Indicator) is a quantitative measure of some aspect of system behavior that reflects user experience. Analogy: SLI is the thermometer for system health. Formal: SLI = measured probability or rate that a specific user-facing condition holds over a defined window.

---

What is SLI?

SLI stands for Service Level Indicator. It is a precise metric representing user experience — for example request success rate, latency under threshold, or data freshness. It is not an SLA (contract) or an SLO (target), though it is the primary input to both. SLIs must be measurable, objective, and tied to user impact.

Key properties and constraints:

• User-focused: maps to customer experience.
• Quantitative: defined as a ratio, count, or distribution.
• Time-windowed: always interpreted over a measurement window.
• Observable: requires instrumentation and reliable telemetry.
• Immutable definition: alteration requires versioning and communication.
• Privacy-aware: must avoid exposing sensitive data.
• Performance-cost trade-off: measurement can add overhead.

Where it fits in modern cloud/SRE workflows:

• Observability foundation: feeds dashboards, alerts, and postmortems.
• SRE practice: basis for SLOs and error budgets that drive release policies.
• Incident response: triggers pagers and remediation playbooks.
• CI/CD gating: informs progressive delivery and automated rollbacks.
• Cost and reliability trade-offs: guides optimization work.

Diagram description (text-only):

• Users send requests to edge -> load balancer routes to services -> services read/write databases and caches -> telemetry collectors gather traces, logs, metrics -> SLI computation engine aggregates metrics over windows -> SLO evaluator compares SLIs to targets -> alerting and automation respond when thresholds exceeded.

SLI in one sentence

An SLI is a measurable signal about how well a system is doing at delivering a particular aspect of user experience.

SLI vs related terms (TABLE REQUIRED)

ID	Term	How it differs from SLI	Common confusion
T1	SLO	SLO is a target on an SLI	People call SLO an SLI
T2	SLA	SLA is a contractual promise often with penalties	SLA includes legal terms and remedies
T3	Metric	Metric is any measurement not tied to user experience	Metrics can be internal only
T4	KPI	KPI is a business metric often higher level	KPIs may not map to system behavior
T5	Error budget	Consumable allowance derived from SLO minus SLI	Sometimes called SLI budget
T6	Alert	Alert is a notification triggered by SLI or metric	Alerts are not SLIs
T7	Telemetry	Raw data source used to compute SLIs	Telemetry is not the SLI itself
T8	Observability	Capability to derive insights from telemetry	Observability is broader than SLIs
T9	Latency P99	Specific latency percentile metric	P99 is a metric used as an SLI
T10	Availability	Often used as an SLI but is an abstract concept	Availability needs precise SLI definition

Row Details (only if any cell says “See details below”)

• None

---

Why does SLI matter?

Business impact:

• Revenue: A degraded SLI for checkout latency or error rate causes abandoned carts and lost revenue.
• Trust: Repeated SLI breaches erode user confidence and retention.
• Risk management: SLIs shape contractual and legal exposure via SLAs and inform mitigation spend.

Engineering impact:

• Incident reduction: Well-defined SLIs help detect regressions earlier and reduce mean time to repair.
• Velocity: Error budgets derived from SLIs enable safe risk-taking for feature delivery.
• Prioritization: SLIs prioritize work on what impacts users most.

SRE framing:

• SLIs feed SLOs which set error budgets.
• Error budgets control release policies and rate of change.
• SLIs reduce toil by automating detection and routing of incidents.
• On-call teams use SLIs to decide paging severity and escalation.

What breaks in production — realistic examples:

1. API authentication service returns 5xx under load causing SLI for success rate to drop.
2. Cache invalidation bug leads to stale responses and freshness SLI breach.
3. Database replica lag causes request latency spikes and P95 latency SLI violation.
4. Edge load balancer misconfiguration drops connections producing availability SLI breach.
5. Deployment mis-specified resource limits causes CPU throttling and throughput SLI failure.

---

Where is SLI used? (TABLE REQUIRED)

ID	Layer/Area	How SLI appears	Typical telemetry	Common tools
L1	Edge / CDN	Cache hit ratio and time to first byte	HTTP logs, edge metrics	CDN-native metrics
L2	Network	Connection error rate and RTT	TCP metrics, flow logs	Cloud VPC logs
L3	Service / API	Success rate and latency percentiles	Traces, request metrics	APM and metrics store
L4	Database / Storage	Query success and replication lag	DB metrics, slow logs	DB monitors
L5	Application UX	Page load goodput and error rate	Browser RUM, metrics	RUM platforms
L6	Background jobs	Task success and backlog	Job queues, metrics	Queue monitors
L7	Kubernetes	Pod readiness and restart rate	K8s metrics, events	K8s observability tools
L8	Serverless	Invocation errors and cold start latency	Function logs, metrics	Serverless metrics
L9	CI/CD	Deployment success and lead time	Pipeline logs, metrics	CI systems
L10	Security	Auth success and anomaly rate	Audit logs, alerts	SIEM and logging

Row Details (only if needed)

• None

---

When should you use SLI?

When it’s necessary:

• You have production users and need objective user experience measurement.
• You operate services with SLAs or internal SLO commitments.
• You need to automate release gating, error budget consumption, or incident escalation.

When it’s optional:

• Very early prototypes or internal-only tooling with limited users.
• Short-lived experiments where fast iteration outweighs measurement cost.

When NOT to use / overuse it:

• Do not create SLIs for every metric; that causes noise.
• Avoid SLIs for low-impact internal telemetry like internal queue length unless it maps to user harm.
• Do not treat SLIs as the only source of truth; use alongside qualitative feedback.

Decision checklist:

• If user-facing and measurable and impacts revenue or UX -> define SLI and SLO.
• If internal-only and no user impact -> prefer metrics and alerts.
• If short-lived experiment and high churn -> defer SLI until stabilization.

Maturity ladder:

• Beginner: Define 1–3 SLIs for core user journeys and set conservative SLOs.
• Intermediate: Add per-service SLIs, error budgets, and automated alerts.
• Advanced: Multi-dimensional SLIs, adaptive SLOs, automation for progressive delivery, and ML-assisted anomaly detection.

---

How does SLI work?

Step-by-step components and workflow:

1. Instrumentation: Add measurement points in code/edge to emit telemetry aligned to SLI definitions.
2. Collection: Telemetry streams flow to collectors (metrics backend, traces, logs).
3. Aggregation: Compute numerator/denominator or aggregate distribution in the SLI engine.
4. Evaluation: Compare SLI over window to SLO target and compute error budget consumption.
5. Alerting & Automation: Trigger alerts, runbooks, or automated rollbacks when thresholds crossed.
6. Reporting: Dashboards and executive reports show SLA/SLO status and trends.
7. Feedback: Postmortems and improvements feed back into SLI refinement.

Data flow and lifecycle:

• Emit -> Collect -> Store -> Compute -> Evaluate -> Act -> Review -> Adjust.

Edge cases and failure modes:

• Telemetry gaps can lead to blind spots.
• Bucket boundaries (latency thresholds) create discontinuities.
• Cardinality explosion in labels undermines aggregation.
• Time-window misalignment results in incorrect evaluation.

Typical architecture patterns for SLI

• Sidecar metrics pattern: instrumentation sidecar exports metrics locally for collection; use when you want isolation and standards enforcement.
• Agent-based collection pattern: host agents scrape and forward metrics; best for legacy workloads and node-level metrics.
• Observability pipeline pattern: streaming pipeline (collector -> processor -> store) with enrichment and reduction; use at scale to reduce storage and compute.
• In-band tracing-derived SLIs: compute SLIs from traces by evaluating spans for errors/latency; best for complex distributed transactions.
• Hybrid edge-to-backend SLI: combine RUM at edge with backend traces to correlate user impact across layers.

Failure modes & mitigation (TABLE REQUIRED)

ID	Failure mode	Symptom	Likely cause	Mitigation	Observability signal
F1	Missing telemetry	Sudden SLI gaps	Collector crash or config change	Instrument health checks and fallback	Missing metrics series
F2	High-cardinality labels	Aggregation slow or OOM	Unbounded tag values in code	Enforce label whitelist	Elevated ingest latency
F3	Clock skew	Wrong windowed SLI	NTP/drift across hosts	Use monotonic timestamps and sync	Timestamps misaligned
F4	Sampling bias	Underreported errors	Aggressive sampling of traces	Use stratified sampling for errors	Trace sample ratio changes
F5	Metric drift	Baseline shifts slowly	Code change or dependency update	Canary and baseline monitoring	Slow trend in baseline
F6	Definition mismatch	Different SLIs reported	Multiple teams use different SLI defs	Centralize SLI catalog	Conflicting dashboards
F7	Storage loss	Historical SLI lost	Retention misconfig or outage	Multi-region storage and backups	Gaps in historical datapoints

Row Details (only if needed)

• None

---

Key Concepts, Keywords & Terminology for SLI

Glossary (40+ terms)

• SLI — A measured indicator of service behavior related to user experience — Basis for SLOs and alerts — Pitfall: vague definitions.
• SLO — Target value for an SLI over a window — Drives error budget — Pitfall: unrealistic targets.
• SLA — Contractual agreement with penalties — Business-level guarantee — Pitfall: legal obligations ignored.
• Error budget — Allowed tolerance for SLO breach — Enables risk-based release — Pitfall: mismanagement halts delivery.
• Availability — Percent of successful user requests — Reflects uptime — Pitfall: undefined success criteria.
• Latency — Time taken to respond to requests — Impacts perceived performance — Pitfall: relying on averages.
• Throughput — Number of requests per unit time — Measures capacity — Pitfall: conflated with performance.
• Success rate — Fraction of requests meeting success criteria — Clear user-focused SLI — Pitfall: ignores partial failures.
• Freshness — Age of data shown to user — Important for caches and feeds — Pitfall: not accounting for eventual consistency.
• Durability — Guarantee that data persists — Storage-related SLI — Pitfall: conflating availability with durability.
• Partition tolerance — System behavior during network partitions — Affects SLIs — Pitfall: not testing partitions.
• Observability — Ability to infer system state from telemetry — Enables SLI creation — Pitfall: collecting logs without structure.
• Telemetry — Logs, metrics, traces and events — Raw inputs for SLIs — Pitfall: inconsistent schemas.
• Metric — Quantitative measurement of system property — General concept — Pitfall: many metrics are not SLIs.
• Trace — Distributed trace of a transaction — Helps diagnose SLI breaches — Pitfall: sampling hides failures.
• Log — Unstructured event data — Useful for root cause analysis — Pitfall: noisy and costly.
• Monitoring — Process of tracking metrics and alerts — Operational function — Pitfall: reactive only.
• Alerting — Automated notifications on threshold breaches — Incident trigger — Pitfall: alert fatigue.
• Incident Response — Steps to handle outages — Uses SLIs for severity — Pitfall: no documented runbooks.
• Runbook — Step-by-step operational guide — Reduces toil — Pitfall: unmaintained runbooks.
• Playbook — Higher-level procedural guide — For escalation and coordination — Pitfall: ambiguous roles.
• Canary — Gradual rollout to subset of users — Controlled risk — Pitfall: canary not representative.
• Rollback — Reverting to prior version — Safety mechanism — Pitfall: data migrations block rollback.
• Chaos testing — Deliberate failure injection — Validates resilience — Pitfall: no guardrails.
• Burn rate — Rate of error budget consumption — Informs mitigations — Pitfall: miscalculated windows.
• Service Level Management — Organizational practice of SLO governance — Operationalizes SLIs — Pitfall: lack of executive buy-in.
• Cardinality — Number of unique label values — Affects metric cost — Pitfall: high-cardinality explosions.
• Retention — How long telemetry is stored — Balances cost and analysis — Pitfall: insufficient history for trend analysis.
• Aggregation window — Time window for computing SLI — Affects smoothing and detection — Pitfall: misaligned windows.
• Percentile — Statistical measure for latency like P95 — Useful SLI candidate — Pitfall: unstable with small sample sizes.
• Root cause analysis — Finding underlying cause of outages — Uses SLIs for evidence — Pitfall: superficial fixes.
• Blameless postmortem — Culture for learning from incidents — Encourages SLI improvements — Pitfall: skipped actions.
• Service ownership — Team accountable for SLI health — Ensures care — Pitfall: ambiguous ownership.
• Automation — Scripts or systems that act on SLI breaches — Reduces toil — Pitfall: poor test coverage.
• RUM — Real user monitoring for front-end experience — User-centric SLI source — Pitfall: sampling or privacy issues.
• Synthetic testing — Automated simulated user tests — Supplements SLIs — Pitfall: not representative of real traffic.
• SLI engine — Component that computes and evaluates SLIs — Central point for policy — Pitfall: single point of misconfiguration.
• Adaptive SLOs — SLOs that adjust based on context or seasonality — Advanced reliability technique — Pitfall: complexity and fairness.

---

How to Measure SLI (Metrics, SLIs, SLOs) (TABLE REQUIRED)

ID	Metric/SLI	What it tells you	How to measure	Starting target	Gotchas
M1	Request success rate	Fraction of successful user requests	Successful responses / total over window	99.9% for core APIs	Define success precisely
M2	P95 latency	High-percentile user latency	95th percentile over window	200–500ms app endpoints	Need stable sample size
M3	Error rate by endpoint	Localize failures	Error count / total per endpoint	0.1% for critical endpoints	High-cardinality explosion
M4	Time to first byte (TTFB)	Edge responsiveness	Median time from request to first byte	100ms for CDN edge	Affected by network variance
M5	Cache hit ratio	How often cache serves fresh data	Hits / (hits+misses)	95% for read-heavy caches	Stale data risk
M6	Data freshness	Age of data presented to user	Max or median age metric	Application dependent	Eventual consistency edge cases
M7	Job success rate	Background task reliability	Successes / total tasks	99% for critical jobs	Retries may mask failures
M8	Replication lag	Data staleness across replicas	Seconds behind leader	<1s for low-latency apps	Sporadic spikes matter
M9	Deployment success	Fraction of successful deploys	Successful deploys / total	99% automated deploys	Migrations complicate rollback
M10	Cold start time	Serverless init latency	Median init time per invocation	<100ms for hot paths	Cold starts vary by language
M11	Resource saturation	CPU IO saturation affecting UX	Percentage utilization vs capacity	Keep headroom 20–40%	Autoscaling latency
M12	User-visible errors	Errors surfaced to users	Count of UI errors / sessions	As low as feasible	RUM sampling issues

Row Details (only if needed)

• None

Best tools to measure SLI

Use the following tool sections to describe recommended tooling.

Tool — Prometheus / OpenTelemetry metrics stack

• What it measures for SLI: Time-series metrics, counters, histograms for latency and success rates.
• Best-fit environment: Kubernetes, microservices, cloud-native.
• Setup outline:
• Instrument code with OpenTelemetry or client libraries.
• Expose metrics endpoint per service.
• Use Prometheus operator in Kubernetes or hosted ingestion.
• Configure recording rules for SLI numerators and denominators.
• Store long-term metrics in remote write backend.
• Strengths:
• Open standards and ecosystem.
• Good for custom metrics and scraping.
• Limitations:
• High-cardinality cost; retention needs planning.
• Single-node Prometheus scaling requires remote storage.

Tool — Tracing platform (OpenTelemetry collector + backend)

• What it measures for SLI: Transaction latency, error propagation, distributed context.
• Best-fit environment: Distributed microservices and multi-hop requests.
• Setup outline:
• Instrument critical transactions with spans.
• Configure sampling to preserve error traces.
• Use trace processing to derive SLI events.
• Correlate traces to metrics/join on trace IDs.
• Strengths:
• Root cause across services.
• Rich context for debugging.
• Limitations:
• Sampling trade-offs and high ingestion cost.

Tool — RUM / Browser monitoring

• What it measures for SLI: Page load, first input delay, real user errors.
• Best-fit environment: Web frontends and SPAs.
• Setup outline:
• Add lightweight RUM SDK to frontend.
• Capture key vitals and error events.
• Tag telemetry with user region and cohort.
• Strengths:
• Direct user experience measurement.
• Captures client-side issues invisible to backend.
• Limitations:
• Privacy constraints and sampling biases.

Tool — Synthetic monitoring

• What it measures for SLI: Endpoint availability and latency from representative locations.
• Best-fit environment: External availability checks and SLA verification.
• Setup outline:
• Define user-critical transactions as scripts.
• Schedule global probing from multiple regions.
• Compare synthetic results to real-user SLIs.
• Strengths:
• Predictable coverage and early detection.
• Limitations:
• Can miss real-user variations and workloads.

Tool — Cloud provider metrics & logs

• What it measures for SLI: Infrastructure-level metrics like network errors, load balancer health.
• Best-fit environment: IaaS/PaaS services.
• Setup outline:
• Enable platform logging and metrics.
• Forward critical signals to SLI evaluation pipeline.
• Correlate platform metrics with application SLIs.
• Strengths:
• Cloud-native integration and managed telemetry.
• Limitations:
• Vendor-specific formats and retention limits.

Recommended dashboards & alerts for SLI

Executive dashboard:

• Panels:
• Service-level SLI summary across products and target status.
• Error budget burn rate and projected exhaustion date.
• Top degraded SLOs with business impact tag.
• Trend of SLIs over last 30/90 days and anomalies.
• Why: High-level visibility for stakeholders to prioritize investment.

On-call dashboard:

• Panels:
• Real-time SLI status with per-region breakdown.
• Only the SLOs that can trigger paging.
• Recent alerts and active incidents.
• Top traces and logs correlation for failing endpoints.
• Why: Rapid triage for responders.

Debug dashboard:

• Panels:
• Detailed per-endpoint latency distributions and success rate.
• Service map with downstream SLA dependencies.
• Pod/container resource metrics and events.
• Recent deployments and correlating error spikes.
• Why: Deep-dive for engineers to troubleshoot.

Alerting guidance:

• What should page vs ticket:
• Page: SLI breach with high burn rate or impacting payment/checkout flows.
• Ticket: Non-urgent SLI degradation within error budget and low business impact.
• Burn-rate guidance:
• Thresholds: 1x normal burn for info, 3x triggers urgent review, 5x triggers immediate mitigation and halting risky releases.
• Noise reduction tactics:
• Deduplicate grouped alerts by root cause signature.
• Group alerts by service and region.
• Suppress low-severity alerts during planned maintenance windows.

---

Implementation Guide (Step-by-step)

1) Prerequisites: – Clear ownership per service. – Instrumentation standards and libraries selected. – Centralized telemetry pipeline and storage plan. – SLO policy and governance defined.

2) Instrumentation plan: – Identify user journeys and map to endpoints. – Define precise SLI numerator and denominator. – Add metrics/traces/logs with consistent labels. – Enforce low-cardinality labels and schema.

3) Data collection: – Deploy collectors and exporters. – Configure reliable transport and buffering. – Implement backpressure and sampling strategies. – Monitor collector health and throughput.

4) SLO design: – Choose window length (30d common) and rolling vs calendar. – Set starting targets conservatively, then iterate. – Define burn-rate actions and escalation thresholds.

5) Dashboards: – Build executive, on-call, debug dashboards. – Use recording rules or precomputed aggregations for performance. – Add annotations for deployments and incidents.

6) Alerts & routing: – Map SLI-triggered alerts to on-call rotations. – Implement paging escalation policies and notification channels. – Add automated mitigations where safe.

7) Runbooks & automation: – Create runbooks tied to specific SLI breaches. – Automate repetitive remediation like autoscaling or traffic shifting. – Ensure runbooks are small, testable, and versioned.

8) Validation (load/chaos/game days): – Perform load tests and verify SLI behavior under stress. – Run chaos experiments to validate resilience and runbook efficacy. – Execute game days to exercise human response.

9) Continuous improvement: – Postmortem all incidents and feed improvements into SLI definitions. – Track action items and verify closures. – Periodically review SLO targets against business changes.

Checklists:

Pre-production checklist:

• Ownership assigned and SLI definitions documented.
• Instrumentation deployed to staging with simulated traffic.
• Dashboard panels render expected numbers.
• Alerting rules exercised in staging.

Production readiness checklist:

• Telemetry retention and storage validated.
• Error budget policies configured.
• Runbooks and playbooks accessible from incident tooling.
• On-call trained and rotation confirmed.

Incident checklist specific to SLI:

• Confirm SLI computation correctness.
• Identify affected user cohort and rollback window.
• Execute immediate mitigation and document steps.
• Post-incident review and action assignment.

---

Use Cases of SLI

Provide common service-level scenarios.

1) Public API availability – Context: External API consumed by partners. – Problem: Downtime interrupts partner workflows. – Why SLI helps: Objective availability measurement for compliance. – What to measure: Successful responses per second and P99 latency. – Typical tools: API gateway metrics and external synthetic checks.

2) Checkout performance for e-commerce – Context: Checkout determines revenue. – Problem: Slow checkout increases abandonment. – Why SLI helps: Tie latency to conversion and revenue. – What to measure: P95 latency for checkout API and payment success rate. – Typical tools: APM, RUM, payment gateway metrics.

3) Streaming data freshness – Context: Real-time feed for trading or recommendations. – Problem: Stale data degrades decision quality. – Why SLI helps: Measure freshness to guarantee timeliness. – What to measure: Median and max data age. – Typical tools: Streaming metrics and database replication monitors.

4) Authentication reliability – Context: Single sign-on service for product suite. – Problem: Auth errors block user sessions. – Why SLI helps: Prioritize fixes for identity service. – What to measure: Auth success rate and latency. – Typical tools: Identity provider logs and tracing.

5) Background processing – Context: ETL pipelines ingest nightly data. – Problem: Backlog causes delayed reports. – Why SLI helps: Quantify task throughput and deadlines. – What to measure: Job success rate and queue depth. – Typical tools: Job queue monitors and scheduler metrics.

6) Mobile app crash rates – Context: Mobile client used by millions. – Problem: Crashes reduce retention. – Why SLI helps: Detect regressions early after release. – What to measure: Crashes per session and ANR rates. – Typical tools: Mobile SDK telemetry and crash reporting.

7) Search relevance latency – Context: Search service powering UX. – Problem: Slow or inaccurate results frustrate users. – Why SLI helps: Balance speed and relevance. – What to measure: Query latency and success of top-k relevance metrics. – Typical tools: Search engine metrics and A/B test telemetry.

8) Multi-region failover readiness – Context: Active-passive region failover. – Problem: Failover takes too long or loses data. – Why SLI helps: Test readiness and set expectations. – What to measure: Failover completion time and client success rate post-failover. – Typical tools: Synthetic probes and orchestration logs.

---

Scenario Examples (Realistic, End-to-End)

Scenario #1 — Kubernetes API service experiencing P95 spikes

Context: A microservice on Kubernetes serves user API requests.
Goal: Keep P95 latency under 300ms and maintain 99.9% success rate.
Why SLI matters here: SLI directly impacts user satisfaction and downstream services.
Architecture / workflow: Ingress -> Service -> Pod replicas -> Backend DB with cache. Metrics via OpenTelemetry scraped to Prometheus.
Step-by-step implementation:

1. Define SLI: P95 latency and success rate per endpoint over 30d rolling window.
2. Instrument endpoints with histograms and counters.
3. Configure Prometheus recording rules to compute denominators and percentiles.
4. Build dashboards and set alerting on SLO burn rate >3x.
5. Automate horizontal pod autoscaler based on request latency.
6. Run load tests and chaos to validate. What to measure: P95, success rate, pod CPU, pod restarts, kube events.
   Tools to use and why: Prometheus for metrics, Grafana dashboards, OpenTelemetry for instrumentation, K8s HPA for autoscaling.
   Common pitfalls: High-cardinality labels per user or request; sampling hiding slow traces.
   Validation: Synthetic load test and canary deployment with traffic split.
   Outcome: Detect latency spike early, auto-scale pods, and avoid SLO breach.

Scenario #2 — Serverless image-processing pipeline

Context: Event-driven image processing using managed functions.
Goal: Ensure image processing success rate > 99% and cold start <200ms for critical requests.
Why SLI matters here: Processing failure directly delays user content.
Architecture / workflow: Object storage triggers functions -> functions process -> write result -> notify. Telemetry via function provider metrics and custom traces.
Step-by-step implementation:

1. Define SLI for invocation success and cold start latency.
2. Add instrumentation to record processing time and errors.
3. Configure provider metrics ingestion and compute SLI in central engine.
4. Create alerts for error budget consumption and high cold start rates.
5. Use provisioned concurrency for hot paths.
   What to measure: Invocation errors, duration, cold starts, downstream storage success.
   Tools to use and why: Provider function metrics, tracing, synthetic tests for cold starts.
   Common pitfalls: Hidden retries masking errors, cost of provisioned concurrency.
   Validation: Simulated bursts and step function orchestration tests.
   Outcome: Stable processing with acceptable latency and controlled cost.

Scenario #3 — Incident response and postmortem after a DB outage

Context: Production database outage caused session failures.
Goal: Restore service and ensure similar incidents are prevented.
Why SLI matters here: SLI shows customer-facing impact and informs severity.
Architecture / workflow: Application -> DB cluster with read replicas. Telemetry includes DB metrics and app success rates.
Step-by-step implementation:

1. On-call receives page triggered by SLI breach of success rate.
2. Triage using dashboards and traces to identify DB saturation.
3. Execute runbook to failover to replica or scale DB nodes.
4. Restore traffic and monitor SLI stabilization.
5. Conduct blameless postmortem, identify root cause, and update SLI definitions and runbooks.
   What to measure: Success rate, DB CPU, connections, replication lag.
   Tools to use and why: APM for traces, DB monitoring, incident tracking.
   Common pitfalls: Incorrect SLI computation leading to delayed paging.
   Validation: Postmortem action verification and scheduled failover drills.
   Outcome: Faster restore next time and improved failover automation.

Scenario #4 — Cost vs performance trade-off for image CDN

Context: Serving images globally using CDN with tiered cache and origin.
Goal: Balance cost with latency; keep TTFB <150ms for target regions while reducing origin egress cost by 20%.
Why SLI matters here: SLI quantifies user impact of caching policies and infra cost changes.
Architecture / workflow: Edge CDN -> Cache layers -> Origin storage. Telemetry via edge logs and origin metrics.
Step-by-step implementation:

1. Define SLIs: TTFB and cache hit ratio per region.
2. Implement cache-control policy experiments with A/B cohorts.
3. Monitor SLI impact and cost metrics simultaneously.
4. Adjust TTLs and edge rules for regions with acceptable user impact.
5. Automate TTL tuning based on observed burn to SLO vs cost.
   What to measure: TTFB, cache hit ratio, origin egress cost per region.
   Tools to use and why: CDN metrics, cost analytics, synthetic probes.
   Common pitfalls: A/B cohorts not representative; missing correlation between cost and impact.
   Validation: Run controlled experiments and rollback if SLI degrades.
   Outcome: Achieve cost reduction while maintaining acceptable user latency.

---

Common Mistakes, Anti-patterns, and Troubleshooting

List of mistakes with symptom -> root cause -> fix (15+ items).

1. Symptom: Alerts trigger but no user impact. Root cause: Internal metric used as SLI. Fix: Re-evaluate SLI definition to be user-facing.
2. Symptom: SLI gaps appear intermittently. Root cause: Telemetry collection outage. Fix: Add collector health checks and redundancy.
3. Symptom: High alert volume. Root cause: Too many SLIs or low thresholds. Fix: Consolidate SLIs and tune thresholds; add suppression.
4. Symptom: SLOs never reached. Root cause: Unattainable targets. Fix: Reset SLOs after baseline analysis and iteratively improve.
5. Symptom: Post-deploy SLI regressions. Root cause: Poor canarying. Fix: Implement canary analysis and automatic rollback on burn spikes.
6. Symptom: Latency percentiles fluctuate wildly. Root cause: Small sample sizes or unfiltered bots. Fix: Increase sampling or filter noise.
7. Symptom: SLI reports contradict logs. Root cause: Definition mismatch or time-window misalignment. Fix: Align definitions and timestamps.
8. Symptom: Cost spikes from telemetry. Root cause: High-cardinality labels. Fix: Limit labels and use aggregation rules.
9. Symptom: Errors masked by retries. Root cause: Retries in clients hide source failures. Fix: Instrument original failure reasons and retry metrics.
10. Symptom: On-call confusion during incident. Root cause: Missing runbooks mapped to SLI. Fix: Create targeted runbooks and practice game days.
11. Symptom: SLO burnout halted all deployments. Root cause: Rigid governance not tied to risk. Fix: Add exception workflows and risk tiers.
12. Symptom: Synthetic checks green but users complain. Root cause: Synthetic not representative. Fix: Add RUM-based SLIs and diversify probes.
13. Symptom: Missing long-tail errors. Root cause: Trace sampling drops rare failures. Fix: Preserve error traces and use dynamic sampling.
14. Symptom: Slow SLI updates after deploy. Root cause: Metric aggregation window too large. Fix: Use smaller evaluation windows for rapid detection.
15. Symptom: Multiple teams reporting different SLI numbers. Root cause: No central SLI catalog. Fix: Establish canonical SLI definitions and governance.
16. Symptom: Dashboard performance issues. Root cause: Live queries over large retention. Fix: Use precomputed recording rules and materialized views.
17. Symptom: Security incident exposes telemetry. Root cause: Sensitive data in logs/labels. Fix: Mask PII and enforce data governance.
18. Symptom: SLIs encourage unsafe shortcuts. Root cause: Perverse incentives. Fix: Align SLOs with safety and security constraints.
19. Symptom: Observability blind spots in chaos tests. Root cause: Insufficient telemetry during failures. Fix: Instrument fail-paths and increase retention for incidents.
20. Symptom: SLIs do not reflect mobile issues. Root cause: No RUM or crash telemetry. Fix: Add mobile SDK telemetry and session-level SLIs.
21. Symptom: Alert dedupe not working. Root cause: Different alert signatures for same root cause. Fix: Group alerts by causal fingerprint or trace ID.

---

Best Practices & Operating Model

Ownership and on-call:

• Assign a service owner accountable for SLI health.
• On-call rotations include an SLO guard responsible for error budget monitoring.
• Define escalation paths for SLI breaches.

Runbooks vs playbooks:

• Runbooks: Concrete step-by-step remediation actions for specific SLI incidents.
• Playbooks: Higher-level coordination and communication templates for complex incidents.
• Maintain both in a searchable, versioned repository.

Safe deployments:

• Use canary and progressive rollout tied to SLO burn-rate evaluations.
• Automate rollback triggers based on burn rate thresholds.
• Use feature flags to reduce blast radius.

Toil reduction and automation:

• Automate enrichments, dedupe, and incident routing.
• Use runbook automation for repetitive fixes and safe mitigations.
• Invest in lightweight remediation scripts over manual touch.

Security basics:

• Do not log PII; mask sensitive fields in telemetry.
• Ensure telemetry pipeline ACLs and encryption at rest and in transit.
• Audit telemetry access and maintain retention policies aligned with compliance.

Weekly/monthly routines:

• Weekly: Review active error budget consumption and pending SLI changes.
• Monthly: SLO sanity check and trend analysis, review runbook relevance.
• Quarterly: SLI catalog audit and cross-team alignment.

What to review in postmortems related to SLI:

• Was SLI definition accurate for user impact?
• Did telemetry provide necessary context?
• Were alerts actionable and timely?
• Were runbooks effective and followed?
• Action items to improve SLI or instrumentation.

---

Tooling & Integration Map for SLI (TABLE REQUIRED)

ID	Category	What it does	Key integrations	Notes
I1	Metrics storage	Stores time-series metrics and recording rules	Prometheus OSS and remote write	Long-term retention via remote store
I2	Tracing backend	Stores and queries traces with sampling controls	OpenTelemetry and APM	Critical for distributed SLI debugging
I3	RUM / Frontend	Collects browser mobile telemetry	SDKs and tagging	Privacy constraints apply
I4	Synthetic monitoring	Executes scripted probes	CDN and global probes	Useful for SLA verification
I5	Alerting platform	Manages alerts and routing	Pager, Slack, Ops tools	Supports dedupe and escalation
I6	Visualization	Dashboards for SLI and SLO status	Grafana and BI tools	Use precomputed metrics
I7	Incident management	Tracks incidents and postmortems	Ticketing and runbook tools	Workflow automation helpful
I8	Log aggregation	Centralized log search and retention	Log shippers and SIEM	Correlate logs to SLIs
I9	CI/CD	Deploy gating and pipelines	Source control and pipelines	Enforce SLO checks in pipelines
I10	Cost analytics	Correlates cost with SLI changes	Cloud billing APIs	Helps balance cost vs reliability

Row Details (only if needed)

• None

---

Frequently Asked Questions (FAQs)

What is the difference between SLI and SLO?

SLI is the measured indicator; SLO is the target value set on that indicator.

How long should SLI windows be?

Common windows are 7d and 30d rolling; choose based on traffic patterns and business needs.

Can SLIs be computed from logs?

Yes, but logs must be structured and reliably ingested to compute numerators and denominators.

How many SLIs should a service have?

Start with 1–3 core SLIs per service; avoid proliferation.

Should synthetic checks count as SLIs?

They can supplement SLIs but are not a full replacement for real-user SLIs.

How do I handle low traffic services?

Use longer windows or aggregate across similar services to achieve stable SLI estimates.

How to prevent high-cardinality in SLI metrics?

Limit label sets, bucket user identifiers, and pre-aggregate dimensions.

Are SLOs public?

Depends; SLAs are public for contracts; internal SLOs can be private for operational use.

How to tie SLIs to business metrics?

Map SLIs to conversion, retention, and revenue impact via experiments and analytics.

How often should we review SLIs?

Monthly for operational health; quarterly for strategic alignment.

Can SLIs be adaptive or dynamic?

Advanced setups use adaptive SLOs, but they add complexity and governance needs.

How to test SLI instrumentation?

Use staging with replayed traffic and synthetic tests; validate numerator and denominator logic.

What is burn rate and how to use it?

Burn rate is the rate of error budget consumption; use thresholds to pause risky activities.

How to handle privacy with RUM?

Mask or sample data, and follow data residency and consent rules.

Do SLIs replace monitoring?

No, SLIs complement monitoring and trace-based diagnostics.

How to measure data freshness SLI?

Track timestamps of source events vs served content and compute age distributions.

How to manage SLIs across microservices?

Centralize SLI catalog and use dependency mapping to surface downstream impacts.

---

Conclusion

SLIs are the practical, measurable foundation of modern reliability practices. They align engineering work to user impact, enable risk-aware delivery, and provide objective inputs for incident response and continuous improvement.

Next 7 days plan:

• Day 1: Identify top 3 user journeys and draft SLI definitions.
• Day 2: Instrument one endpoint with metrics and validate ingestion.
• Day 3: Create recording rules and build an on-call dashboard.
• Day 4: Configure alerting on error budget burn-rate thresholds.
• Day 5: Run a mini game day and verify runbooks.
• Day 6: Review results and adjust SLO targets.
• Day 7: Document SLI catalog and assign ownership.

---

Appendix — SLI Keyword Cluster (SEO)

• Primary keywords
• SLI
• Service Level Indicator
• Service level indicator definition
• SLI vs SLO
• SLI measurement

• SLI best practices

• Secondary keywords

• SLO setup
• error budget
• observability for SLI
• SLI dashboard
• SLI alerting

• SLI instrumentation

• Long-tail questions

• what is an SLI in site reliability engineering
• how to measure an SLI for APIs
• how to compute SLI from metrics
• SLI examples for ecommerce checkout
• difference between SLI SLO SLA explained
• how to design SLI for serverless functions
• best tools to measure SLI in kubernetes
• how to set SLO targets based on SLI
• how to calculate error budget burn rate
• how often should you review SLI and SLO
• how to use synthetic monitoring as SLI
• how to avoid high cardinality in SLI metrics
• how to test SLI instrumentation in staging
• how to correlate SLIs with business KPIs
• how to handle privacy in RUM for SLI
• how to do canary SLI analysis
• how to design runbooks for SLI incidents
• how to use tracing to debug SLI breaches
• how to compute data freshness SLI

• how to maintain an SLI catalog

• Related terminology

• SLO governance
• observability pipeline
• prometheus recording rules
• openTelemetry SLI
• RUM metrics
• synthetic checks
• burn-rate policy
• canary deployment SLI
• error budget policy
• incident response SLI
• blameless postmortem
• trace sampling strategy
• metrics retention
• cardinality control
• service ownership
• runbook automation
• chaos engineering SLI
• adaptive SLOs
• multi-region failover SLI
• function cold start SLI
• cache hit ratio SLI
• TTFB SLI
• P95 latency SLI
• P99 latency considerations
• deployment success SLI
• job backlog SLI
• replication lag SLI
• session error rate SLI
• user-visible error SLI
• availability SLI definition
• throughput SLI
• success rate SLI
• SLI engine
• telemetry schema
• SLI dashboard templates
• SLI alert tuning
• SLI labeling guidelines
• SLI sampling rules
• SLI retention strategy

---