The real cost of not having observability when you scale

A startup in Amsterdam raised a EUR 4 million Series A in March. By June they were serving 80,000 requests per minute across six services. In July, their checkout flow started intermittently returning 500s. Not always. Not for every user. For roughly 3% of requests, and only during peak hours, and only for users on mobile, and only when the payment service was under load.

They had a Grafana dashboard. It had been set up by a contractor six months earlier. It showed CPU usage, memory usage, request count, and a green line for “up.” All four lines were green. The dashboard was useless because it was answering the wrong question. It was telling them their servers were running. It was not telling them why their users could not check out.

They spent eleven days debugging this. Eleven days of a 3% checkout failure rate on a platform processing EUR 2 million per month in transactions. That is roughly EUR 660 per day in failed revenue, not counting the customers who tried twice, got the same error, and went to a competitor. Total cost of those eleven days: somewhere north of EUR 20,000 in lost revenue, plus 200 engineering hours across four engineers, plus the damage of a load-bearing feature being broken for a quarter of the month.

The root cause was a connection pool exhaustion in the payment service that only manifested when the inventory service responded slowly enough to hold connections open past the pool’s timeout. No single service was broken. The interaction between two services under specific load conditions was broken. You cannot debug that with a CPU graph. You need traces.

They had no distributed tracing. They had no structured logs that correlated across services. They had metrics that told them what was happening inside each service but not what was happening between them. They were observability-poor despite having all the tools installed.

This is the pattern I see at almost every startup that hits scale. They install Prometheus and Grafana, maybe follow a tutorial, and think they have observability. They have monitoring. Monitoring and observability are not the same thing, and the difference becomes expensive exactly when you need it most.

Monitoring versus observability

Monitoring tells you that something is wrong. Observability tells you why.

Monitoring is a health check. Is the server up. Is CPU below 80%. Is error rate below 1%. These are yes/no questions. They are useful for alerting. They are useless for debugging.

Observability is the ability to ask arbitrary questions of your system without deploying new code to answer them. Why is the checkout failing for mobile users. Which service is the bottleneck. What is the latency distribution for this specific endpoint at this specific time. Which downstream call is timing out.

The difference is in the data you collect. Monitoring collects aggregates. Observability preserves context. A metric that says “payment-service error rate: 3.2%” is monitoring. A trace that shows “this specific request failed because the inventory service took 840ms to respond, which caused the payment service to hold a database connection for 840ms, which caused the connection pool to exhaust, which caused the 34th request in the queue to time out” is observability.

You need both. Most startups have only the first one.

What you actually need: the three signals

There are three signals in a complete observability stack. Metrics, logs, and traces. I am going to cover what each one does, what it costs to set up, and the specific thing I instrument first in each category when I walk into a team that is flying blind.

Metrics: the cheap signal that most teams get wrong

Metrics are numbers aggregated over time. Request count, error count, latency percentile, memory usage, queue depth. They are cheap to collect, cheap to store, and fast to query. Prometheus is the standard. It is free, open source, and runs in your cluster as a single pod.

# prometheus-values.yaml, the configuration I start with
prometheus:
  prometheusSpec:
    retention: 15d
    resources:
      requests:
        cpu: 500m
        memory: 2Gi
      limits:
        cpu: 1
        memory: 4Gi
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false

The cost of running Prometheus is the memory it uses to store time-series data. For a startup scraping 50 services with 200 series each, that is 10,000 active series. At 15 bytes per sample and a 15-second scrape interval, you are storing roughly 1MB per minute. The 4Gi memory limit above is generous for this scale. On a Hetzner cluster, that is a fraction of a EUR 20 node.

The mistake teams make with metrics is not the collection. It is the cardinality. Every unique label combination creates a new time series. If you label your HTTP requests with path, status_code, and user_id, and you have 10,000 users, you have created a cardinality bomb. Prometheus will ingest it, store it, and eventually run out of memory. Then it crashes. Then you lose all your metrics during the crash, which is always the moment you need them.

# Wrong: user_id creates unbounded cardinality
REQUEST_COUNT = Counter(
    "http_requests_total",
    "Total HTTP requests",
    ["method", "path", "status_code", "user_id"]  # do not do this
)

# Right: user_id is excluded, cardinality is bounded
REQUEST_COUNT = Counter(
    "http_requests_total",
    "Total HTTP requests",
    ["method", "path", "status_code"]  # this is fine
)

The rule I enforce: no label whose value space is unbounded. User IDs, request IDs, session IDs, email addresses. None of these belong in metric labels. They belong in logs and traces, where high cardinality is expected and handled.

The four metrics I instrument first in every service:

1. Request count, labeled by method, path, and status code.
2. Request latency, as a histogram with buckets tuned to your SLO.
3. Error count, labeled by error type.
4. Resource saturation: queue depth, connection pool usage, goroutine count. Whichever one is the bottleneck for this service.

from prometheus_client import Counter, Histogram, Gauge

REQUESTS = Counter(
    "http_requests_total",
    "Total HTTP requests",
    ["method", "path", "status"]
)

LATENCY = Histogram(
    "http_request_duration_seconds",
    "Request latency",
    ["method", "path"],
    buckets=[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0]
)

DB_POOL_IN_USE = Gauge(
    "db_pool_in_use_connections",
    "Database connections currently in use"
)

Those four metrics answer 80% of the questions I get asked during incidents. Is traffic up. Are we slow. Are we erroring. Is a resource exhausted. Everything else is a refinement.

Logs: the signal everyone collects but nobody structures

Logs are the most familiar signal and the most misused one. Every team has logs. Most teams have logs that say {"level":"info","msg":"processing request"} with no request ID, no user context, no correlation to other services.

The single most important thing about logs is structured logging with a correlation ID. When a request enters your system, assign it a trace ID. Pass that ID to every service the request touches. Every log line includes that ID. When something fails, you grep for the trace ID and you see the entire journey of that one request across every service.

import structlog
import uuid

logger = structlog.get_logger()

def middleware(request):
    trace_id = request.headers.get("x-trace-id", str(uuid.uuid4()))
    request.state.trace_id = trace_id
    
    logger.bind(trace_id=trace_id).info(
        "request_received",
        method=request.method,
        path=request.path,
        user_id=request.user.id if request.user else None,
    )
    
    response = await call_next(request)
    
    logger.bind(trace_id=trace_id).info(
        "request_completed",
        status=response.status_code,
        duration_ms=response.elapsed_ms,
    )
    return response

That trace_id in every log line is the difference between a 30-minute debugging session and a 3-hour one. When the Amsterdam startup added trace IDs to their logs, they found the problematic requests in minutes. When they added distributed tracing on top, they found the root cause in an hour.

Logs are expensive at scale. This is the one signal where the cost grows with usage, not with services. A startup generating 50GB of logs per day on a managed logging service like Loki or Elasticsearch is paying for storage and indexing. The cost control here is log levels. Debug logs should not ship to production. Info logs should be selective. Warn and error logs should be comprehensive.

# loki-values.yaml, the configuration I use for startup-scale logging
loki:
  limits_config:
    retention_period: 14d
    max_streams_per_user: 10000
    ingestion_rate_mb: 10
    ingestion_burst_size_mb: 20

Loki is cheaper than Elasticsearch because it indexes labels, not content. You trade query flexibility for storage efficiency. For a startup, this is the right tradeoff. You are not running a security information and event management platform. You are trying to find out why a request failed.

Traces: the signal nobody sets up until it is too late

Distributed tracing is the signal that would have saved the Amsterdam startup eleven days. It is also the signal that most startups do not have, because it is the hardest to set up and the least understood.

A trace is a tree of spans. Each span represents a unit of work: an HTTP request, a database query, a downstream service call. The trace shows the entire path of a request through your system, with timing for each hop.

OpenTelemetry is the standard. It is the project that merged OpenTracing and OpenCensus, and it is the one I recommend without reservation. It auto-instruments HTTP, database, and messaging libraries in most languages, which means you get traces without rewriting your application code.

# opentelemetry setup for a Python service
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.psycopg2 import Psycopg2Instrumentor
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

trace.set_tracer_provider(TracerProvider(
    resource=Resource.create({"service.name": "payment-service"})
))

trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://otel-collector:4317"))
))

FastAPIInstrumentor.instrument_app(app)
Psycopg2Instrumentor().instrument()

The OTel collector runs as a sidecar or a cluster-wide service. It receives spans from your applications, batches them, and exports to a backend. For startups, I use Tempo or Jaeger as the trace backend. Both are open source. Tempo is Grafana-native, which means your traces, metrics, and logs are all in one UI. That integration matters more than people realize. During an incident, switching between three tabs to correlate signals costs minutes. One UI costs seconds.

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  tempo:
    endpoint: tempo:4317
  prometheus:
    endpoint: 0.0.0.0:8889

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [tempo]
    metrics:
      receivers: [otlp]
      exporters: [prometheus]

The cost of tracing is higher than metrics because each trace is a record, not an aggregate. The cost control is sampling. You do not need to trace every request. At startup scale, tracing 10% of requests is enough to understand the system. When something fails, you trace 100% of failing requests by configuring the sampler to keep all error spans.

from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased, ALWAYS_ON

# Sample 10% of normal traffic, keep 100% of errors
sampler = ParentBased(
    root=TraceIdRatioBased(rate=0.1),
)

This is the configuration that balances cost and visibility. 10% sampling gives you enough traces to see normal behavior patterns. Error-based sampling ensures every failure is captured. The combination means you spend most of your tracing budget on the requests that matter.

What it costs to build versus what it costs to skip

The observability stack I just described costs roughly EUR 0 in software licenses. Prometheus, Grafana, Loki, Tempo, OpenTelemetry. All open source. The cost is infrastructure and time.

Infrastructure cost for a startup running 5 to 10 services on a single Kubernetes cluster: one additional node for the observability stack, or roughly 2 CPU and 4GB of RAM carved out of existing nodes. On Hetzner, that is EUR 10 to EUR 20 per month. On AWS or GCP, EUR 40 to EUR 80 per month. Round it to EUR 50 per month as a planning number.

Time cost: 5 to 7 days of focused work for someone who has done it before. Instrumenting services, configuring collectors, building dashboards that answer real questions, writing alerting rules. If you have never done it, double that. If you are learning OpenTelemetry from scratch, triple it and accept that your first attempt will have gaps.

The Amsterdam startup spent EUR 20,000 in lost revenue and 200 engineering hours debugging a problem that distributed tracing would have identified in under an hour. Their observability stack, had they built it before scaling, would have cost EUR 50 per month and one week of engineering time. The ratio of loss to prevention is roughly 40 to 1, and that is for a single incident. A startup without observability will have multiple incidents like this as it scales. The first one pays for the stack many times over.

What I instrument first, in order

When I walk into a team that has no observability, I do not build everything at once. I build in this order, because each step builds on the last and each step independently reduces blind spots.

First: metrics on the four golden signals for every service. Request count, latency, error rate, saturation. This takes two days and gives you dashboards that tell you if something is wrong. It does not tell you why, but it tells you when, and that is the first question during an incident.

Second: structured logs with trace IDs. This takes one day and gives you the ability to follow a single request through a single service. Combined with metrics, you can now identify the failing service and look at its logs for the failing requests.

Third: distributed tracing with OpenTelemetry. This takes three to four days and gives you the ability to follow a single request across all services. This is the step that answers the “why” questions. It is also the step most teams skip, and it is the step that would have saved the Amsterdam startup eleven days.

Fourth: alerting rules on top of metrics. This takes one day and gives you the ability to know about problems before users do. The alerts I write are: error rate above 1% for 5 minutes, p99 latency above 500ms for 5 minutes, any service down for 2 minutes. Three alerts per service. Not 340.

Fifth: dashboards that combine all three signals. A Grafana dashboard that shows the metrics graph, with log lines overlaid, and a link to the trace for the selected time window. This is the dashboard you use during incidents. It takes one day to build and it is the difference between debugging in one tab and debugging in three.

Total: 8 to 10 days for a complete observability stack. You can do it in less if you are experienced. You should not do it in less if you are not, because the gaps you leave will be the gaps that bite you.

The dashboard anti-pattern

One more thing, because I see it everywhere. Teams build dashboards that look impressive and communicate nothing. A dashboard with 40 panels, each showing a different metric, is not observability. It is a museum. Nobody looks at 40 panels during an incident. They look at the one panel that tells them where the fire is, and they need to find it in under 10 seconds.

The dashboard I build for every team has exactly four panels on the first row: error rate, p99 latency, request rate, saturation. The second row has a service map showing which services are calling which, with latency and error rate on each edge. The third row has log lines for the selected time window. One dashboard. One screen. During an incident, you open it, you look at the first row to see which signal is red, you look at the service map to see which edge is the bottleneck, you look at the logs to see what the error message is. Thirty seconds from question to clue.

Everything else is a secondary dashboard for specific debugging. Those are useful. But the incident dashboard is the one that matters, and it should be designed for a stressed engineer at 2am, not for a VP who wants to see pretty graphs in a meeting.

The honest summary

The Amsterdam startup built their observability stack after the incident. It took six days. Their next incident, two months later, was a cache invalidation bug that affected 0.5% of requests. They found it in 40 minutes. The trace showed the cache miss, the logs showed the stale data, the fix was a one-line config change. The difference between 11 days and 40 minutes was not talent or effort. It was visibility.

If you are a startup reading this and recognizing the pattern, Basecamp exists for exactly this reason: digitalaultis.com/basecamp

Observability is not a luxury you add after you scale. It is the thing that lets you scale without breaking. The cost of building it is fixed and known. The cost of not building it is variable, unpredictable, and always higher than you think. Build it before the incident, not during it.