Service Mesh

Overview

A service mesh moves communication between services into an infrastructure layer: a helper process alongside each service intercepts all traffic and applies retries, timeouts, mutual encryption, balancing and observability.

The application stops knowing about those concerns. A retry policy comes to hold for all services, in all languages, without touching code.

It is this section's technology with the largest distance between adoption and need — and this document treats that frankly, because the decision to adopt is frequently made for the wrong reason.

Problem

In a mesh with many services, each one needs retries, timeouts, a circuit breaker, discovery, mutual encryption and distributed tracing.

In one language, a shared library solves it. In five languages, that is five libraries, with behaviors that diverge — and a policy change requires updating and redeploying dozens of services.

That is the real pain point: a uniform communication policy, in a polyglot environment, without redeploying everything.

Core Concepts

The sidecar and the control plane

Each service instance gains a process alongside it — the sidecar — that intercepts all inbound and outbound traffic. The service talks to localhost; the sidecar does the rest.

The control plane distributes configuration and certificates to all the sidecars. It is where the policy is declared once.

service A → sidecar A ══mTLS══> sidecar B → service B
                 ↑                  ↑
                 └── control plane ──┘

The consequence: changing the timeout policy of the whole environment is editing configuration, not code.

What it actually delivers

Mutual encryption between services, with automatic certificate rotation. That is, in practice, the strongest argument. Implementing it in the application, in several languages, with rotation, is considerable and error-prone work.

Per-call balancing. It solves the problem described in gRPC: long multiplexed connections that pin clients to instances.

Uniform retries, timeouts and circuit breaking.

Observability with no instrumentation. Service-to-service traffic metrics — latency, error rate, volume — for every pair, without touching code.

Traffic splitting. Gradual rollouts and tests with a fraction of the traffic, controlled by configuration.

Service-to-service authorization. Which service can call which, declared centrally.

What it costs

Being specific, because it is what decides:

Latency. Two additional hops per call — one at each sidecar. Typically a few milliseconds, and relevant in long chains.

Resources. One sidecar process per instance. Across hundreds of instances, that is CPU and memory that add up.

Diagnostic complexity. When something fails, the question "was it the application or the sidecar?" has to be answered before anything else. And the error the application sees may have been generated by the mesh.

Operational knowledge. It is an entire platform, with its own configuration model, its own failure modes and its own upgrades. Somebody needs to master it.

Coupling to the platform. Leaving later is expensive.

Layered retries are the characteristic failure mode

If the mesh retries and the application does too, the attempts multiply.

application: 3 attempts
mesh:        3 attempts per attempt
result:      9 calls to the destination

The factor compounds at every hop: in a chain of three, that is 9³ = 729 calls at the end. A mild degradation at the destination becomes an avalanche. See retry storms.

The rule: retry in one layer only, and know which one.

The honest adoption criterion

A mesh pays for itself when several of these are true at the same time:

dozens of services, not a handful
several languages
mutual encryption between services is a requirement
mature container orchestration already exists
somebody operates the platform
the cost of divergent policy already hurts today

With few services, one language and a shared library, the mesh adds an entire platform to solve a problem that does not exist yet.

It is worth being direct: most adoptions that go wrong start from "everybody is using it" and not from any of the lines above.

It does not replace the gateway

A mesh handles traffic between services. A gateway handles the traffic that enters.

They are different layers, and they coexist in most designs.

Mental Model

The mesh moves communication from the application to the infrastructure. That is profit when there are many services and many languages; it is a platform to operate with no corresponding problem when there are not.

When to Use

• Dozens of services, in several languages.
• Mutual encryption between services is a requirement.
• A uniform communication policy without redeploying.
• Per-call balancing — especially with gRPC.
• Traffic splitting for gradual rollout.
• Service-to-service observability without instrumenting each one.
• Mature orchestration and a platform team already exist.

When Not to Use

With few services. A shared library solves it.

With a single language. The main gain disappears.

With nobody to operate the platform.

When the additional latency is unacceptable.

For inbound traffic. See gateway.

Without resolving layered retries first.

To "solve" architecture problems. A mesh does not fix badly drawn service boundaries — it makes it easier not to notice they are wrong.

Alternatives

• A shared library — the same policy, with no extra process and no latency. It requires one language, or one library per language.
• Discovery and balancing in the platform — orchestrators already offer part of this.
• Mutual encryption without a mesh — more laborious and possible.
• A sidecar-less mesh — models that put the function on the node instead of per instance, reducing consumption and latency.

Trade-offs

With a mesh	A library
Language-independent	One per language
Policy with no redeploy	Redeploy everything
Additional latency	None
Resources per instance	None
A platform to operate	A dependency to version
Diagnosis across two layers	One

Failure Modes

Multiplied retries.

A sidecar consuming more than the service. Common in lightweight services.

An error attributed to the wrong layer. Hours lost before looking at the mesh.

An unavailable control plane. Configuration frozen; sidecars keep the last one received, and nothing changes.

An expired certificate. Service-to-service communication stops, and the cause is not obvious.

A wrong configuration isolating a service. A badly written authorization policy cuts the traffic.

Startup order. The service comes up before the sidecar and the first calls fail.

Common Mistakes

Adopting it as a trend.

Retrying in the mesh and in the application.

Not sizing the sidecars' consumption.

Not training the team beforehand.

Expecting it to solve service boundary problems.

Adopting everything at once instead of starting from one capability — typically mutual encryption — and expanding.

Real-World Example

A financial technology company with 60 services, in four languages, adopted a service mesh. The trigger was legitimate: a regulatory requirement for encryption between all internal services, with certificate rotation.

Implementing that in four languages was estimated at five months of work distributed across several teams, with a risk of divergence.

The mesh delivered it in six weeks, and the rest came along with it: per-service-pair metrics, a uniform timeout policy and gradual rollout by traffic splitting.

Three problems:

An avalanche from retries. The applications already retried three times. The mesh was configured with three. In a mild degradation of a query service, the three-hop chain generated 729 calls per original request. The service, which was slow, went down completely. The diagnosis took six hours because the application's metrics showed three attempts, and the destination at the end of the chain showed 729. Retries were removed from the applications and centralized in the mesh.

The sidecars' consumption. Small, low-traffic services came to consume more in the sidecar than in the application itself. The infrastructure cost rose 30% before anybody connected the dots. Solved with limits per service profile.

An expired certificate. A failure in the control plane prevented rotation. The certificates expired overnight, and all service-to-service communication stopped. The alerts pointed at connection failures in dozens of services simultaneously, and the root cause took 40 minutes to find. An alert for approaching expiration came to exist.

The conclusion recorded: the adoption was right because there was a concrete requirement the alternative did not serve well. A neighboring team, with 8 services in a single language, adopted the same mesh "to standardize" and removed it fourteen months later — the operational cost was real and the benefit did not exist in that context.

Related Concepts

• API Gateways — inbound traffic.
• gRPC — where per-call balancing matters.
• Retries — the multiplication risk.
• Observability.

Practical Exercise

If you use a mesh, check how many layers retry: the application, the mesh, the HTTP client.

Multiply the factors of each layer to get the factor per hop, then raise that factor to the number of hops in your longest chain — the amplification is exponential in depth, not linear. That is the number of calls a single request can generate in a degradation.

Interview Questions

• What is the strongest argument for adopting a mesh?
• How do layered retries produce an avalanche?
• When is a shared library the better choice?

Further Reading

• Morgan, William. What's a service mesh? And why do I need one?, 2017.
• Calçado, Phil. Pattern: Service Mesh, 2017.
• Newman, Sam. Building Microservices. 2nd ed. O'Reilly, 2021.