Imagine: a Go service handles 2,000 requests per second, but under peak load of 5,000 it starts returning 503. The logs are clean, no errors, but latency grows from 50 ms to 2 seconds. Sound familiar? This is a classic situation where concurrency in Go is implemented "by the textbook" but without production patterns. Let's break down how a practitioner solves such a problem and where the scheme usually breaks.
Why Go and gRPC are not just "another stack"
Go wins where predictable concurrency is needed. Goroutines are cheap (initial stack ~2 KB), the scheduler efficiently distributes them across cores, and channels provide a simple way to synchronize. gRPC over HTTP/2 adds strict typing via protobuf, multiplexing, and streaming. Together they allow building services that scale easily and process requests quickly.
But the devil is in the details. For example, context.Context is a mandatory parameter for any RPC call. If you ignore it, timeouts don't work, goroutines leak, and the service stops shutting down correctly. This is one of the first traps for beginners.
Task breakdown: from slow response to stable service
Let's take a typical order service. When requesting /orders/{id}, it goes to the database, then to the cache, then to the payment gateway — all sequentially. Under a load of 2,000 RPS, the average response time is 120 ms, but the 95th percentile is already 800 ms. Add gRPC streaming for status updates — and you get a goroutine leak if the stream is not closed when the connection breaks.
What a practitioner does:
- Parallelizes independent calls. The database and cache are queried simultaneously via
errgroup.Group. Response time drops to 70 ms, the 95th percentile to 300 ms. - Adds timeouts and retries with backoff. For the payment gateway, sets a deadline of 500 ms and 2 retries with exponential backoff. This reduces the number of errors during short-term failures but requires idempotent operations.
- Enables monitoring via OpenTelemetry. Traces show that 60% of the time is spent on protobuf serialization during streaming — a non-obvious conclusion that is not visible in regular logs.
If this is not done, under peak load goroutines accumulate, memory grows, GC starts firing more often — and the service degrades avalanche-like. You can notice the problem in advance by the go_goroutines metric and GC execution time. If it's already too late, only graceful shutdown with connection draining and forced termination of stuck goroutines will help.
Where the scheme itself breaks
Even a properly built service can fail. Here are three scenarios encountered in production.
1. Conditions changed: gRPC stream is not closed
The client disconnected, but the server continues sending messages to the goroutine. If you don't listen to ctx.Done() in the stream loop, the goroutine lives forever. Over time, thousands accumulate. How to notice: growth of go_goroutines without load growth. What to do: always use select with ctx.Done() and close the stream on signal.
2. Rule not applied: logging middleware doesn't see errors
The standard interceptor logs only the gRPC response code, but not business errors inside. As a result, logs show "OK", but users complain. How to notice: discrepancy between error metrics on the client and server. What to do: add error extraction from context to the middleware or use custom codes.
3. Tool stopped working: pprof doesn't provide a profile
With profiling enabled in production without limits, pprof itself starts creating load. Or the pprof port is closed by a firewall. How to notice: the profile is not captured, although the endpoint is accessible. What to do: move pprof to a separate port with authentication and use selective profiling.
Learning paths: self-study vs structured
Many start with documentation and articles. It's free but requires discipline: without feedback, it's easy to miss important details — for example, that context.WithTimeout must be canceled, otherwise a leak. Practice on pet projects gives experience but doesn't simulate real failures.
Courses provide structure and error analysis. For example, in the course Go for Backend (gRPC), they cover exactly such scenarios: how to set up graceful shutdown, how to write table-driven tests for concurrent code, how to avoid leaks in streams. An alternative is mentoring or code review in a team, but that's more expensive in time and money.
Who this path suits
If you are already familiar with the basics of Go and want to move to the level of production development — this material is for you. If you are just starting to program, you should first master basic syntax and at least one project. Don't expect the course to replace real experience: it provides tools, but you will have to apply them yourself.
In conclusion, it's worth emphasizing: reliable and fast backends in Go are built not on magic but on understanding concurrency, timeouts, and monitoring. The course Go for Backend (gRPC) helps systematically cover these topics and avoid typical mistakes — details and enrollment on the course page.
You can start here: Go for Backend.