How to Automate Production System Monitoring with Prometheus and Grafana: A Step-by-Step Guide

Woken Up in the Middle of the Night? Time to Build Observability

Imagine: it's 3 AM, you're asleep, and your phone is blowing up with alerts. A microservice has crashed, users are complaining, and you don't know where to start the investigation. Sound familiar? If your production system runs without automated monitoring, you're not just losing money—you're risking your reputation.

But here's the good news: in recent years, the Prometheus + Grafana stack has become the gold standard for observability. It's open source, scales to thousands of services, and lets you configure alerts so you only wake up when it really matters.

In this article, we'll break down a real-world case: how to automate monitoring for a microservice application in one day—from metric collection to dashboards and alerts. You'll get ready-made configs and understand how to not just "install Prometheus," but build a system that truly helps with on-call duty.

The Problem: Microservices Grow, But Monitoring Doesn't

Let's say you have a typical application: a React frontend, a Go backend (three microservices—auth, order, payment), PostgreSQL, Redis, and a few workers for background tasks. Everything runs on Kubernetes.

Problems the team faces:
- No unified dashboard: some people look at logs in Kibana, others check metrics in a cloud console.
- Alerts come in batches: "CPU 90%," but it's unclear which service is to blame.
- Incidents take hours to investigate: you need to manually gather data from five sources.

Goal: in 24 hours, set up a stack that answers three questions:
1. What's the current state of the system?
2. What went wrong?
3. Who should fix it?

The Solution: Architecture with Prometheus and Grafana

We'll build a minimal but production-ready architecture:

Important: we're not installing everything "out of the box." We focus on automation—so configs are versioned in Git, and dashboards are imported with a single command.

Step 1: Install Prometheus Operator in Kubernetes

The fastest way is to use Prometheus Operator. It manages the lifecycle of Prometheus instances, alert rules, and ServiceMonitors.

Install via Helm:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack --namespace monitoring --create-namespace

This single command deploys:
- Prometheus (with 15-day retention settings)
- Grafana (with pre-installed Kubernetes dashboards)
- Alertmanager
- node_exporter and kube-state-metrics

After 5 minutes, check everything is working:

kubectl get pods -n monitoring

You should see all pods in Running status.

Step 2: Add Metrics from Application Code

Prometheus collects metrics via the /metrics HTTP endpoint. To have your microservices expose metrics, add the client library.

Example for Go (order microservice):

package main

import (
    "net/http"
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
    httpRequestsTotal = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "http_requests_total",
            Help: "Total number of HTTP requests",
        },
        []string{"method", "endpoint", "status"},
    )
    requestDuration = prometheus.NewHistogramVec(
        prometheus.HistogramOpts{
            Name:    "http_request_duration_seconds",
            Help:    "Duration of HTTP requests",
            Buckets: prometheus.DefBuckets,
        },
        []string{"method", "endpoint"},
    )
)

func init() {
    prometheus.MustRegister(httpRequestsTotal)
    prometheus.MustRegister(requestDuration)
}

func main() {
    http.Handle("/metrics", promhttp.Handler())
    // ... rest of the code
}

After deploying the application, create a ServiceMonitor—it tells Prometheus where to scrape metrics:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: order-service-monitor
  namespace: monitoring
  labels:
    release: prometheus
spec:
  selector:
    matchLabels:
      app: order-service
  endpoints:
  - port: http
    path: /metrics
    interval: 15s

Prometheus Operator automatically picks up this resource and starts scraping.

Step 3: Create a Dashboard in Grafana

Grafana comes bundled with kube-prometheus-stack. Access it:

kubectl port-forward svc/prometheus-grafana 3000:80 -n monitoring

Login: admin, password: prom-operator (default).

Now create a dashboard for our order service. Import a ready-made JSON (ID 11074 from Grafana Labs) or write your own.

Example simple dashboard (import via API):

{
  "title": "Order Service Dashboard",
  "panels": [
    {
      "title": "HTTP Requests Rate",
      "type": "graph",
      "targets": [
        {
          "expr": "rate(http_requests_total{endpoint=\"/api/orders\"}[5m])",
          "legendFormat": "{{status}}"
        }
      ]
    },
    {
      "title": "P99 Latency",
      "type": "graph",
      "targets": [
        {
          "expr": "histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))",
          "legendFormat": "p99"
        }
      ]
    }
  ]
}

Important tip: use dashboard variables (e.g., $service) to switch between microservices without duplicating panels.

Step 4: Configure Alerts in Alertmanager

Alerts are the heart of automation. We want to receive notifications only when something truly needs attention.

Create an alert rule in PrometheusRule:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: order-service-alerts
  namespace: monitoring
  labels:
    release: prometheus
spec:
  groups:
  - name: order-service
    rules:
    - alert: HighErrorRate
      expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.05
      for: 2m
      labels:
        severity: critical
      annotations:
        summary: "High error rate on order service"
        description: "Error rate is {{ $value | humanizePercentage }} for the last 5 minutes"
    - alert: HighLatency
      expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 2
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "P99 latency is high"
        description: "P99 latency is {{ $value }}s"

Configure Alertmanager to send to Telegram:

apiVersion: v1
kind: Secret
metadata:
  name: alertmanager-config
  namespace: monitoring
data:
  alertmanager.yaml: |
    global:
      resolve_timeout: 5m
    route:
      receiver: 'telegram'
      group_wait: 10s
      group_interval: 5m
      repeat_interval: 4h
    receivers:
    - name: 'telegram'
      telegram_configs:
      - bot_token: '<YOUR_BOT_TOKEN>'
        chat_id: <YOUR_CHAT_ID>
        parse_mode: 'HTML'

Apply the config, and you'll receive alerts directly in Telegram. Now nighttime calls will only be for real issues.

Step 5: Automate Deployment (IaC)

To avoid manual setup every time, use GitOps with ArgoCD or Flux. Store all configs in Git:

├── helm/
│   └── kube-prometheus-stack/
│       └── values.yaml
├── servicemonitors/
│   ├── order-service.yaml
│   └── auth-service.yaml
├── prometheusrules/
│   └── alerts.yaml
└── grafana/
    └── dashboards/
        └── order-service.json

When code or configs change, the CI/CD pipeline automatically updates monitoring. This eliminates human error and saves hours.

Results: What Changed in a Day

After implementing the stack, the team achieved:
- Unified dashboard—Grafana shows the state of all microservices.
- Reduced investigation time from 2 hours to 15 minutes: dashboards show which endpoint is slow.
- Noise-free alerts—only critical errors and high latency.
- Automated deployment—a new microservice is added with a simple PR in Git.

Specific numbers (from our experience):
- Incident detection time dropped from 10 minutes to 30 seconds.
- False alerts reduced by 70% thanks to grouping and for conditions.
- Deploying a new service with monitoring now takes 5 minutes instead of an hour.

Conclusion: Observability Is Not an Option, It's a Necessity

Prometheus and Grafana are a powerful duo, but without proper architecture, they become "just another tool." The key to success is automation: ServiceMonitor, PrometheusRule, and GitOps.

If you want to dive deeper into building a production observability system—from SLI/SLO to on-call and postmortems—check out the Observability course at asibiont.com. There, you'll learn not just how to install Prometheus, but also how to design alerts, integrate distributed tracing, and work with Loki. It's practical experience that will save you weeks of experimentation.

In the meantime, grab the configs from this article and try deploying the stack today. Your production system will thank you.