Kubernetes Without Night Shifts: A DevOps Prompt Playbook for Deploys, HPA, Secrets, and GitOps
Most Kubernetes incidents don't start with a catastrophic outage. They start with a typo in a YAML indentation, a missing readinessProbe, an HPA that scales on the wrong metric, or a Secret committed to Git in plain text. According to the CNCF Annual Survey, Kubernetes adoption in production keeps climbing year over year, and with it the surface area for exactly these small, expensive mistakes. The 3 a.m. page is rarely a mystery — it's usually something a reviewer could have caught.
That's where well-crafted prompts to an LLM change the economics of your on-call rotation. A good prompt isn't "write me a deployment." It's a structured request that encodes your constraints: resource limits, probe timeouts, rollout strategy, namespace conventions, and the failure mode you're trying to avoid. Used this way, an AI assistant becomes a second pair of eyes on every manifest and pipeline diff.
Below is a playbook of prompts organized by difficulty — from basic manifest generation to expert-level GitOps and incident triage. Each entry follows the same structure: the task, the prompt, and an example result you can adapt. All commands and APIs reference real, current Kubernetes and tooling behavior (see the official Kubernetes documentation and KEDA docs for verification).
Basic Prompts: Getting Clean Manifests on the First Try
1. Production-Ready Deployment with Probes and Resources
Task: Turn a vague service description into a Deployment that won't get OOMKilled or flap during rollouts.
Prompt:
Act as a Kubernetes SRE. Generate a Deployment manifest for a Node.js API
listening on port 3000. Requirements:
- image: registry.example.com/api:1.4.2
- 3 replicas, RollingUpdate with maxUnavailable: 0
- requests: cpu 250m / memory 256Mi; limits: cpu 500m / memory 512Mi
- livenessProbe on /healthz, readinessProbe on /ready, startupProbe on /startup
- runAsNonRoot: true, readOnlyRootFilesystem: true
Explain each probe's failureThreshold choice in one line.
Example result: The model returns a manifest with startupProbe.failureThreshold: 30 and periodSeconds: 10 (giving slow boots up to 5 minutes), a readinessProbe that gates traffic, and a livenessProbe with failureThreshold: 3. It explains that separating startup from liveness prevents the classic "app killed during boot" loop. maxUnavailable: 0 combined with maxSurge: 1 keeps capacity during rollout.
2. Explaining a CrashLoopBackOff from Logs
Task: Diagnose why a pod is restarting without reading a 2,000-line log by hand.
Prompt:
Here are the last 40 lines of `kubectl logs --previous` and the output of
`kubectl describe pod`. Identify the most likely root cause of the
CrashLoopBackOff. Give me: (1) the hypothesis, (2) the exact kubectl command
to confirm it, (3) the manifest fix.
[logs]
...
[describe]
...
Example result: A common pattern: describe shows Last State: Terminated, Reason: OOMKilled while logs show the process starting normally. The model points to memory limits being too low, suggests kubectl top pod to confirm, and proposes raising the limit or fixing a memory leak. It also warns not to raise limits blindly — a leak will just take longer to crash.
3. ConfigMap and Secret Wiring Without Leaks
Task: Connect configuration to a Deployment correctly, keeping secrets out of the image.
Prompt:
Show me the correct way to inject a ConfigMap (app-config) as env vars and a
Secret (db-credentials) as a mounted file into an existing Deployment.
Use envFrom for the ConfigMap but a volume mount for the Secret. Explain why
env vars are a poor choice for secrets.
Example result: The model returns envFrom.configMapRef, a volumes entry using secret.secretName, and volumeMounts with readOnly: true. The explanation: environment variables leak into kubectl describe pod, crash dumps, and child processes, while mounted files can be rotated and are less exposed. It references the Kubernetes Secrets good practices.
Advanced Prompts: Autoscaling, CI/CD, and Failures
4. Horizontal Pod Autoscaler on a Custom Metric
Task: Scale on request throughput, not just CPU.
Prompt:
Write an HPA v2 manifest that scales a Deployment between 2 and 20 replicas
targeting 100 requests-per-second per pod, using a custom metric exposed via
the Prometheus Adapter. Include the behavior block: scale up fast, scale down
slowly with a 5-minute stabilization window. Explain why stabilizing on
downscale matters more than upscale.
Example result: The output uses autoscaling/v2, metrics: - type: Pods with a Pods metric selector, and a behavior.scaleDown.stabilizationWindowSeconds: 300. The reasoning: aggressive scale-down after a short traffic dip causes thrashing and connection resets; a stabilization window smooths this. The model also reminds you that HPA needs metrics-server (for CPU/memory) plus the adapter for custom metrics — both are real, documented dependencies.
5. Fixing a Failing CI Pipeline from Error Output
Task: Move from "the pipeline is red" to a concrete fix.
Prompt:
My GitHub Actions job fails at the `docker build` step. Here is the workflow
YAML and the failing log excerpt. Diagnose the error, give me the corrected
step, and suggest a cache configuration to speed up future runs.
[workflow.yaml]
...
[failing log]
...
Example result: Typical root cause: a COPY package*.json step placed after RUN npm install, breaking layer caching, or a missing setup-buildx-action. The model reorders the Dockerfile steps, adds actions/cache for the npm directory, and adds docker/build-push-action with cache-from: type=gha. It notes that build caching is a real GitHub Actions feature documented in the Buildx cache docs.
6. GitOps: Structuring a Repository for Argo CD
Task: Design a repo layout that supports multiple environments without copy-paste.
Prompt:
Design a GitOps repository structure for Argo CD with three environments
(dev, staging, prod) using Kustomize overlays. The base should hold shared
manifests; overlays should only patch environment-specific values like
replicas and image tags. Show the directory tree and a sample Application
manifest with automated sync and self-heal.
Example result: The model produces a base/ + overlays/{dev,staging,prod}/ tree, a sample kustomization.yaml per overlay, and an Argo CD Application with syncPolicy.automated: {prune: true, selfHeal: true}. It flags prune: true as powerful but dangerous — a wrong commit can delete resources — and recommends starting with manual sync in prod. This aligns with Argo CD sync policy documentation.
7. Secrets Management Beyond base64
Task: Stop treating base64 as encryption.
Prompt:
Compare three approaches to Kubernetes secrets: (1) native Secret objects,
(2) Sealed Secrets, (3) External Secrets Operator pulling from a cloud vault.
For each, give the threat model it addresses, a short manifest example, and
when NOT to use it. Be specific about what base64 does and does not protect.
Example result: The model states plainly that base64 is encoding, not encryption, and anyone with get secret RBAC can read it. It shows a SealedSecret example and an ExternalSecret referencing AWS Secrets Manager, then recommends External Secrets when you already have a central vault, and Sealed Secrets for small teams without one. Each tool is real and actively maintained.
Expert Prompts: Incident Response and Infrastructure as Code
8. Post-Mortem Draft from Timeline Data
Task: Turn raw incident notes into a blameless post-mortem skeleton.
Prompt:
Here is a timeline of an incident: alert fired, first responder, actions taken,
mitigation, recovery. Draft a blameless post-mortem with sections: Impact,
Timeline, Root Cause, Detection, Response, Action Items. Keep the tone factual,
no blame. Mark any claim you cannot verify from the timeline as [needs data].
Example result: The model produces a structured doc and correctly leaves gaps marked [needs data] rather than inventing numbers. This is the key discipline: an LLM that refuses to fabricate the MTTR is more useful than one that guesses. The Google SRE Book is the canonical reference for the blameless format.
9. Scaling on Queue Depth with KEDA
Task: Scale workers based on backlog, not CPU.
Prompt:
Write a KEDA ScaledObject that scales a worker Deployment from 0 to 50 based on
the length of an AWS SQS queue, targeting 10 messages per pod. Include
scale-to-zero and a cooldown period. Explain the trade-off of scaling to zero
for latency-sensitive workloads.
Example result: The model returns a ScaledObject with minReplicaCount: 0, a triggers block for the SQS scaler, and cooldownPeriod: 300. It warns that scale-to-zero adds cold-start latency, so it's ideal for batch workers and wrong for user-facing APIs. KEDA and its SQS scaler are documented at keda.sh.
10. Reviewing a Manifest for Security Gaps
Task: Get a security review before merging.
Prompt:
Review this Pod spec against the Kubernetes Pod Security Standards (restricted
profile). List every violation, the exact field to change, and the corrected
YAML. Check: privileged, allowPrivilegeEscalation, capabilities, seccompProfile,
runAsNonRoot, and hostPath mounts.
Example result: The model flags allowPrivilegeEscalation not set to false, a missing seccompProfile: RuntimeDefault, and a hostPath mount. It returns a corrected spec. The restricted profile is defined in the official Pod Security Standards — a real, verifiable spec you can point teammates to.
11. Terraform vs. Helm: Choosing for Your Team
Task: Make an architecture decision, not just a manifest.
Prompt:
I need to manage ~30 microservices on EKS. Compare managing Kubernetes
resources with Terraform (kubernetes/helm providers) versus Helm charts versus
pure GitOps with Argo CD. Give a decision matrix: team size, drift handling,
rollback, secrets, learning curve. End with a recommendation for a 6-person team.
Example result: The model produces a Markdown table comparing the three, notes that Terraform's Kubernetes provider can fight with Argo CD over reconciliation (both want to own state), and recommends GitOps + Helm for a small team. This drift-conflict point is a well-known operational pitfall discussed in Argo CD and Terraform integration notes.
How to Get Reliable Output Every Time
A prompt is only as good as its constraints. Across all the examples above, three habits consistently produce usable results. First, always paste the real error output and the real YAML — the model reasons far better from concrete data than from descriptions. Second, demand the reasoning, not just the artifact: "explain why" forces the model to surface assumptions you can challenge. Third, ask it to mark uncertainty explicitly ([needs data], "I'm not sure, verify with X"). An assistant that admits gaps is safer on-call than one that hallucinates a flag.
Treat every generated manifest as a draft that still passes kubectl apply --dry-run=server, kubeconform, and a human review. The goal isn't to remove the engineer from the loop — it's to remove the 3 a.m. typo from the loop.
If you want to build these habits systematically, ASI Biont publishes practical, text-based guides on AI-assisted engineering, and our prompt collections for Docker, RAG systems, and UI/UX design follow the same task → prompt → result structure you saw here. Start with one prompt, apply it to a real manifest this week, and watch your on-call rotation get quieter.
Comments