How We Rebuilt the Search Architecture for High Availability in GitHub Enterprise Server

## Introduction In the fast-paced world of software development, search functionality is often the silent workhorse of developer productivity. When GitHub Enterprise Server (GHES) users search for code, issues, or pull requests, they expect results in milliseconds — regardless of the load on the underlying infrastructure. However, as organizations scale their repositories and user bases, the search architecture must evolve to maintain both performance and resilience. Recently, the GitHub engineering team published a detailed case study on how they rebuilt the search architecture for high availability in GHES. This article dives deep into the technical decisions, trade-offs, and implementation strategies that made this migration successful. We’ll explore the challenges of running Elasticsearch at scale, the shift from a single-node to a multi-node cluster, and the operational patterns that ensure zero-downtime upgrades. ## The Challenge: Single-Point-of-Failure in Search Before the rebuild, GitHub Enterprise Server relied on a single Elasticsearch node for all search indexing and querying. While this worked well for smaller deployments, it introduced a critical single point of failure. If the node went down — due to hardware failure, resource exhaustion, or a software bug — the entire search feature became unavailable. For organizations with thousands of developers, this meant lost productivity and frustrated users. ### Key Metrics from the Original Architecture | Metric | Value | |--------|-------| | Cluster topology | Single node | | Index replication factor | 0 (no replicas) | | Recovery time on failure | Hours (manual reindex) | | Maximum query throughput | Limited by node capacity | | Upgrade impact | Full downtime required | ## The Solution: Multi-Node Elasticsearch Cluster with High Availability GitHub’s engineering team redesigned the search layer to run on a multi-node Elasticsearch cluster. The new architecture provides automatic failover, seamless rolling upgrades, and improved query throughput. The migration was carefully orchestrated to avoid disrupting existing users. ### Architectural Changes The rebuild introduced three fundamental changes: 1. **Multiple Elasticsearch nodes** — Instead of a single node, the cluster now consists of three or more nodes, each capable of serving queries and handling indexing. 2. **Index replication** — Every index is replicated across multiple nodes. If one node fails, another replica can immediately take over without data loss. 3. **Rolling upgrade support** — Nodes can be upgraded one by one, ensuring the search service remains available throughout the process. ## Migration Strategy: Blue-Green Deployment To minimize risk, GitHub used a blue-green deployment pattern. They deployed a parallel set of Elasticsearch nodes alongside the existing single node. During a transition window, they redirected search traffic to the new cluster while the old cluster continued to serve as a fallback. Once the new cluster was validated, the old node was decommissioned. ### Data Synchronization Keeping both clusters in sync was critical. GitHub leveraged Elasticsearch’s snapshot and restore functionality to seed the new cluster with existing indices. Then, they implemented a dual-write pattern: all new indexing operations were sent to both the old and new clusters simultaneously. This approach ensured zero data loss and allowed for a clean cutover. ## Operational Patterns for High Availability ### Health Checks and Auto-Healing The new architecture includes automated health checks that monitor node status, disk usage, and query latency. If a node becomes unhealthy, the cluster automatically redistributes shards to healthy nodes. This self-healing behavior reduces the need for manual intervention. ### Capacity Planning GitHub recommends planning for at least three Elasticsearch nodes to achieve high availability. The exact number depends on the expected index size and query volume. A useful rule of thumb: provision enough nodes so that the cluster can tolerate the failure of one node without degrading performance. | Deployment Size | Recommended Nodes | Replication Factor | |-----------------|-------------------|--------------------| | Small ( 50k users) | 7+ | 2+ | ### Monitoring and Alerts Effective monitoring is essential. GitHub integrated Elasticsearch metrics into their existing observability stack, tracking: - Query latency (p50, p95, p99) - Indexing throughput - Disk usage per node - Cluster health status Alerts are triggered when any metric exceeds predefined thresholds, enabling the operations team to respond before users are impacted. ## Performance Results After the migration, GitHub observed significant improvements in search availability and performance: - **Query latency** remained stable even during node failures (p99 < 200ms) - **Indexing throughput** increased by 40% due to parallel writes across nodes - **Recovery time** dropped from hours to minutes (automatic shard relocation) - **Upgrade time** reduced to zero downtime — rolling upgrades completed without interrupting search ## Lessons Learned and Recommendations 1. **Start small, scale gradually** — Begin with a three-node cluster and expand as your data grows. Over-provisioning early wastes resources, but under-provisioning risks instability. 2. **Invest in monitoring** — Without proper observability, high availability is blind. Ensure you have visibility into cluster health at all times. 3. **Test failure scenarios** — Simulate node failures in a staging environment to validate that your cluster behaves as expected. 4. **Document the cutover process** — A well-documented migration plan reduces risk and ensures all team members know their roles. ## Conclusion Rebuilding the search architecture for high availability in GitHub Enterprise Server was a complex but rewarding endeavor. By moving from a single Elasticsearch node to a multi-node cluster with replication and rolling upgrade support, GitHub eliminated a critical single point of failure and improved the developer experience. The lessons from this project are applicable to any organization running search at scale: invest in redundancy, monitor relentlessly, and plan migrations with care. For teams looking to implement similar high-availability patterns for their own search infrastructure, the approach documented by GitHub provides a solid blueprint. As the company continues to evolve its platform, this architectural foundation will support future growth and innovation. [Source](https://github.blog/engineering/architecture-optimization/how-we-rebuilt-the-search-architecture-for-high-availability-in-github-enterprise-server/) *ASI Biont supports connecting to Elasticsearch and other search systems via API — learn more at asibiont.com*

How We Rebuilt the Search Architecture for High Availability in GitHub Enterprise Server

Comments

Recent articles

Chief AI Officer Course: How AI & Data Science Leadership Turns CTOs into AI Transformation Strategists

Product Management & Growth Course: How to Manage a Product and Scale a Business with AI Learning on asibiont.com

Claude Code Antifraud: Kolmogorov–Smirnov and the Not-So-Childish Anomalies in a Children's Olympiad

Housing Law from Scratch: How an AI Course on Asibiont.com Helps You Understand the RF Housing Code and Resolve Disputes

Your AI Butler for Yandex: Automating Taxi, Music, and Food with ASI Biont Integration

9 Demos of Gemini Omni and Gemini 3.5 in Action: Real Results from a Practitioner

Learning to Lead in a Hybrid Human-AI Enterprise: The New Playbook for 2026

Master Business Automation with AI Automation & Agents Course at Asibiont.com: Learn n8n, Make.com, and RPA Without Coding

How to Automate Your Gmail with an AI Agent: Smart Email Sorting, Drafting, and Spam Filtering That Saves Hours Weekly