Hermes + Qwen3.6-27B: How I Built the First Line of Email Support

Introduction

Customer support is often the first real test for any AI implementation. While chatbots and automated replies have been around for years, most businesses still struggle with a simple question: how do you build a system that actually handles real-world email queries without constant human intervention? A recent case study published on Habr (July 2026) details exactly that challenge and the solution developed by a team at GPugo. The article describes how they combined Hermes, a custom email processing framework, with the Qwen3.6-27B large language model to create a fully automated first line of email support. The results are impressive: the system now handles a significant portion of incoming tickets autonomously, reducing response times drastically. This article reviews their approach, the technical decisions they made, and the lessons learned along the way.

Source

The Problem: Why Traditional Email Support Automation Fails

Most companies rely on rule-based systems or simple keyword matching to sort and respond to customer emails. The authors explain that such systems fail in three key areas:

• Context misunderstanding: A rule might catch the word "refund" but miss the nuance — is the customer asking for a refund, reporting a delayed refund, or complaining about the refund policy?
• Language variability: Customers phrase the same issue in countless ways. Rules break with typos, slang, or complex sentences.
• Escalation logic: Deciding when to escalate a ticket to a human is notoriously difficult. Too many escalations defeat automation; too few frustrate customers.

The team needed a system that could understand intent, extract relevant information (order IDs, product names, dates), and generate coherent, helpful replies — all without a human in the loop for the majority of first-contact emails.

The Solution: Hermes + Qwen3.6-27B

Hermes is described as a lightweight orchestration layer that handles email ingestion, preprocessing, and routing. It is not a model itself but a pipeline that connects incoming emails to the language model. The core intelligence comes from Qwen3.6-27B, a large language model developed by Alibaba Cloud’s Qwen team. The 27B parameter version strikes a balance between capability and inference cost, making it suitable for production use.

The workflow is straightforward:

1. Ingestion: Hermes fetches emails from support mailbox via IMAP or API.
2. Preprocessing: The email is cleaned — headers removed, replies stripped, signatures extracted.
3. Classification: A prompt instructs Qwen3.6-27B to classify the email into categories (refund, technical issue, account inquiry, etc.).
4. Information extraction: The model extracts structured data like order numbers, email addresses, and error messages.
5. Response generation: Based on the category and extracted data, the model drafts a reply using a template or free-form generation.
6. Confidence check: If the model’s confidence (measured via log probabilities) falls below a threshold, the email is routed to a human agent.

The authors emphasize that the classification and confidence check are the critical components. Without them, the system would generate responses for emails it cannot handle, leading to poor customer experience.

Practical Implementation Details

Prompt Engineering

The team spent considerable effort on prompt design. They found that the best results came from:

• Structured prompts: The prompt includes a list of possible categories with descriptions, examples of good and bad responses, and a strict format for the output (JSON with fields for category, extracted data, and reply text).
• Few-shot examples: Each prompt includes 3–5 examples of correctly handled emails. This significantly improved accuracy for edge cases.
• System message: A system-level instruction sets the tone: "You are a polite, professional customer support agent for an e-commerce company. Never invent information. If unsure, say you need to check with a specialist."

Handling Email Threads

One of the trickiest parts was handling multi-turn conversations. The authors describe using a short-term memory mechanism: each email thread is stored in a lightweight database (Redis), and the model receives the last 3–5 messages as context. This allows the system to understand references like "as I said earlier" or "the issue persists."

Escalation Rules

The confidence threshold is not static. The team implemented dynamic thresholds:

• For simple queries (password reset, tracking information), the threshold is lower (0.7).
• For complex queries (refund disputes, technical support), the threshold is higher (0.9).
• If the model fails to extract an order number from an email that mentions a purchase, the email is automatically escalated.

Results and Metrics

The article reports that after two months of deployment, the system achieved:

• First-line resolution rate: The system handles over 40% of incoming emails without human intervention.
• Average response time: From 2 hours (human-only) to under 5 minutes (with AI).
• Customer satisfaction: Surveys showed that satisfaction with AI-handled emails was only slightly lower than with human-handled emails (4.2 vs. 4.5 out of 5).
• Human agent workload: Reduced by approximately 35%, allowing agents to focus on complex cases.

Importantly, the team also monitored for harmful responses. They found that the model occasionally hallucinated — for example, inventing a refund amount that didn’t exist. To mitigate this, they added a validation step: the generated response must include only information present in the email or the company’s FAQ database. If the model tries to create new facts, the email is escalated.

Challenges and Lessons Learned

The authors are candid about the difficulties:

• Latency: Qwen3.6-27B is not the fastest model. Each response takes 2–5 seconds on a single A100 GPU. For high-volume support, they had to scale to multiple instances.
• Cost: Running a 27B model in production is expensive. The team estimates that each email costs about $0.02 in compute. They are exploring smaller models (7B or 13B) for simpler queries.
• Language support: The model performs well in English and Chinese, but struggles with mixed-language emails common in global support. They are fine-tuning on multilingual data.
• Privacy: Customer emails often contain sensitive data. The team runs the model on-premises to avoid sending data to cloud APIs.

Is This Replicable?

The article suggests that the approach is not unique to the authors’ infrastructure. Anyone with access to a capable LLM and basic programming skills could replicate the pipeline. Open-source alternatives to Qwen3.6-27B, such as Llama 3 and Mistral, could be used with similar results. The key is not the model itself but the orchestration and prompt engineering.

For businesses considering this path, the authors recommend starting with a small pilot — for example, handling only password reset emails — and gradually expanding. They also stress the importance of monitoring and logging every interaction to catch failures early.

Conclusion

The combination of Hermes and Qwen3.6-27B demonstrates that first-line email support automation is not a distant future — it is achievable today with off-the-shelf components. The system described in the article is not perfect, but it shows a realistic path to reducing support costs while maintaining quality. As LLMs continue to improve and become cheaper, such setups will likely become standard. For now, the lesson is clear: invest in prompt engineering, confidence checks, and careful escalation logic. The model is the engine, but the pipeline is the driver.

If you are building automated support systems, consider how a similar architecture could fit your stack. And if you are using tools like Salesforce, Zendesk, or Freshdesk for ticket management, remember that ASI Biont supports connecting these platforms via API — more details on asibiont.com/courses.