How to Train AI to See What It Has Never Seen: Zero-Shot Learning and Beyond

Introduction

Imagine an AI that can recognize a platypus without ever having seen a photo of one during training. Or a model that can identify a new type of cancer cell from a single scan, despite never being exposed to that specific anomaly before. This isn't science fiction—it's the cutting edge of computer vision research, where engineers and scientists are learning to train AI to see what it has never seen.

A recent article on Habr details a fascinating project where developers successfully taught an AI system to identify objects that were completely absent from its training dataset. This breakthrough, known as zero-shot learning, represents a paradigm shift in how we think about machine perception. Instead of requiring millions of labeled examples for every possible category, these models can generalize to novel concepts using semantic relationships and auxiliary information.

In this expert guide, we'll explore the techniques behind this achievement, provide practical code examples, and discuss real-world applications. Whether you're a data scientist, a machine learning engineer, or just an AI enthusiast, you'll learn how to implement similar approaches and understand the limitations of current technology. Let's dive into the mechanics of teaching AI to see the unseen.

The Core Problem: Why Traditional Training Fails

Traditional supervised learning for computer vision relies on massive labeled datasets—think ImageNet with its 14 million images across 20,000 categories. A model learns to map visual features to specific labels. But this approach has a critical flaw: it can only recognize what it has seen during training. If you show it a picture of a 'quokka' (a small marsupial from Australia) and the model was never trained on quokkas, it will confidently misclassify it as a 'wallaby' or 'kangaroo'.

This limitation becomes severe in dynamic environments where new categories emerge constantly—new product designs, rare wildlife, novel medical conditions. Retraining models from scratch is expensive, time-consuming, and often impractical. The Habr article highlights that the development team faced exactly this challenge: they needed a system that could adapt to new visual concepts without requiring a complete retraining cycle.

Zero-Shot Learning: The Core Concept

Zero-shot learning (ZSL) solves this by leveraging auxiliary information—typically semantic embeddings—to bridge the gap between seen and unseen classes. Instead of learning a direct mapping from images to labels, the model learns a mapping from images to a semantic space, and then compares that embedding to the semantic representations of candidate labels.

For example, if the model knows that a 'zebra' has stripes, lives in Africa, and resembles a horse, it can infer that a striped, horse-like animal in an African savanna image is likely a zebra—even if it was never trained on zebra images. The Habr project used a similar approach, combining visual features with textual descriptions extracted from Wikipedia and other sources.

Key Components of a Zero-Shot Learning System

The project described in the source article implemented a variant of ZSL called 'generalized zero-shot learning' (GZSL), where the model must handle both seen and unseen classes simultaneously—a significantly harder task.

Technical Deep Dive: How the Project Achieved This

According to the Habr article, the developers built their system using a transformer-based architecture for both visual and textual processing. Here's a breakdown of their approach:

1. Data Preparation

They started with a standard dataset of 200 common animal species (the 'seen' classes). For each species, they collected:
- 500-1000 labeled images
- A short text description from Wikipedia (2-3 sentences summarizing key features)
- A set of 85 binary attributes (e.g., 'has horns', 'striped pattern', 'lives in groups')

For the 'unseen' classes (20 novel species like the 'saiga antelope'), they only had text descriptions and attribute vectors—no images.

2. Model Architecture

The team used a dual-encoder setup:
- Vision Transformer (ViT-B/16) pretrained on ImageNet-21k as the image encoder
- DistilBERT as the text encoder for processing descriptions
- A projection head mapping both embeddings to a 512-dimensional shared space

3. Training Objective

The model was trained to maximize the cosine similarity between an image and its correct label's semantic embedding, while minimizing similarity to incorrect labels. They used a contrastive loss function similar to CLIP but with an added regularization term for attribute consistency.

4. Inference on Unseen Classes

When presented with a novel image, the system:
1. Extracts visual features using the ViT encoder
2. Projects to the shared embedding space
3. Computes cosine similarity with all candidate class embeddings (both seen and unseen)
4. Returns the top-5 most similar classes

Practical Code Example (Simplified)

import torch
import torch.nn as nn
from transformers import ViTModel, DistilBertModel

class ZeroShotModel(nn.Module):
    def __init__(self, embed_dim=512):
        super().__init__()
        self.visual_encoder = ViTModel.from_pretrained('google/vit-base-patch16-224')
        self.text_encoder = DistilBertModel.from_pretrained('distilbert-base-uncased')
        self.visual_proj = nn.Linear(768, embed_dim)
        self.text_proj = nn.Linear(768, embed_dim)

    def encode_image(self, images):
        features = self.visual_encoder(images).last_hidden_state[:, 0, :]
        return nn.functional.normalize(self.visual_proj(features), dim=-1)

    def encode_text(self, input_ids, attention_mask):
        features = self.text_encoder(input_ids, attention_mask).last_hidden_state[:, 0, :]
        return nn.functional.normalize(self.text_proj(features), dim=-1)

    def forward(self, images, input_ids, attention_mask):
        visual_embeds = self.encode_image(images)
        text_embeds = self.encode_text(input_ids, attention_mask)
        return visual_embeds, text_embeds

# Usage
model = ZeroShotModel()
image = torch.randn(1, 3, 224, 224)
text = torch.randint(0, 30522, (1, 128))
mask = torch.ones(1, 128)
img_emb, txt_emb = model(image, text, mask)
similarity = torch.cosine_similarity(img_emb, txt_emb)

Note: This is a simplified example. The actual project used more sophisticated training techniques including hard negative mining and attribute consistency loss.

Results and Performance

According to the Habr article, the final model achieved:
- Top-1 accuracy on seen classes: 87.3% (compared to 91.2% for a fully supervised baseline)
- Top-5 accuracy on unseen classes: 62.1% (random chance would be ~1% for 20 classes)
- Harmonic mean (GZSL metric): 52.4%—a strong result for generalized zero-shot learning

These numbers demonstrate that while zero-shot learning still lags behind fully supervised approaches, it offers a practical way to handle novel categories without retraining.

Real-World Applications

The techniques described in the article have immediate practical value:

1. Medical Imaging

Radiologists frequently encounter rare conditions with limited training data. A zero-shot system can flag potential anomalies based on textual descriptions from medical literature. For instance, if a new lung opacity pattern emerges, the model can reference radiology reports to identify it.

2. Wildlife Conservation

Camera traps generate millions of images of wildlife, but many species are rare and underrepresented in training datasets. Zero-shot learning allows conservationists to identify new species automatically. ASI Biont supports connecting to wildlife camera trap APIs for real-time species identification.

3. Manufacturing Quality Control

Factories constantly introduce new product variants. Instead of retraining defect detection models for each new design, manufacturers can use zero-shot learning with text descriptions of acceptable and defective features.

4. E-commerce Product Categorization

Online marketplaces add thousands of new product types daily. Zero-shot classification enables automatic tagging without manual labeling for each new category.

Challenges and Limitations

The Habr article also candidly discusses the project's failures:

• Domain shift: The model struggled when unseen classes were visually similar to seen classes (e.g., distinguishing two similar antelope species)
• Attribute ambiguity: Some attributes like 'beautiful' or 'endangered' are subjective and led to inconsistent embeddings
• Bias amplification: The model inherited gender and cultural biases from the text descriptions
• Computational cost: Generating semantic embeddings for thousands of candidate classes during inference is memory-intensive

Future Directions

The researchers outline several improvements for future work:

1. Dynamic attribute learning: Instead of predefined attributes, let the model discover relevant features from text
2. Multi-modal fusion: Incorporate audio or depth data alongside visual and text inputs
3. Active learning loop: When confidence is low, the system requests human feedback to refine its semantic understanding
4. Few-shot extension: Combine zero-shot with few-shot examples for even better performance

Conclusion

Zero-shot learning represents a major leap toward more flexible, adaptable AI systems. The project detailed in the Habr article demonstrates that it's possible to train AI to see what it has never seen—with careful architecture design, appropriate semantic embeddings, and robust training strategies. While not yet perfect, the technology is already practical for many real-world applications where retraining is costly or impossible.

For developers and organizations looking to implement similar systems, the key takeaway is that success depends on high-quality semantic descriptors and a well-designed compatibility function. The field is evolving rapidly, and we can expect even more impressive results in the coming years as models become better at reasoning about visual concepts through language.

Source