Vision Without a CV Team: Prompt-Driven Pipelines from Object Detection to Video Analytics

Vision Without a CV Team: Prompt-Driven Pipelines from Object Detection to Video Analytics

You do not need a team of CV engineers to ship a working computer vision product in 2026. What you need is a clear pipeline, honest evaluation, and a disciplined way of talking to multimodal models and coding assistants. Multimodal LLMs can now read an image, describe a defect, generate a PyTorch training script, and help you export a model to ONNX or TensorRT. The bottleneck moved from "can we train a detector?" to "can we define the task precisely enough?"

This article is a collection of prompts organized by pipeline stage: dataset preparation and augmentation, model selection and training (YOLO, Detectron2, CLIP), quality evaluation (mAP, IoU), inference optimization (ONNX, TensorRT), and cross-modal workflows for real cases — from manufacturing quality control to video stream analytics. Each prompt comes with the task it solves and a concrete example. Treat them as starting points, not magic: always verify outputs against your own data.

Stage 1 — Dataset Preparation and Augmentation

Prompt 1: Audit a folder of images before labeling

Task: Before you spend money on annotation, find duplicates, corrupt files, and class imbalance.

Prompt:

You are a data quality engineer. I have a folder of images for an object
detection dataset. Write a Python script using Pillow, imagehash, and
pandas that:
1. Scans ./raw_images recursively for .jpg/.png files.
2. Flags files that fail to open or have zero size.
3. Computes a perceptual hash (phash) and groups near-duplicates.
4. Outputs a CSV with columns: path, width, height, phash, duplicate_group.
5. Prints a summary: total images, duplicates, unreadable files.
Do not use any paid APIs. Include requirements.txt.

Example result: The script returns a CSV where 812 of 5,400 images are grouped into 96 duplicate clusters, mostly burst frames from the same camera. You delete them before annotation — annotation cost drops noticeably, and the detector stops overfitting to repeated scenes.

Prompt 2: Generate augmentation config for a specific failure mode

Task: Your model fails on low-light footage. Augment for that, not randomly.

Prompt:

I train an object detector (Ultralytics YOLO) on industrial parts photographed
under inconsistent lighting. My main failure mode is underexposed images.
Propose an Albumentations pipeline that:
- applies RandomBrightnessContrast and CLAHE with realistic ranges,
- adds mild motion blur and Gaussian noise,
- keeps bounding boxes valid (bbox_params),
- is safe for small objects (no aggressive downscale).
Return runnable Python using albumentations and explain each transform's range.

Example result: You get a working A.Compose with bbox_params=A.BboxParams(format='yolo', label_fields=['class_labels']). After retraining with this pipeline, underexposed validation images are detected far more reliably. The key lesson: augmentations should mirror real failure modes, not be a random grab bag.

Stage 2 — Model Selection and Training

Prompt 3: Choose between YOLO, Detectron2, and a foundation model

Task: Make an architecture decision with explicit trade-offs, not vibes.

Prompt:

Compare three options for my task: detecting small surface defects on metal
sheets, ~2,000 labeled images, target 30 FPS on an NVIDIA T4, on-prem deployment.
Options: (a) Ultralytics YOLO, (b) Detectron2 Faster R-CNN, (c) a zero-shot
approach with Grounding DINO or CLIP-based classification.
Return a Markdown table with columns: option, strengths, weaknesses, labeling
needs, inference cost, recommended when. Then give a recommendation and the
exact first command to start training.

Example result: The table makes the trade-off explicit: YOLO wins on latency and tooling, Detectron2 wins on small-object accuracy at higher compute cost, zero-shot wins only if labels are scarce. For 2,000 images and 30 FPS, YOLO is the pragmatic default — and you can later distill a foundation model's pseudo-labels into it.

Prompt 4: Turn a paper's idea into a training script

Task: Go from a concept to runnable code without hand-waving.

Prompt:

Write a minimal PyTorch training loop for a Faster R-CNN style detector using
torchvision.models.detection. Requirements:
- Custom Dataset class reading COCO-format annotations.
- Correct handling of variable-size targets.
- AdamW optimizer, cosine schedule, gradient clipping.
- Validation step computing loss only (no mAP here).
- Save best checkpoint by validation loss.
Add comments explaining why each part exists. Flag anything version-sensitive.

Example result: A single-file script you can run today. The assistant explicitly warns that torchvision detection models expect targets as a list of dicts with boxes and labels, which is the exact place beginners usually break the loop.

Stage 3 — Evaluation: mAP, IoU, and Honest Metrics

Prompt 5: Implement IoU and mAP from scratch to understand them

Task: Stop treating metrics as a black box.

Prompt:

Explain IoU and mAP for object detection, then implement both in NumPy:
1. iou(box_a, box_b) for xyxy boxes.
2. A function computing AP at a single IoU threshold using precision-recall
   interpolation (the 11-point or all-point method — pick one and justify).
3. mAP across IoU thresholds 0.5:0.05:0.95.
Include a tiny test with hand-computed expected values.

Example result: You get tested functions and, more importantly, intuition: IoU measures overlap, AP integrates the precision-recall curve, mAP averages across thresholds and classes. Now when a vendor quotes "99% accuracy," you know to ask for mAP@[.5:.95] and the per-class breakdown.

Prompt 6: Debug a suspiciously high metric

Task: Catch leakage and evaluation bugs before they reach production.

Prompt:

My detector reports mAP@0.5 = 0.97 on validation but performs poorly in the
field. List the 10 most likely causes of an inflated validation metric in
object detection, ordered by probability. For each, give a concrete check
I can run on my dataset and code. Include: train/val leakage, duplicate
images across splits, class imbalance in val, annotation errors, and NMS
threshold effects.

Example result: The top hit is usually duplicate or near-duplicate images spanning train and validation splits — the exact problem Prompt 1 catches. The checklist turns a vague "the model is weird" into a bug hunt.

Stage 4 — Inference Optimization: ONNX and TensorRT

Prompt 7: Export to ONNX and verify numerical parity

Task: Move from a training framework to a portable, fast runtime.

Prompt:

Show how to export my trained PyTorch detector to ONNX with a fixed input
shape, then verify that ONNX Runtime outputs match PyTorch within tolerance.
Include:
- torch.onnx.export with dynamic_axes where appropriate,
- an onnxruntime inference session,
- a numeric comparison script printing max absolute difference,
- common pitfalls: NMS inside vs outside the graph, opset version.

Example result: A reproducible export plus a parity check. Typical pitfall surfaced: post-processing (NMS) often stays outside the ONNX graph, so you must reimplement it or use a model that outputs raw boxes. The assistant warns that opset choice affects operator support.

Prompt 8: Plan a TensorRT optimization pass

Task: Squeeze latency on NVIDIA hardware safely.

Prompt:

My ONNX model runs at 18 FPS on a T4; I need 30+. Outline a TensorRT
optimization plan: FP16 vs INT8, calibration for INT8, engine building,
and how to validate accuracy after quantization. Give a decision table:
precision, expected speedup, accuracy risk, when to use. Mention that INT8
requires a calibration dataset and can hurt small-object recall.

Example result: FP16 is the low-risk first step and often closes most of the gap. INT8 gets you further but needs calibration data and careful validation — especially for small defects, where quantization error hurts most. You leave with a staged plan rather than a leap of faith.

Stage 5 — Multimodal Workflows: Text + Image + Video

Prompt 9: Zero-shot classification with CLIP

Task: Classify images without training a classifier.

Prompt:

Using open_clip or transformers CLIP, write code that classifies images into
my categories: ["scratch", "dent", "stain", "ok"] with no training.
Explain prompt engineering for CLIP: why "a photo of a scratched metal
surface" beats "scratch", and how to build a confusion matrix from zero-shot
predictions. Include batching and a confidence threshold.

Example result: A working zero-shot classifier. It is not as accurate as a fine-tuned model, but it gives you a fast baseline and can auto-label a first batch, which you then correct — a realistic way to bootstrap labels when you have zero annotated data.

Prompt 10: Describe video frames and detect events

Task: Turn raw video into structured events using a multimodal model.

Prompt:

I have a video stream from a warehouse camera. Design a pipeline that:
1. Samples frames (e.g., 1 fps) with OpenCV or ffmpeg.
2. Runs an object detector to find people and forklifts.
3. Sends selected frames to a vision-language model with the prompt:
   "Describe any safety-relevant event in one sentence: who, what, where."
4. Writes events to a JSONL log with timestamps.
Give the architecture, the frame-sampling trade-off, and cost considerations.

Example result: You get a two-tier design: cheap detection on every sampled frame, expensive vision-language reasoning only on frames that trigger a rule (e.g., a person inside a marked zone). This keeps cost sane while producing human-readable event logs that non-engineers can review.

Prompt 11: Build a human-in-the-loop review queue

Task: Use model uncertainty to route work to people efficiently.

Prompt:

Design a review workflow for a defect-detection system. The model outputs a
confidence score per detection. Propose thresholds that route items into:
auto-accept, human review, auto-reject. Explain how to tune thresholds using
a labeled validation set and how to measure the cost of false negatives versus
false positives in a manufacturing context.

Example result: A three-zone policy with thresholds derived from your own validation data, plus a clear statement that in manufacturing a missed defect (false negative) usually costs far more than a false alarm — so you bias thresholds accordingly.

Quick Reference

A few principles hold across all of them. Verify every generated script against your own data — models hallucinate APIs. Prefer honest metrics (mAP@[.5:.95], per-class) over single flattering numbers. And treat augmentation and quantization as accuracy decisions, not free wins. If you want to go deeper, the Ultralytics documentation, the Detectron2 docs, the OpenCLIP repository, and the ONNX Runtime and TensorRT guides are the primary sources worth reading. Start with one prompt, run it on your real images, and let the pipeline grow from there — the CV team can be you plus a well-asked question.