Designing a Gesture-Controlled Website Without Figma: Staying the Author When AI Is Your Hands

Introduction

Imagine building a website where users navigate by waving their hands, swiping in the air, or tapping invisible buttons—all without a single mouse click or touchscreen tap. It sounds like science fiction, but a recent project by a team of independent developers has turned this vision into a practical reality. What makes this story even more remarkable is that they designed the entire interface without using Figma, Sketch, or any traditional visual design tool. Instead, they relied on AI to translate their ideas directly into code, effectively making the AI their hands while they remained the creative authors.

This article explores their journey, the challenges they faced, the tools they used, and the lessons learned. It’s a case study in how designers can leverage AI to prototype and build gesture-based interfaces, even if they don’t have access to conventional design software. Whether you’re a seasoned UX designer, a developer curious about gesture control, or a founder exploring new interaction paradigms, this story offers actionable insights and a fresh perspective on the future of web design.

The Challenge: Designing Without a Traditional Canvas

The developers set out to create a website that responds to hand gestures captured by a standard webcam. The goal was to allow users to scroll, click, zoom, and navigate through content by moving their hands in front of the camera. This is not a new concept—gesture recognition has been around for years—but building a polished, responsive, and accessible web interface from scratch posed unique constraints.

First, the team did not have access to Figma or any visual design tool. They couldn’t create pixel-perfect mockups or interactive prototypes in the traditional sense. Second, they wanted to avoid proprietary hardware like Leap Motion or Microsoft Kinect, opting instead for a software-only solution that works with any laptop webcam. Third, they had to ensure the interface was intuitive enough for first-time users, many of whom have never interacted with a gesture-controlled website.

The material published on Habr describes how the team approached this problem by treating AI as both a design assistant and a coding partner. Instead of designing visually, they described their intended interactions in natural language, letting a large language model generate the underlying HTML, CSS, and JavaScript. This turned the design process into a dialogue: they would say "make a button that appears when the user holds their hand still for two seconds," and the AI would produce the code.

The Solution: AI-Assisted Gesture Recognition and Interface Generation

The core of the project is a gesture recognition engine built using TensorFlow.js and the MediaPipe Hands model. This open-source library detects 21 key points on each hand in real time, allowing the system to interpret gestures like pointing, swiping, pinching, and waving. The developers integrated this engine with a custom event system that maps specific hand poses to browser actions (e.g., a pinch gesture triggers a click event, a swipe scrolls the page).

To design the interface without Figma, the team used a combination of:
- Natural language prompts to ChatGPT-4o and Claude 3.5 Sonnet, describing the layout, colors, and behavior they wanted.
- Iterative code generation, where the AI would produce a web page snippet, and the developers would test it with real hand gestures, then refine the prompt.
- A live preview environment that allowed them to see changes immediately without reloading the browser.

The result is a single-page application that displays a gallery of images, text blocks, and navigation controls, all operable via hand gestures. For example, holding up one finger triggers a tooltip; making a fist closes a modal; and spreading all five fingers returns to the home screen.

Real-World Cases and Practical Examples

Case 1: Navigating a Photo Gallery

One of the early prototypes was a photo gallery where users could swipe left or right to browse images. The developers found that users quickly adapted to the gesture, but struggled with the sensitivity of the swipe detection. After several iterations, they implemented a "dead zone"—a small area in the center of the screen where hand movement does not trigger any action—to prevent accidental swipes. This simple adjustment reduced misclicks by 40% in user testing.

Case 2: Accessible Checkout Flow

Another use case was a mock e-commerce checkout page. The team designed a three-step form where users confirm their cart, enter shipping details, and place the order. Instead of using a keyboard, users selected fields by pointing at them and typed using an on-screen virtual keyboard controlled by finger movements. While typing was slower than a physical keyboard, users with motor impairments reported that the gesture interface felt more comfortable for short inputs.

Case 3: Interactive Presentation

The developers also built a presentation mode where speakers can advance slides, zoom into diagrams, and highlight text using hand gestures. This is particularly useful for virtual meetings or public talks where the speaker wants to remain hands-free. The AI helped generate slide templates that adapt to gesture input, such as a "zoom" gesture that enlarges a selected region of the slide.

Results and Key Metrics

After three months of development, the team published an open-source demo of the gesture-controlled website. According to their own benchmarks:
- Gesture recognition accuracy reached 92% under good lighting conditions (with a well-lit room and a 720p webcam).
- Average response time between a gesture and the corresponding UI action was under 200 milliseconds, which feels instantaneous to most users.
- User onboarding time (the time it takes a new user to successfully perform their first intentional gesture) averaged 45 seconds, compared to 2-3 minutes for similar projects that used traditional design tools.

The developers attribute the fast onboarding to the AI-assisted design process, which allowed them to test and refine the UI in real time. Because they could rapidly generate alternative layouts and interaction patterns, they converged on an intuitive design much faster than if they had used a visual tool like Figma.

Tools and Technologies Used

The project relied on several key technologies, all available and actively maintained as of 2026:

The authors note that they did not use any visual design software—all UI decisions were made by describing the desired outcome to the AI, which then produced the code. This approach is a notable departure from traditional design workflows.

Lessons Learned: How to Stay the Author When AI Is Your Hands

The most important takeaway from this project is that AI does not replace the designer—it amplifies their ability to execute. The developers had a clear vision of what they wanted to build, but they lacked the time and tools to create pixel-perfect mockups. By treating AI as a coding assistant, they could focus on the creative and strategic aspects of design: defining user flows, deciding which gestures to support, and ensuring accessibility.

Here are the key lessons from their experience:

1. Describe, don’t draw. Instead of sketching wireframes, write detailed descriptions of the interface’s behavior. For example, “a navigation bar that appears when the user raises their hand above their head” is a specification that an AI can turn into code.

2. Iterate quickly. The AI can generate dozens of variations of a component in seconds. Test each one with real gesture input, and use the results to refine your prompts.

3. Respect physical constraints. Gesture interfaces are not a replacement for traditional input methods. They work best for short, discrete actions (swiping, clicking) and less well for precision tasks (typing long text). Design accordingly.

4. Plan for fallbacks. Not every user will have a webcam, and not every webcam will work well in all lighting conditions. Always provide keyboard and mouse alternatives.

5. Document your prompts. The team found that keeping a log of successful prompts helped them reproduce good results and avoid repeating mistakes. This effectively becomes a design system in natural language.

Challenges and Limitations

No project is without its difficulties. The developers encountered several hurdles:

• Lighting sensitivity. The MediaPipe model degrades significantly in low light or with backlighting. They added a visual indicator that warns users when lighting is insufficient.
• Gesture fatigue. Users reported that holding their hand up for extended periods caused fatigue. The team introduced “rest” gestures (e.g., lowering the hand) to pause interaction.
• Browser compatibility. While MediaPipe runs in Chrome, Firefox, and Edge, Safari has limited support for WebGL, which the model requires. They recommended Chrome for the best experience.
• AI hallucinations. Occasionally, the AI would generate code that didn’t work correctly with the gesture engine. The developers had to manually debug or re-prompt.

Despite these challenges, the team considers the project a success and has made the code available on GitHub under an MIT license.

The Bigger Picture: What This Means for Web Design

This case study is part of a larger trend: the democratization of interface design through AI. When anyone can describe a website in plain language and have an AI build it, the role of the designer shifts from pixel-pusher to experience architect. The ability to create gesture-controlled interfaces without Figma shows that the barriers to entry are lower than ever.

However, this does not mean that visual design tools are obsolete. Figma remains the industry standard for collaborative design, and many teams will continue to use it for complex projects. What changes is the speed of prototyping: instead of spending days creating mockups, designers can now generate functional code in hours.

For founders and product managers, this means you can test gesture-based interactions with real users before investing in a full development cycle. For developers, it means you can build sophisticated frontends without deep design skills. And for users, it means more accessible, intuitive interfaces that adapt to natural human movement.

Conclusion

The story of building a gesture-controlled website without Figma is a testament to the power of AI as a creative partner. The developers proved that you don’t need expensive tools or years of design training to craft innovative user experiences. By describing their vision in words, iterating with AI, and testing with real gestures, they created a functional, responsive interface that works on any laptop with a webcam.

Their approach offers a blueprint for anyone who wants to explore gesture-based interactions: start with a clear idea, use AI to generate the code, test relentlessly, and always keep the user’s physical comfort in mind. The result is not just a website—it’s a glimpse into a future where our hands are the only input device we need.

If you’re interested in implementing gesture control for your own projects, consider exploring the tools mentioned above. And if you’re looking for a platform that integrates AI-powered design with practical business applications, ASI Biont supports connecting to services like OpenAI and TensorFlow.js through API—details available on asibiont.com/courses.

This article is based on a detailed case study published by independent developers. You can read the original source material here: Source.