Gesture-Driven Web Design Without Figma: How to Stay an Author When Your Hands Are AI

Imagine designing a website not by dragging pixels on a screen, but by waving your hand in the air. It sounds like science fiction, but as of July 2026, it is a working reality. A team of developers recently shared a detailed case study on Habr, describing how they built a gesture-controlled website design tool that completely bypasses traditional editors like Figma or Sketch. The project raises a profound question: in an era where AI and gesture recognition are taking over creative tools, how does a designer remain the true author of their work?

The article, published on Habr, documents the creation of a web-based design environment where users control layout, typography, and color through hand gestures captured by a standard webcam. No mouse, no keyboard, no Figma — just your hands and a browser. The key insight? The system does not generate designs autonomously. Instead, it interprets the user’s physical movements as direct commands, preserving the designer’s intent at every step. This is a deliberate philosophical choice: the tool is a medium, not a creator.

The Core Innovation: Gesture as a Design Language

The developers behind this project faced a fundamental challenge: how to map complex design actions — like resizing a div, adjusting a gradient, or aligning elements — onto a limited set of hand gestures. Their solution was to create a layered gesture vocabulary. Simple actions, like selecting an element, use a pinch gesture. More complex actions, like adjusting opacity, rely on a combination of hand distance and rotation. The system uses a standard webcam and a lightweight machine learning model (MediaPipe Hands) to track 21 key points on each hand in real time.

What sets this apart from other gesture-based interfaces is the feedback loop. The interface provides immediate visual and haptic-like cues: elements highlight when your hand approaches, a subtle shadow follows your palm, and the cursor transforms into an animated hand icon. This reduces the learning curve significantly. According to the article, test users with no prior experience could create a basic landing page in under 15 minutes — a speed that rivals traditional mouse-based workflows.

Why Not Figma? The Case for Radical Automation

Figma is the industry standard for UI design, but it is inherently a tool for the seated, screen-bound professional. The Habr article argues that this creates a barrier for people with physical disabilities, as well as for designers who want to work in more dynamic environments — like standing at a whiteboard or presenting to a client. The gesture-based tool is not trying to replace Figma for complex, multi-layer designs. Instead, it targets rapid prototyping, brainstorming sessions, and accessibility use cases.

One surprising finding from the case study: gesture-based design actually improved collaboration. When two people stood in front of the same screen, they could take turns controlling the interface with their hands, while the other person provided verbal feedback. This is something Figma’s co-editing still struggles with, as it requires every participant to have their own device and account. The gesture tool turned design into a shared, physical experience.

The AI Question: Who Is the Author?

The most provocative section of the article deals with authorship. In the age of generative AI tools like DALL-E and Midjourney, designers often feel like they are just prompt engineers. The gesture-based tool takes the opposite approach: the AI does not generate anything on its own. It only recognizes gestures and translates them into design commands. Every pixel on the screen is directly controlled by a human hand. This means the designer can claim full authorship of the final product — no AI wrote the code, no algorithm chose the colors. It is an antidote to the creeping feeling that our creative tools are becoming too autonomous.

The article quotes one of the developers (paraphrased): "The hand is the oldest design tool. We are just giving it a digital voice." This philosophy resonates with a growing movement in the UX community that pushes back against fully automated design. The tool is open-source, and the team invites others to fork it and add their own gesture sets.

Technical Implementation: From Webcam to Web Design

For those interested in the technical side, the article provides a detailed breakdown. The system architecture consists of three layers:

1. Gesture Recognition Layer: Uses MediaPipe Hands to extract hand landmarks. Runs entirely in the browser via TensorFlow.js, so no server-side processing is needed. Latency is under 50ms.
2. Gesture-to-Command Mapping: A custom state machine that interprets sequences of gestures. For example, a closed fist followed by a clockwise rotation triggers a “group selected elements” command.
3. Rendering Engine: A lightweight DOM manipulator built with vanilla JavaScript and CSS Grid. The tool outputs standard HTML/CSS code that can be exported directly.

The developers encountered several challenges. The biggest was gesture fatigue — users’ arms got tired after 20 minutes of continuous use. Their solution was to introduce “gesture shortcuts” for repetitive actions (e.g., a double tap with the index finger to undo). Another challenge was distinguishing intentional gestures from casual hand movements. They solved this by requiring a “calibration” phase at the start of each session, where the user holds a specific pose for three seconds.

Real-World Applications and Limitations

The tool is not ready for production-level design work. It lacks advanced features like vector editing, component libraries, or responsive breakpoints. But the article describes several real-world use cases where it shines:

• Accessibility: A designer with repetitive strain injury (RSI) used the tool to create wireframes for six months without pain.
• Education: Design teachers used it in classrooms to demonstrate layout principles, with students taking turns at the screen.
• Brainstorming: Teams used it during sprint planning to quickly sketch UI concepts without needing to set up a digital whiteboard.

The limitations are clear: the tool requires good lighting, a decent webcam, and a willingness to learn a new interaction paradigm. It is not a replacement for Figma, but a complement for specific scenarios.

Comparison with Other Gesture Design Tools

The Future: Gesture as a Standard Input

This project is part of a larger trend. In 2025 and 2026, several major tech companies have released gesture SDKs for web browsers. Apple’s Vision Pro popularized hand tracking, and Meta’s Quest headsets have made gesture input a common expectation. But this tool is unique because it works on any device with a webcam — no VR headset required. The developers are now working on adding support for multiple users simultaneously, which would allow co-located teams to design together in real time.

Conclusion: The Hand Is Mightier Than the Prompt

The gesture-based design tool described in the Habr article is not just a technical curiosity. It is a philosophical statement about the role of the human in the design process. In an industry obsessed with automation, this project reminds us that the most powerful design tool is still the human hand. By removing the mouse and the screen-based UI, it forces designers to be more intentional, more physical, and more present in their work. The result is a design process that feels more like sculpture than software.

For anyone who has ever felt like a passenger in their own creative tools, this project offers a glimpse of a different future — one where the designer is always the driver, and the AI is just the steering wheel.

Source