Launch HN: Coasty (YC S26) – An API for Computer-Use Agents

Launch HN: Coasty (YC S26) – An API for Computer-Use Agents

In the rapidly evolving landscape of artificial intelligence, a significant shift is underway: AI agents are no longer confined to text-based interactions. They are now learning to interact with the digital world as humans do—by clicking, typing, and navigating graphical user interfaces (GUIs). The latest milestone in this domain is the launch of Coasty (YC S26), an API designed specifically for building computer-use agents. This article explores what Coasty offers, the technical challenges it addresses, and the implications for developers and businesses.

What Is Coasty?

Coasty is a new API platform from Y Combinator's S26 batch that enables developers to create AI agents capable of performing actions on a computer screen. Instead of relying on traditional automation scripts or browser extensions, Coasty provides a unified interface for agents to observe screen states, interpret elements, and execute actions such as clicking buttons, filling forms, and navigating menus. The core idea is to give AI models a "digital hand" to interact with any software application, from legacy enterprise systems to modern web apps.

According to the official documentation at Coasty's website, the API is built around three primary functions: screenshot, click, and type. Developers can chain these actions with AI reasoning to create autonomous workflows. For example, an agent might take a screenshot of a CRM dashboard, identify a lead's profile, click the "Send Email" button, and type a personalized message—all without human intervention.

The Problem: Why Computer-Use Agents Matter

Traditional automation methods have significant limitations. Robotic Process Automation (RPA) tools, while powerful, require manual setup and are brittle—any change in the UI can break the workflow. Similarly, APIs are not always available for legacy or third-party applications. Many businesses still rely on software that lacks modern API support, making automation difficult.

Computer-use agents solve this by mimicking human behavior. They can adapt to UI changes because they "see" the screen and interpret it in real time. This opens up automation possibilities for tasks that were previously impossible or too costly to automate, such as data entry in old accounting software, testing user interfaces across multiple platforms, or automating customer support workflows.

How Coasty Works

The Coasty API is straightforward. Developers send a request to capture the current screen state, and the API returns a structured representation of visible elements (buttons, text fields, icons). The developer's AI model then decides which action to take next—clicking a button or typing text. The API executes the action and returns the new screen state, creating a loop of perception and action.

A key feature is the semantic understanding of screen elements. Coasty doesn't just return raw pixel coordinates; it provides labels and metadata that help AI models make informed decisions. For instance, the API can identify a button labeled "Submit" even if its CSS class changes. This abstraction layer reduces the complexity of building reliable agents.

Practical Use Cases

1. Automated Data Entry

A medium-sized logistics company uses Coasty to automate data transfer between its legacy ERP system and a modern analytics dashboard. Previously, employees spent hours manually copying shipment statuses. With Coasty, an agent logs into the ERP, navigates through menus, extracts data from tables, and inputs it into the new system. The process is now fully automated, reducing errors and freeing up staff for higher-value tasks.

2. UI Testing for SaaS Products

A startup developing a project management tool needed to test its interface across different browsers and devices. Instead of writing complex Selenium scripts, the team used Coasty to create agents that simulate user journeys—signing up, creating projects, and assigning tasks. The agents adapt to UI changes automatically, making maintenance significantly easier.

3. Customer Support Automation

A tech support company integrated Coasty with its ticketing system. When a customer reports a bug, an agent logs into the application, recreates the steps described, and captures the error screen. This reduces the time support agents spend on diagnostics and provides developers with reproducible bug reports.

Technical Considerations

Building computer-use agents is not trivial. The developers at Coasty encountered several challenges during development. One major issue is latency—screenshots and actions take time, especially over remote connections. Coasty optimizes by caching screen states and batching actions where possible. Another challenge is security: agents that can click anywhere on a screen pose a risk if not properly sandboxed. The platform includes permission controls and action logs to ensure safe operation.

Another consideration is the AI model itself. Coasty provides the API for actions, but the reasoning layer—deciding what to do next—is left to the developer. Many teams use large language models (LLMs) like GPT-4 or Claude for this purpose, though any model capable of processing visual and textual input can be used. The documentation suggests that models fine-tuned on UI navigation tasks yield the best results.

Comparison with Existing Solutions

Several companies have explored computer-use agents. For example, Anthropic's Claude can use a computer through its "computer use" beta, and OpenAI's GPT-4 Vision can analyze screenshots. However, Coasty differentiates itself by providing a dedicated API for executing actions, rather than just interpreting them. This makes it easier to integrate into existing software stacks without building custom automation pipelines.

The Future of Automation

Coasty represents a broader trend: the convergence of AI vision and action. As models become better at understanding screens, the barrier to automation will continue to drop. In the near future, we may see agents that can navigate entire operating systems, manage software installations, or even assist with complex data analysis tasks.

For developers, the message is clear: now is the time to experiment with computer-use agents. Whether you are building internal tools, enhancing customer experiences, or creating new products, platforms like Coasty provide the building blocks. The API is currently available with a free tier for early adopters, and the documentation offers detailed examples to get started.

Conclusion

The launch of Coasty (YC S26) marks a practical step toward making computer-use agents accessible to a wider audience. By providing a clean API for screen interaction, the project lowers the entry barrier for developers interested in AI-driven automation. While challenges remain—particularly around reliability and security—the potential applications are vast. As the ecosystem matures, computer-use agents could become as common as chatbots in the automation toolkit.

For those interested in exploring further, the full documentation and API reference are available at Coasty's official site. The future of automation is not just about processing data—it's about interacting with the digital world as humans do. Coasty offers a glimpse into that future.