web_search and browser_scrape: How AI Extracts Data in the SPA Era
The modern internet is not just HTML pages. Most websites today are built on JavaScript (SPA — Single Page Applications), where content loads dynamically, and standard HTTP request parsing is useless. For AI agents collecting data, this is a real challenge. But there is a solution: two tools — web_search for finding information through search engines and browser_scrape for extracting content from JS sites via headless Chromium on the server. Let's break down how they work and when to use which.
What is web_search and why is it needed?
web_search is an AI tool that automates internet searches. Instead of manually Googling and analyzing results, the neural network itself forms queries, follows links, and collects relevant data. This is ideal for:
- Gathering news on a topic
- Monitoring product prices
- Analyzing reviews and ratings
- Finding contact information
Example: An AI agent searches for the latest articles on machine learning. It sends a query "ML 2026 trends," gets 10 links, parses them, and returns a brief summary. This is faster than manually flipping through pages.
| Tool | Data Type | Example Use Case |
|---|---|---|
| web_search | Text search results | News gathering, monitoring |
| browser_scrape | Dynamic SPA content | Parsing React/Vue sites |
The SPA Problem: Why Standard Parsing Doesn't Work
SPA sites (e.g., Trello, Airbnb, many modern landing pages) load content via JavaScript after the initial HTML load. If you try to parse such a site with a simple HTTP request, you'll get an empty shell. You need a full browser engine that executes JS code. This is where browser_scrape comes into play.
How browser_scrape Works with headless Chromium
browser_scrape uses headless mode of Chromium — a browser without a graphical interface that runs on the server. It:
- Loads the entire page (including JS)
- Waits for dynamic content to load
- Extracts needed elements (text, images, data)
Example: You want to parse a list of courses on an SPA platform. headless Chromium opens the page, executes scripts, and after 2-3 seconds you get the full HTML. This is indispensable for parsing modern web applications.
Practical Tips for AI Search
- Choose the right tool: For static sites (blogs, Wikipedia), web_search is sufficient. For SPA (online stores, SaaS), use browser_scrape.
- Set delays: The headless browser can be blocked for fast requests. Add pauses of 1-2 seconds.
- Use proxies: To avoid blocks during mass data collection.
- Check robots.txt: Respect the site's policy.
Conclusion
web_search and browser_scrape are two powerful tools for AI agents working with internet data. The first is for quick searches through search engines, the second for deep parsing of JS sites. Together, they cover 90% of information gathering tasks. If you want to learn how to use them in practice, check out the free courses on ASI Biont — there are modules on AI agents and web scraping. Start now, without restrictions or payment!
Comments