In the era of information overload, searching the internet is no longer just typing a query into a browser. Modern AI agents and parsers face two fundamentally different tasks: finding relevant data (search) and extracting content from dynamic websites that load via JavaScript. For this, there are two tools — web_search for searching through search engines and browser_scrape for working with SPAs (Single Page Applications). Let's break down how they differ and how to use them.
web_search: Finding Data Through Search Engines
The web_search tool is not just sending a query to Google. It's an API interface that uses search engines (Google, Bing, Yandex) and returns structured results: titles, snippets, URLs. It's ideal for:
- Collecting news by keywords
- Monitoring brand mentions
- Finding company contacts
How it works:
1. An HTTP request is made to the search API
2. A JSON response with ranked links is received
3. The AI agent analyzes snippets and selects the needed ones
Example usage:
result = web_search.query("headless browser parsing")
for item in result['items']:
print(f"{item['title']} - {item['link']}")
browser_scrape: Extracting Content from SPAs and JS Sites
Unlike static HTML pages, modern SPAs (React, Vue, Angular) render content dynamically. A regular HTTP request will return empty HTML or a loading spinner. Here, browser_scrape comes to the rescue — a headless browser (Chromium without a GUI) that runs on the server.
Key capabilities:
- User emulation: clicks, scrolls, form filling
- Waiting for AJAX requests to load
- Extracting data after full rendering
Example scenario:
Suppose you need to collect prices from an online store built as an SPA. browser_scrape:
1. Opens the page in headless mode
2. Waits for the .price selector to appear (using waitForSelector)
3. Extracts text via page.evaluate()
4. Returns the result in JSON
Tool Comparison
| Parameter | web_search | browser_scrape |
|---|---|---|
| Data type | Text snippets | Full DOM content |
| Speed | Instant (0.5-2 sec) | Depends on rendering (2-10 sec) |
| Load | Low (API request) | High (browser emulation) |
| Application | Link search, monitoring | Data extraction from SPAs |
| JavaScript support | No | Yes (full) |
When to Choose web_search?
- Only links and brief descriptions are needed
- High speed is critical (e.g., real-time monitoring)
- The target site is static or has an HTML version
When to Choose browser_scrape?
- The site loads content via JavaScript (SPA, AJAX)
- Authentication or form interaction is required
- Data not in the search index is needed (e.g., internal pages of a personal account)
Practical Tips
- Combine tools. Use
web_searchto find pages, thenbrowser_scrapeto extract details. - Respect robots.txt. Even with headless parsing, follow politeness policy — no more than 1 request per second.
- Handle errors. SPAs can hang — set timeouts of 10-15 seconds.
- Use LSI words. In the context of parsing: crawling, scraping, data collection automation, web driver, headless browser, rendering, asynchronicity.
Conclusion
web_search and browser_scrape are two essential tools in an AI agent's arsenal. The first is for quick reconnaissance, the second for deep extraction. Mastering them will allow you to automate data collection from any site, from simple blogs to complex SPAs. Start small: set up news search by a key topic, then try extracting content from a dynamic page. Good luck with automation!
Comments