web_search and browser_scrape: How AI Extracts Data from JS Sites and SPAs

In the era of information overload, searching the internet is no longer just typing a query into a browser. Modern AI agents and parsers face two fundamentally different tasks: finding relevant data (search) and extracting content from dynamic websites that load via JavaScript. For this, there are two tools — **web_search** for searching through search engines and **browser_scrape** for working with SPAs (Single Page Applications). Let's break down how they differ and how to use them. ## web_search: Finding Data Through Search Engines The **web_search** tool is not just sending a query to Google. It's an API interface that uses search engines (Google, Bing, Yandex) and returns structured results: titles, snippets, URLs. It's ideal for: - Collecting news by keywords - Monitoring brand mentions - Finding company contacts **How it works:** 1. An HTTP request is made to the search API 2. A JSON response with ranked links is received 3. The AI agent analyzes snippets and selects the needed ones Example usage: ```python result = web_search.query("headless browser parsing") for item in result['items']: print(f"{item['title']} - {item['link']}") ``` ## browser_scrape: Extracting Content from SPAs and JS Sites Unlike static HTML pages, modern SPAs (React, Vue, Angular) render content dynamically. A regular HTTP request will return empty HTML or a loading spinner. Here, **browser_scrape** comes to the rescue — a headless browser (Chromium without a GUI) that runs on the server. **Key capabilities:** - User emulation: clicks, scrolls, form filling - Waiting for AJAX requests to load - Extracting data after full rendering **Example scenario:** Suppose you need to collect prices from an online store built as an SPA. `browser_scrape`: 1. Opens the page in headless mode 2. Waits for the `.price` selector to appear (using `waitForSelector`) 3. Extracts text via `page.evaluate()` 4. Returns the result in JSON ## Tool Comparison | Parameter | web_search | browser_scrape | |-----------|------------|----------------| | Data type | Text snippets | Full DOM content | | Speed | Instant (0.5-2 sec) | Depends on rendering (2-10 sec) | | Load | Low (API request) | High (browser emulation) | | Application | Link search, monitoring | Data extraction from SPAs | | JavaScript support | No | Yes (full) | ## When to Choose web_search? - Only links and brief descriptions are needed - High speed is critical (e.g., real-time monitoring) - The target site is static or has an HTML version ## When to Choose browser_scrape? - The site loads content via JavaScript (SPA, AJAX) - Authentication or form interaction is required - Data not in the search index is needed (e.g., internal pages of a personal account) ## Practical Tips 1. **Combine tools.** Use `web_search` to find pages, then `browser_scrape` to extract details. 2. **Respect robots.txt.** Even with headless parsing, follow politeness policy — no more than 1 request per second. 3. **Handle errors.** SPAs can hang — set timeouts of 10-15 seconds. 4. **Use LSI words.** In the context of parsing: crawling, scraping, data collection automation, web driver, headless browser, rendering, asynchronicity. ## Conclusion **web_search** and **browser_scrape** are two essential tools in an AI agent's arsenal. The first is for quick reconnaissance, the second for deep extraction. Mastering them will allow you to automate data collection from any site, from simple blogs to complex SPAs. Start small: set up news search by a key topic, then try extracting content from a dynamic page. Good luck with automation!

web_search and browser_scrape: How AI Extracts Data from JS Sites and SPAs

Comments

Recent articles

MCP Servers and Tools for AI: How Model Context Protocol Is Transforming Learning with AI

AI Testing: How to Test AI Applications and Models — From Eval Metrics to CI/CD

AI for Working with Code: Generation, Refactoring, and Debugging via execute_python

Course 'Video Editing and Animation': How AI Accelerates Learning on ASI Biont

Go and Rust — Systems Programming: How AI Is Changing Learning on ASI Biont

Prompt Engineering Pro: How AI is Changing the Approach to Learning on ASI Biont

AI Agent in Telegram: How @asibiont_bot Replies to Subscribers and Automates Communication

How to Write Prompts for Vibe Coding: A Constructor for Effective Queries

SQL Mastery (Window Functions, Indexes): How AI is Changing Advanced SQL Learning