How an AI Agent Searches the Internet: web_search and Playwright for Parsing JS Websites

## How an AI Agent Searches the Internet: web_search and Playwright for Parsing JS Websites In 2026, AI agents have become indispensable assistants in data collection: from price monitoring to news analysis. But how exactly does artificial intelligence find information in real time? The answer lies in a combination of two tools: **web_search** (e.g., via search engine APIs or libraries like SerpAPI) and **Playwright** — a powerful library for managing headless browsers. In this article, we'll break down how AI uses these technologies to parse dynamic websites where content is generated by JavaScript. ### Why Can't AI Just "Google"? Traditional search methods (e.g., requests + BeautifulSoup) are often powerless against modern websites. Many resources load content via JS requests, use bot protection, or dynamic URLs. An AI agent solves this problem in two steps: 1. **Web_search** — finds relevant links (via Google, Bing APIs, or open search engines). 2. **Playwright** — emulates user actions in a headless browser: scrolls pages, clicks "Load More" buttons, collects data from dynamic elements. For example, parsing an online store built on React or Vue.js is impossible without Playwright — a simple GET request would return empty HTML. ### Step 1: web_search — Finding Relevant Sources The AI agent starts with a search. For this, it uses: - **Search engine APIs** (Google Custom Search, Bing Web Search); - **Specialized tools** (SerpAPI, ScrapingBee); - **Open search engines** (DuckDuckGo without API, but via emulation). Example query in Python using the `googlesearch-python` library: ```python from googlesearch import search query = "price iPhone 15 Pro Max 2026" for url in search(query, num_results=5): print(url) ``` This step returns a list of URLs, which the AI filters by relevance (e.g., ignoring ad links). | Tool | Pros | Cons | |------------|-------|--------| | Google API | High accuracy, up to 100 requests/day free | Paid subscription for large volumes | | SerpAPI | Support for 30+ countries, parsing Google results | Cost: from $50/month | | DuckDuckGo | Free, no limits | Less data, may block IPs | ### Step 2: Playwright — Parsing Dynamic Content Playwright by Microsoft is a library for managing Chromium, Firefox, and WebKit. It launches a headless browser that executes JavaScript and renders the DOM tree. For AI, this is an ideal tool because: - **Emulates a user** (cookies, User-Agent, geolocation); - **Works with SPAs** (Single Page Applications); - **Supports screenshots and PDFs** (for subsequent analysis). Example of parsing news headlines: ```python from playwright.sync_api import sync_playwright with sync_playwright() as p: browser = p.chromium.launch(headless=True) page = browser.new_page() page.goto("https://example-news-site.com") headlines = page.locator("h2.news-title").all_text_contents() print(headlines) browser.close() ``` ### How an AI Agent Combines web_search and Playwright? Consider a real scenario: AI collects flight prices for a travel service. 1. **Web_search**: Search query "flights Moscow-Tokyo June 2026" → gets 10 links to aggregators. 2. **Playwright**: Opens each link, waits for the JS price widget to load, extracts data into JSON. 3. **Processing**: AI compares results, finds the minimum price, and sends it to the user. This approach saves hours of manual data collection and is suitable for competitor monitoring, review analysis, or trend tracking. ### Tips for Optimizing Parsing - **Avoid blocks**: Use proxies and random User-Agents. - **Cache results**: Save page HTML to a local database (e.g., SQLite). - **Limit requests**: Playwright allows setting `page.wait_for_timeout()` between actions. - **Validate data**: AI can check that collected data is correct.

How an AI Agent Searches the Internet: web_search and Playwright for Parsing JS Websites

Comments

Recent articles

Eternal Memory for AI: How an Agent Remembers Everything Between Sessions Using a Vector Database

Execute Python: How AI Runs Code in a Sandbox and Ensures Security

AI Targeting Specialist: Creative Generation, Audience Analysis, and Auto-Launching Ads — How Artificial Intelligence Replaces an Entire Marketing Department

Multimodal AI Agents: How Image, Video, and Audio Analysis Is Transforming Business in 2026

How to Connect Your AI Model in 2026: OpenRouter, Custom APIs, and Proxies — A Complete Guide

AI Agent as Micro SaaS: A Ready-Made Business on a Single Agent

AI Agent for Email Marketing: How to Automate Newsletters and Emails via Gmail in 2026

Vibe Coding for Mobile Apps: React Native and Flutter via AI

How an AI Agent Searches the Internet: web_search and Playwright for Parsing JS Sites