How an AI Agent Searches the Internet: web_search and Playwright for Parsing JS Websites

How an AI Agent Searches the Internet: web_search and Playwright for Parsing JS Websites

In 2026, AI agents have become indispensable assistants in data collection: from price monitoring to news analysis. But how exactly does artificial intelligence find information in real time? The answer lies in a combination of two tools: web_search (e.g., via search engine APIs or libraries like SerpAPI) and Playwright — a powerful library for managing headless browsers. In this article, we'll break down how AI uses these technologies to parse dynamic websites where content is generated by JavaScript.

Why Can't AI Just "Google"?

Traditional search methods (e.g., requests + BeautifulSoup) are often powerless against modern websites. Many resources load content via JS requests, use bot protection, or dynamic URLs. An AI agent solves this problem in two steps:
1. Web_search — finds relevant links (via Google, Bing APIs, or open search engines).
2. Playwright — emulates user actions in a headless browser: scrolls pages, clicks "Load More" buttons, collects data from dynamic elements.

For example, parsing an online store built on React or Vue.js is impossible without Playwright — a simple GET request would return empty HTML.

Step 1: web_search — Finding Relevant Sources

The AI agent starts with a search. For this, it uses:
- Search engine APIs (Google Custom Search, Bing Web Search);
- Specialized tools (SerpAPI, ScrapingBee);
- Open search engines (DuckDuckGo without API, but via emulation).

Example query in Python using the googlesearch-python library:

from googlesearch import search

query = "price iPhone 15 Pro Max 2026"
for url in search(query, num_results=5):
    print(url)

This step returns a list of URLs, which the AI filters by relevance (e.g., ignoring ad links).

Step 2: Playwright — Parsing Dynamic Content

Playwright by Microsoft is a library for managing Chromium, Firefox, and WebKit. It launches a headless browser that executes JavaScript and renders the DOM tree. For AI, this is an ideal tool because:
- Emulates a user (cookies, User-Agent, geolocation);
- Works with SPAs (Single Page Applications);
- Supports screenshots and PDFs (for subsequent analysis).

Example of parsing news headlines:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example-news-site.com")
    headlines = page.locator("h2.news-title").all_text_contents()
    print(headlines)
    browser.close()

How an AI Agent Combines web_search and Playwright?

Consider a real scenario: AI collects flight prices for a travel service.
1. Web_search: Search query "flights Moscow-Tokyo June 2026" → gets 10 links to aggregators.
2. Playwright: Opens each link, waits for the JS price widget to load, extracts data into JSON.
3. Processing: AI compares results, finds the minimum price, and sends it to the user.

This approach saves hours of manual data collection and is suitable for competitor monitoring, review analysis, or trend tracking.

Tips for Optimizing Parsing

• Avoid blocks: Use proxies and random User-Agents.
• Cache results: Save page HTML to a local database (e.g., SQLite).
• Limit requests: Playwright allows setting page.wait_for_timeout() between actions.
• Validate data: AI can check that collected data is correct.