How an AI Agent Searches the Internet: web_search and Playwright for Parsing JS Sites

Introduction

Modern AI agents, such as GPT-4 with tools, Claude, or specialized search bots, face a problem: the internet is not only static HTML but also dynamic JavaScript. Conventional search methods (web_search) return raw text results but cannot "see" content loaded by scripts. This is where Playwright comes to the rescue—a browser automation tool that can work with a headless browser. In this article, we will explore how an AI agent combines web_search and Playwright for comprehensive data parsing.

How Does web_search Work in AI Agents?

Internet search for AI is typically implemented through search engine APIs (Google, Bing) or open libraries. The algorithm is simple:

1. The agent sends a query to the search engine.
2. It receives a list of URLs and snippets (brief descriptions).
3. It extracts text from the found pages (often via HTTP requests).

Problem: Many modern websites (e.g., online stores, portfolios, web applications) load content via JavaScript. Without executing scripts, the HTML code is empty or contains only a page skeleton. A regular HTTP client (via requests or fetch) cannot retrieve the actual content.

Playwright: A Headless Browser for Parsing

Playwright is a library for controlling browsers (Chromium, Firefox, WebKit) in automated mode. It allows you to:

• Load a page with full JavaScript rendering.
• Click buttons, fill forms, scroll.
• Wait for dynamic elements to appear.
• Extract the DOM tree after all asynchronous loads.

Example Code in Python:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto('https://example.com/dynamic-site')
    # Wait for content to load
    page.wait_for_selector('.product-title')
    text = page.inner_text('.product-title')
    print(text)
    browser.close()

This code opens a headless browser, loads the page, waits for an element with the class product-title to appear, and extracts its text. Without Playwright, you would get an empty string.

Combined Approach: web_search + Playwright

An effective AI agent should first find relevant pages (via web_search) and then parse their content using Playwright. The algorithm:

1. Search: The agent sends a query (e.g., "buy ASUS laptop price 2025").
2. Result Analysis: The most suitable URLs are selected from the snippets.
3. Parsing: For each URL, a Playwright headless browser is launched.
4. Data Extraction: The agent parses prices, specifications, reviews.
5. Aggregation: Results are compiled into a table or report.

Example Architecture:

AI Agent
    |
    ├── web_search (Google API)
    |       → list of URLs
    |
    └── Playwright (headless Chrome)
            → load JS site
            → extract data

Practical Example: Parsing an Online Store

Suppose an AI agent needs to collect prices for iPhone 15 from a site that uses React. A regular requests.get() would return only an empty div with id="root". Playwright, however, executes scripts and displays the products.

Code for Extracting Prices:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto('https://shop.example.com/iphone')
    page.wait_for_selector('.price')
    prices = page.locator('.price').all_inner_texts()
    print(prices)  # ['$899', '$1099', ...]
    browser.close()

Advantages and Limitations

Pros:
- Access to dynamic content.
- Ability to interact (buttons, infinite scroll).
- Unified interface for different browsers.

Cons:
- Consumes more resources (memory, CPU).
- Slower than HTTP requests (due to browser launch).
- Sites may block headless browsers (via User-Agent or WebDriver detection).

Conclusion

AI ag