AI Agent and Internet Search: How web_search and Playwright Work for Parsing JS Sites

How an AI Agent Searches the Internet: web_search and Playwright for Parsing

Modern AI agents are no longer just chatbots. They can independently search for information on the internet, analyze it, and make decisions. But how exactly does this process work? In this article, we'll break down two key tools: web_search (internet search) and Playwright (a headless browser for parsing dynamic websites).

Introduction

When you ask an AI agent to find the latest news or compare product prices, it doesn't "know" the answer in advance. Instead, it launches a chain of actions: sends a query to a search engine, analyzes the results, then follows links and collects data. The problem is that many modern websites use JavaScript to load content. A regular HTTP request (like with requests or curl) doesn't see dynamic elements. This is where Playwright comes to the rescue—a tool that emulates the work of a real browser.

How web_search Works in AI Agents

Web_search is a module that integrates with search APIs (Google, Bing, DuckDuckGo) or uses custom parsers. In AI agents, it typically performs three steps:

1. Query Formulation — The AI translates the user's request into the search engine's language (removes stop words, adds operators).
2. Retrieving Results — The agent receives a list of URLs, titles, and snippets.
3. Filtering and Reranking — The AI evaluates the relevance of links (e.g., by publication date or domain authority).

Example of simple Python code using googlesearch-python:

from googlesearch import search

query = "AI agents data parsing 2025"
for url in search(query, num_results=5):
    print(url)

But this is just the beginning. To get the page content, parsing is needed.

Why Playwright is Better Than Regular Parsing

Traditional parsers (BeautifulSoup, Scrapy) work with the HTML code that comes from the server. But modern web applications (React, Angular, Vue) load data via APIs and render it on the client side. Without a headless browser, you'll only see an empty page skeleton. Playwright solves this problem:

• It launches a full browser (Chromium, Firefox, WebKit).
• It waits for JavaScript to load.
• It can click buttons, scroll the page, and fill out forms.

Example: Parsing Dynamic Content with Playwright

Suppose we need to collect data from a site that loads products after scrolling (infinite scroll). Here's how it looks:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)  # headless — without GUI
    page = browser.new_page()
    page.goto("https://example.com/catalog")

    # Scroll to the bottom of the page
    for _ in range(5):
        page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
        page.wait_for_timeout(2000)  # wait for loading

    # Extract text
    items = page.query_selector_all(".product-title")
    for item in items:
        print(item.inner_text())

    browser.close()

This code mimics user behavior: scrolls, waits, collects data. Playwright can also:
- Intercept network requests — to get JSON responses from APIs.
- Work with iframes — for parsing embedded widgets.
- Take screenshots — for visual verification.

Combining web_search and Playwright

In a real AI agent, these two tools work in tandem:

1. Web_search finds relevant URLs.
2. Playwright loads each URL with browser emulation.
3. AI analyzes the obtained content (e.g., using NLP) and forms a response.

Example scenario:
- User: "Find prices for iPhone 16 in Moscow stores."
- Agent: performs a search → opens the top 3 stores via Playwright → extracts prices from dynamic blocks → returns a comparison table.

Conclusion

Using web_search and Playwright allows AI agents to work with any