How an AI Agent Searches the Internet: web_search and Playwright for Parsing Dynamic Websites

Introduction

Modern AI agents, from chatbots to automated data analysis systems, increasingly use the internet as a source of information. But how do they find the necessary data on websites that load content via JavaScript? A regular HTTP request is powerless here — you'll only get an empty shell. The solution is a combination of web_search for finding relevant pages and Playwright for parsing dynamic content. In this article, we'll explore how these tools work together and provide practical examples.

How web_search Works in AI Agents

Web search (or internet search) is the first stage that allows the agent to find relevant pages. Typically, this involves APIs of search engines (Google, Bing) or open indexes. The agent sends a query, receives a list of URLs and metadata (titles, snippets), and then decides which pages to analyze.

Example of web_search in Action:

• Query: "latest news artificial intelligence 2025"
• Result: a list of 10-20 URLs with titles and brief descriptions.
• Filtering: AI selects 3-5 URLs that appear most relevant.

However, the problem is that many modern websites (e.g., SPAs built with React or Vue) do not serve content as static HTML. This is where Playwright comes to the rescue.

Playwright: A Headless Browser for Parsing JS Websites

Playwright is a library from Microsoft for managing headless browsers (Chrome, Firefox, Safari). It allows you to load a page, execute JavaScript, wait for elements to appear, and extract data. Unlike simple parsers (BeautifulSoup, Requests), Playwright renders the page like a real browser.

Key Features of Playwright:

• JavaScript Rendering: loads dynamic content.
• Element Waiting: automatically pauses until the required block appears.
• Screenshots and PDFs: for debugging or saving.
• Mobile Emulation Support: testing responsive websites.

Step-by-Step Example: web_search + Playwright

Imagine an AI agent needs to collect product reviews from a React-based website. Here's the algorithm:

1. Web search: send a query "reviews for product X site:example.com". Get the URL of the reviews page.
2. Playwright: open a headless browser, navigate to the URL, wait for content to load (e.g., using page.waitForSelector('.review-class')).
3. Parsing: extract review text, ratings, dates.
4. Processing: AI analyzes the data and forms a response.

Python Code (Simplified):

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto('https://example.com/reviews')
    # Wait for dynamic elements to load
    page.wait_for_selector('.review-item', timeout=10000)
    reviews = page.query_selector_all('.review-text')
    for review in reviews:
        print(review.inner_text())
    browser.close()

Advantages and Limitations

Advantages:

• Flexibility: Playwright works with any website, including SPAs.
• Precision: can parse data that appears after clicks or scrolling.
• Integration with AI: easy to automate the "search-analyze-output" chain.

Limitations:

• Resource Intensity: a headless browser requires more memory than a simple HTTP request.
• Blocking: some websites detect bots (proxies, User-Agent rotation needed).
• Speed: for large volumes of data, it may be slower than APIs.

Conclusion

The combination of web_search and Playwright is a powerful tool for AI agents that need fresh data from the internet. You can automate review collection, price monitoring, news analysis, and much more. Start small: set up keyword searches and add Playwright for parsing dynamic pages. If you want to dive deeper, study Playwright's documentation and try integrating it with your AI agent via an API.

Ready to try it? Contact us