How an AI Agent Finds Data on the Internet: web_search and Playwright for Parsing JS Sites

What is AI Search and Why Conventional Methods Fall Short

Modern AI agents, such as voice assistants, chatbots, and automation systems, often need up-to-date data from the internet. However, static HTML pages are just the tip of the iceberg. Most websites today are built on JavaScript (React, Vue, Angular), where content loads dynamically. A simple requests.get() is powerless here. Two tools come to the rescue: web_search (keyword search) and Playwright (headless browser for parsing).

How an AI Agent Uses web_search

Web_search is an API method that allows an AI to send a query to a search engine (Google, Bing, or a custom index) and receive a list of relevant links. The algorithm works in three steps:

1. Query formulation — The AI translates the user's task into a search query (e.g., "latest news about neural networks 2025").
2. Result retrieval — A list of URLs, titles, and snippets is returned.
3. Filtering — The AI selects the most suitable links for further data collection.

Example code (pseudo-logic):

query = "USD exchange rate today"
results = web_search(query, source="google", count=5)
for link in results:
    content = playwright_parse(link.url)

Playwright: A Headless Browser for Dynamic Sites

Playwright is a library from Microsoft for controlling a browser (Chromium, Firefox, WebKit) in headless mode. Unlike BeautifulSoup, it renders JavaScript, performs clicks, waits, and scrolls. This is critical for:
- Sites with infinite scrolling (social media feeds).
- Pages where data is loaded via API after DOM loading.
- Forms requiring input or option selection.

How Playwright Helps in Parsing

1. Waiting for elements — page.wait_for_selector() waits for the desired block to appear.
2. Interaction — You can click a "Show more" button to load content.
3. Data collection — page.evaluate() extracts data directly from JavaScript variables.

Example of parsing a price from a dynamic site:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com/product")
    page.wait_for_selector(".price")
    price = page.text_content(".price")
    browser.close()

Combination of web_search + Playwright: Step-by-Step Scenario

Imagine an AI agent needs to gather technical specifications from different sites:

1. Search — web_search returns 10 URLs with the query "smartphones 2025 review".
2. Selection — The AI filters links by domain (e.g., only .ru).
3. Parsing — Playwright opens each page, waits for the specification table to load, and extracts data.
4. Structuring — Results are combined into JSON for further processing.

This approach saves hours of manual copying and allows the AI agent to work with real data in real time.

Practical Tips for Effective Parsing

• Respect robots.txt — Check if parsing is allowed on the site.
• Use random delays — To avoid overloading the server (Playwright supports pauses).
• Handle errors — If an element is not found, log it and move on.
• Cache results — Don't parse the same pages every time.

Conclusion

An AI agent armed with web_search and Playwright can extract data from the most "complex" sites — from online stores to news portals. This opens up opportunities for automating monitoring, pricing, and competitor analysis. Want to implement such parsing in your project? Start small — try Playwright on one site, then expand to hundreds of URLs. The future is already here: AI searches, and Playwright retrieves.