How an AI Agent Searches the Internet: web_search and Playwright for Parsing JavaScript Sites

Modern AI agents have ceased to be just chatbots. Now they are capable of independently going online, finding relevant information, and processing data in real time. But how does this work on a technical level? In this article, we will break down two key tools: web_search (internet search) and Playwright (headless browser for parsing dynamic sites).

How does web_search work in AI agents?

An AI agent uses web_search as its own browser. Instead of manually entering queries, it:

• Receives a task (e.g., "Find the latest news about neural networks")
• Forms a search query
• Sends it to a search engine (Google, Bing, Yandex)
• Analyzes the results (titles, snippets, links)

Code example:

from langchain.tools import Tool
from langchain.utilities import GoogleSearchAPIWrapper

search = GoogleSearchAPIWrapper()
tool = Tool(
    name="web_search",
    func=search.run,
    description="Internet search"
)

Why does regular parsing fail with JS sites?

Modern sites (React, Vue, Angular) generate content dynamically using JavaScript. Regular HTTP requests (requests, urllib) return only an empty HTML shell, without data. This is where Playwright comes to the rescue.

Playwright for parsing: headless browser in action

Playwright is a library from Microsoft that controls a browser (Chromium, Firefox, WebKit) in headless mode. An AI agent can:

1. Open a page
2. Wait for JavaScript to load
3. Perform scrolling
4. Click buttons
5. Extract the needed data

Practical example:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com")

    # Wait for dynamic content to load
    page.wait_for_selector(".product-price")

    # Extract data
    prices = page.query_selector_all(".product-price")
    for price in prices:
        print(price.inner_text())

    browser.close()

Synergy of web_search + Playwright for AI search

The full cycle of an AI agent's work looks like this:

1. Search — web_search finds relevant URLs
2. Selection — AI analyzes snippets and chooses the best links
3. Parsing — Playwright opens pages, waits for JS, extracts data
4. Analysis — AI processes the collected information and forms a response

Advantages of the approach:
- Works with any sites (SPA, SSR, hybrid)
- Emulates user actions (clicks, scrolls)
- Handles captchas and cookie banners
- Parallel launch of multiple browsers

When is this necessary?

• Collecting data from online stores (prices, reviews)
• Monitoring news and social networks
• Filling out forms and registering accounts
• Bypassing blocks (changing User-Agent, proxies)

Conclusion

The combination of web_search and Playwright turns an AI agent into a powerful tool for working with the internet. If your project needs data collection from dynamic sites, this approach will be an ideal solution.

Want to implement AI search in your business? Contact us at Asibiont — we will help set up parsing of any complexity.