How an AI Agent Searches the Internet: web_search and Playwright for Parsing Dynamic Sites

Introduction

Modern AI agents have gone far beyond simple chatbots. Today, they can independently perform complex tasks, including searching for information on the internet. But how exactly does artificial intelligence find the necessary data? In this article, we will break down two key tools: web_search for working with search engines and Playwright for parsing websites with dynamic JavaScript content. You will learn how an AI agent can efficiently collect information even from the most complex web pages.

Why Do AI Agents Need Internet Search?

AI agents, such as voice assistants or support chatbots, often face limitations in their built-in knowledge. Databases become outdated, and new data appears daily. Using web_search allows the agent to:

• Provide up-to-date answers to user questions.
• Analyze news, prices, and reviews in real time.
• Integrate with external data sources.

However, traditional search via APIs (e.g., Google Custom Search) does not always provide access to the full content of pages. Many websites use JavaScript to load content, and this is where Playwright comes to the rescue.

What is web_search and How to Set It Up?

web_search is a module or function that allows an AI agent to send queries to search engines and retrieve results. This is most often implemented through:

• Search engine APIs (Google, Bing, DuckDuckGo).
• Python libraries (e.g., googlesearch-python).
• Custom scripts that parse HTML result pages.

Example of Setting Up web_search Using Python:

from googlesearch import search

def search_web(query):
    results = []
    for url in search(query, num_results=5):
        results.append(url)
    return results

# Example usage
query = "AI agent data search"
urls = search_web(query)
print(urls)

Important! Directly parsing search engines may violate their terms of service. It is recommended to use official APIs or libraries that respect robots.txt.

Parsing Dynamic Sites with Playwright

After obtaining a list of URLs from web_search, the AI agent needs to extract the content of the pages. Static HTML pages are easily parsed with BeautifulSoup, but modern websites (e.g., online stores, social networks) often load data via JavaScript. Here, Playwright becomes an indispensable tool.

Why Playwright?

Playwright is a library for browser automation (Chromium, Firefox, WebKit) that supports:

• Opening a headless browser (without a graphical interface).
• Waiting for dynamic content to load.
• Emulating user actions (clicks, scrolls).
• Working with iframes, popups, and AJAX requests.

Example of Parsing with Playwright:

from playwright.sync_api import sync_playwright

def parse_js_site(url):
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()
        page.goto(url, wait_until="networkidle")
        # Wait for a specific element to load
        page.wait_for_selector(".content-class", timeout=5000)
        content = page.text_content(".content-class")
        browser.close()
        return content

# Example
url = "https://example.com/dynamic-page"
text = parse_js_site(url)
print(text)

How Does an AI Agent Combine web_search and Playwright?

The full workflow of an AI agent looks like this:

1. User Request: The agent receives a task (e.g., "Find the latest news about AI").
2. Search: Uses web_search to obtain a list of relevant URLs.
3. Filtering: Discards broken or irrelevant links (can be done with ML).
4. Parsing: For each URL, runs Playwright