How an AI Agent Searches the Internet: web_search and Playwright for Parsing

Introduction

Modern AI agents are increasingly stepping beyond static knowledge bases and actively interacting with the internet. Imagine asking a neural network a question, and it doesn't just generate an answer but finds up-to-date data in real time — exchange rates, news, product prices. But how does this work under the hood? At its core are two key tools: web_search for finding information and Playwright for parsing dynamic websites loaded with JavaScript. In this article, we'll explore how an AI agent uses these technologies to collect data from any web page, including those that generate content only after executing scripts.

How an AI Agent Searches the Internet: web_search

web_search is an API or module that allows an AI agent to send queries to search engines (Google, Bing, Yandex) and receive structured results: a list of links, titles, and descriptions. This is the first stage of data collection. The algorithm looks like this:

Parsing the user's query: The AI extracts keywords and intents.
Forming the URL: The agent constructs a query string for the search engine (e.g., ?q=best+laptops+2025).
Request to the search engine: An HTTP request is sent, often emulating a real browser to avoid blocks.
Processing the results: The AI analyzes the obtained links, selects relevant ones, and moves to the next step — parsing.

Example of web_search in Action

Suppose an AI needs to find the latest news about cryptocurrencies. The agent sends a query site:coindesk.com Bitcoin 2025. In response, it receives:

https://coindesk.com/bitcoin-new-high
https://coindesk.com/ethereum-update

Then the AI decides which link to parse and passes the task to Playwright.

Why Playwright Instead of Regular requests?

Traditional parsing with libraries like requests + BeautifulSoup only works with static pages. But the modern internet is full of dynamics: content is loaded via AJAX, React, Vue, or Angular. Playwright solves this problem as it is a headless browser (without a graphical interface) that fully emulates user behavior: executes JavaScript, clicks, scrolls, waits for element loading.

Key Playwright Features for AI Parsing:

Browser emulation: Supports Chromium, Firefox, WebKit.
Element waiting: page.wait_for_selector() ensures data is loaded.
Form filling and clicks: Can simulate user actions (e.g., logging into a personal account).
Data extraction: Via CSS selectors or XPath.
Stealth mode: Masquerades as a real user to bypass anti-bot systems.

Example: Parsing a Dynamic Website with Playwright

Let's look at a practical case: collecting prices from an online store where the catalog is loaded via JavaScript.

Step 1: Installation and Initialization

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto('https://example-shop.com/catalog')

Step 2: Waiting for Data to Load

page.wait_for_selector('.product-card', timeout=10000)

Step 3: Extracting Information

products = page.query_selector_all('.product-card')
for product in products:
    title = product.query_selector('.title').inner_text()
    price = product.query_selector('.price').inner_text()
    print(f'{title}: {price}')

Step 4: Closing the Browser

browser.close()

This code parses the page, waits for JS to load, and extracts data. An AI agent can repeat such actions for multiple pages, collecting a structured dataset.

Optimizing AI Search: Combining web_search and Playwright

An efficient AI agent works in a chain:

How an AI Agent Searches the Internet: web_search and Playwright for Parsing

Introduction