Modern AI agents are increasingly taking on tasks that once required manual labor: from competitor research to price monitoring. But how exactly does artificial intelligence "see" the internet? The answer lies in the tandem of two technologies: web search (search queries) and Playwright (a headless browser for parsing dynamic content). In this article, as a practicing developer, I'll break down how AI bypasses the limitations of static HTML and extracts data from modern JavaScript websites.
Today, June 11, 2026, when 90% of websites use JS frameworks (React, Vue, Angular), the good old requests-based parsing is practically useless. An AI agent must not only find a page but also wait for scripts to load, click buttons, and scroll through feeds. This is where Playwright takes the stage—a tool that emulates real user actions in a browser.
How AI Uses web_search for Data Collection
The first stage of an AI agent's work is intelligent search. Unlike a simple search engine, AI doesn't just enter a query and get links. It:
- Formulates a hypothesis—based on the task (e.g., "find prices for iPhone 17 in three stores"), AI generates 5-10 search phrase variants with LSI keywords (e.g., "buy iPhone 17 price," "iPhone 17 cost delivery," "Apple smartphones price list 2026").
- Performs web_search via API (Google, Bing, or internal indexes).
- Filters results—removes ad pages, forums, and mirrors, keeping only target URLs.
| Stage | AI Action | Example for "price monitoring" task |
|---|---|---|
| 1 | Query generation | "iPhone 17 Pro price in Moscow," "buy iPhone 17 cheap" |
| 2 | Search via API | Obtain 10-20 URLs |
| 3 | Relevance filtering | Remove links to Avito and YouTube |
Important nuance: AI must be able to handle search results as dynamic content. Google, for example, loads results via JavaScript, so web_search often requires a headless browser even at this stage.
Playwright: The Brain of JS Site Parsing
After obtaining the URL list, the most interesting part begins—parsing. Playwright allows the AI agent to open a page in a virtual browser (Chromium, Firefox, WebKit) and perform the same actions as a human:
- Waiting for element loading (
.waitForSelector()) - Infinite scrolling (for social media feeds)
- Clicking "Show more" buttons
- Filling out forms and searching
Why Playwright, Not Selenium or Puppeteer?
Although Puppeteer is also popular, Playwright wins due to:
- Cross-browser support—one code for Chrome, Safari, Edge.
- Automatic waiting—no need for time.sleep().
- Handling iframes and pop-up windows—critical for modern online stores.
Here's an example of Python code that an AI agent uses to extract a price from a dynamic site:
from playwright.sync_api import sync_playwright
import json
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
page.goto('https://example-shop.com/product/iphone-17')
# Wait for JS to load the price
page.wait_for_selector('.price-final', timeout=10000)
price = page.text_content('.price-final')
print(f'Price: {price}')
browser.close()
This code is the foundation of an AI agent's work: it opens a page, waits for dynamic loading, and parses an element that wouldn't appear in static HTML.
How AI Combines web_search and Playwright
The process looks like a pipeline:
- AI receives a task → e.g., "Collect prices for ASUS laptops from three major stores."
- Performs web_search → finds product page URLs.
- Launches Playwright → for each URL, parses the price, name, rating.
- Analyzes and structures → saves data in JSON or CSV.
Practical case: In June 2026, our AI agent (based on LangChain) used this combination to mo
Comments