How an AI Agent Searches the Internet: web_search and Playwright for Parsing JS Sites in 2026

Modern AI agents are increasingly taking on tasks that once required manual labor: from competitor research to price monitoring. But how exactly does artificial intelligence "see" the internet? The answer lies in the tandem of two technologies: **web search** (search queries) and **Playwright** (a headless browser for parsing dynamic content). In this article, as a practicing developer, I'll break down how AI bypasses the limitations of static HTML and extracts data from modern JavaScript websites. Today, June 11, 2026, when 90% of websites use JS frameworks (React, Vue, Angular), the good old requests-based parsing is practically useless. An AI agent must not only find a page but also wait for scripts to load, click buttons, and scroll through feeds. This is where **Playwright** takes the stage—a tool that emulates real user actions in a browser. ## How AI Uses web_search for Data Collection The first stage of an AI agent's work is intelligent search. Unlike a simple search engine, AI doesn't just enter a query and get links. It: 1. **Formulates a hypothesis**—based on the task (e.g., "find prices for iPhone 17 in three stores"), AI generates 5-10 search phrase variants with LSI keywords (e.g., "buy iPhone 17 price," "iPhone 17 cost delivery," "Apple smartphones price list 2026"). 2. **Performs web_search** via API (Google, Bing, or internal indexes). 3. **Filters results**—removes ad pages, forums, and mirrors, keeping only target URLs. | Stage | AI Action | Example for "price monitoring" task | |-------|-----------|--------------------------------------| | 1 | Query generation | "iPhone 17 Pro price in Moscow," "buy iPhone 17 cheap" | | 2 | Search via API | Obtain 10-20 URLs | | 3 | Relevance filtering | Remove links to Avito and YouTube | **Important nuance**: AI must be able to handle **search results** as dynamic content. Google, for example, loads results via JavaScript, so web_search often requires a headless browser even at this stage. ## Playwright: The Brain of JS Site Parsing After obtaining the URL list, the most interesting part begins—**parsing**. Playwright allows the AI agent to open a page in a virtual browser (Chromium, Firefox, WebKit) and perform the same actions as a human: - Waiting for element loading (`.waitForSelector()`) - Infinite scrolling (for social media feeds) - Clicking "Show more" buttons - Filling out forms and searching ### Why Playwright, Not Selenium or Puppeteer? Although **Puppeteer** is also popular, Playwright wins due to: - **Cross-browser support**—one code for Chrome, Safari, Edge. - **Automatic waiting**—no need for `time.sleep()`. - **Handling iframes and pop-up windows**—critical for modern online stores. Here's an example of Python code that an AI agent uses to extract a price from a dynamic site: ```python from playwright.sync_api import sync_playwright import json with sync_playwright() as p: browser = p.chromium.launch(headless=True) page = browser.new_page() page.goto('https://example-shop.com/product/iphone-17') # Wait for JS to load the price page.wait_for_selector('.price-final', timeout=10000) price = page.text_content('.price-final') print(f'Price: {price}') browser.close() ``` This code is the foundation of an AI agent's work: it opens a page, waits for dynamic loading, and parses an element that wouldn't appear in static HTML. ## How AI Combines web_search and Playwright The process looks like a pipeline: 1. **AI receives a task** → e.g., "Collect prices for ASUS laptops from three major stores." 2. **Performs web_search** → finds product page URLs. 3. **Launches Playwright** → for each URL, parses the price, name, rating. 4. **Analyzes and structures** → saves data in JSON or CSV. **Practical case**: In June 2026, our AI agent (based on LangChain) used this combination to mo

How an AI Agent Searches the Internet: web_search and Playwright for Parsing JS Sites in 2026

Comments

Recent articles

Vibe Coding for Mobile Apps: React Native and Flutter via AI

How an AI Agent Searches the Internet: web_search and Playwright for Parsing JS Sites

REST API with AI: How an Agent Connects to Any Service Without Code in 2026

Fine-tuning AI Models: When Retraining Is Needed and When It’s Not — Saving Time and Money

AI for Testing and QA: How Neural Networks Write Unit Tests, Integration Tests, and UI Tests

AI Agent as an Employee: Setting Up Autopilot for Your Business in 2026

AI Targetologist: Full Cycle — From Creative Generation to Auto-Launch of Ads

Vibe Coding Pipeline: From Prompt to Production in One Day — A Complete Guide to CI/CD and Deployment

How to Create an AI Telegram Bot in 30 Minutes Without Code: The Complete 2026 Guide