How an AI Agent Searches the Internet: web_search and Playwright for Parsing Dynamic Websites

## Introduction Modern AI agents go beyond simply answering questions—they actively search for information in real time, analyze data, and make decisions. But how exactly does an AI agent find the necessary information in the vast expanse of the internet? In this article, we will break down two key tools: web_search (search queries) and Playwright (a headless browser for parsing JavaScript websites). You will learn how AI uses them to collect data and get practical tips for setup. ## How an AI Agent Uses web_search ### What is web_search? Web_search is an API or module that allows an AI agent to send queries to search engines (Google, Bing, Yandex) and receive results in a structured format. Instead of manually entering queries into a browser, the AI does it programmatically. ### Example: Searching for News Imagine an AI agent needs to find the latest news about technology. It sends a query `web_search('latest technology news 2025')`, receives a list of links, titles, and snippets, and then analyzes them. ### Advantages - **Speed**: AI processes hundreds of queries in seconds. - **Accuracy**: Results can be filtered by date, region, or content type. - **Automation**: The agent decides when and what to search for on its own. ## The Problem with Dynamic Websites ### Why Regular Parsing Doesn't Work? Many modern websites generate content using JavaScript. This means that when a page loads, data appears only after scripts execute. A regular HTTP request (e.g., via `requests` in Python) cannot retrieve this content—it will only see empty HTML. ### Solution: Headless Browser For such cases, a tool that emulates a real browser is needed. Playwright is one of the most powerful headless browsers, capable of executing JavaScript, clicking buttons, and waiting for data to load. ## Playwright for Parsing: Step-by-Step Guide ### Installation and Setup First, install Playwright: ```bash pip install playwright playwright install ``` ### Basic Example: Parsing Titles Suppose you need to collect article titles from a website that uses JavaScript to display content. Here's how to do it: ```python from playwright.sync_api import sync_playwright with sync_playwright() as p: browser = p.chromium.launch(headless=True) page = browser.new_page() page.goto('https://example.com') # Wait for dynamic content to load page.wait_for_selector('h2.article-title') # Collect titles titles = page.query_selector_all('h2.article-title') for title in titles: print(title.inner_text()) browser.close() ``` ### Advanced Techniques - **Waiting for Load**: Use `page.wait_for_selector` or `page.wait_for_load_state('networkidle')` to wait for all scripts to execute. - **Working with Forms**: Playwright can fill input fields, click buttons, and handle pop-ups. - **Screenshots and PDFs**: Save the visual state of the page for debugging. ## How AI Combines web_search and Playwright ### Practical Scenario An AI agent receives a task: "Find prices for iPhone 15 in three online stores and compare them." The sequence of actions: 1. **Web_search**: The agent sends a query "buy iPhone 15 price" to Google and gets links to stores. 2. **Playwright**: For each store, the agent opens the page, waits for JavaScript to load (e.g., a dynamic price block), and extracts data. 3. **Analysis**: AI compares prices and outputs the result. ### Example Code (Simplified) ```python import asyncio from playwright.async_api import async_playwright async def scrape_price(url): async with async_playwright() as p: browser = await p.chromium.launch(headless=True) page = await browser.new_page() await page.goto(url) await page.wait_for_selector('.price') price = await page.inner_text('.price') await browser.close() return price # Usage

How an AI Agent Searches the Internet: web_search and Playwright for Parsing Dynamic Websites

Recent articles

How to Connect Gmail and Telegram to an AI Agent via a Unified Connector: A Step-by-Step Guide with Credentials

How an AI Agent Finds Data on the Internet: web_search and Playwright for Parsing JS Sites

AI for 1C: How to Automate Accounting and Document Workflow with Neural Networks

Reading Documents on PC via Desktop Agent: .docx, .xlsx, .pdf Without Uploading to Server

How to Connect Gmail and Telegram to an AI Agent via a Unified Connector: A Step-by-Step Guide with Credentials

Top 10 Mistakes When Working with AI Agents: How to Avoid Them and Boost Efficiency

AI and Git: How Neural Networks Manage Your Repository and Automate Code

Neural Networks for Data Analysis: From CSV to Business Dashboards — How AI Agents Are Transforming Business Analytics

REST API with AI: How an Agent Connects to Any Service Without Code