How an AI Agent Searches the Internet: web_search and Playwright for Parsing JS Sites

## Introduction Modern AI agents (e.g., ChatGPT with plugins or assistants based on LangChain) actively use internet search to obtain up-to-date data. However, simply executing a search query is not enough: many sites generate content dynamically using JavaScript. How does AI bypass this problem? In this article, we will break down how web_search and the headless browser Playwright work for parsing sites with dynamic JS content. You will learn how to create an AI agent that not only searches but also collects data from hard-to-reach pages. ## Main Part ### What is web_search for AI? Web_search is a function that allows an AI agent to send HTTP requests to search engines (Google, Bing, DuckDuckGo) and receive results. But the standard requests library cannot handle sites where content is loaded via JavaScript. For example, online stores with infinite product feeds or news portals with lazy loading. **Python code example:** ```python import requests from bs4 import BeautifulSoup # Simple web_search (won't work for JS) query = 'buy laptop' url = f'https://www.google.com/search?q={query}' response = requests.get(url) soup = BeautifulSoup(response.text, 'html.parser') print(soup.title.text) # Only static content ``` ### The Problem with JavaScript Sites Many modern sites use frameworks (React, Vue) or simply load data via API. Regular parsing with requests will only return an empty HTML skeleton. This is where Playwright comes to the rescue — a tool for browser automation. **Playwright launches a headless browser (without a graphical interface) and executes JavaScript on the page, after which the full DOM can be extracted.** ### Using Playwright for Parsing Playwright supports Chromium, Firefox, and WebKit. Installation is simple: ```bash pip install playwright playwright install ``` **Example of parsing dynamic content:** ```python from playwright.sync_api import sync_playwright with sync_playwright() as p: browser = p.chromium.launch(headless=True) page = browser.new_page() page.goto('https://example.com/dynamic-page') # Wait for content to load page.wait_for_selector('.product-list') # Extract data products = page.query_selector_all('.product-title') for product in products: print(product.inner_text()) browser.close() ``` This code opens a page, waits for an element with the class 'product-list' to appear, and collects all product titles. Without Playwright, you would only get an empty div. ### Combining web_search and Playwright for an AI Agent The real power is revealed when the AI agent first performs a search (web_search), gets a list of URLs, and then uses Playwright to parse each URL. This allows collecting data from multiple sources. **Algorithm of operation:** 1. AI receives a request from the user (e.g., "Collect prices for iPhone 15 in three stores"). 2. The agent executes search queries via web_search (e.g., "iPhone 15 price Moscow"). 3. Links to store sites are extracted from the results. 4. For each link, Playwright is launched, which loads the page, waits for JS to load, and extracts the price. 5. The data is structured and returned to the user. **Example integration with LangChain:** ```python from langchain.tools import Tool from playwright.sync_api import sync_playwright class WebScraperTool(Tool): def _run(self, url: str) -> str: with sync_playwright() as p: browser = p.chromium.launch(headless=True) page = browser.new_page() page.goto(url, wait_until='networkidle') content = page.content() browser.close() return content async def _arun(self, url: str) -> str: # Async version pass ``` ### Practical Tips - **Rate limiting:** Use `page.wait_for_timeout()` between actions to avoid overloading the site. - **Handling captchas:** Playwrig

How an AI Agent Searches the Internet: web_search and Playwright for Parsing JS Sites

Recent articles

How to Connect Gmail and Telegram to an AI Agent via a Unified Connector: A Step-by-Step Guide with Credentials

How an AI Agent Finds Data on the Internet: web_search and Playwright for Parsing JS Sites

AI for 1C: How to Automate Accounting and Document Workflow with Neural Networks

Reading Documents on PC via Desktop Agent: .docx, .xlsx, .pdf Without Uploading to Server

How to Connect Gmail and Telegram to an AI Agent via a Unified Connector: A Step-by-Step Guide with Credentials

Top 10 Mistakes When Working with AI Agents: How to Avoid Them and Boost Efficiency

AI and Git: How Neural Networks Manage Your Repository and Automate Code

Neural Networks for Data Analysis: From CSV to Business Dashboards — How AI Agents Are Transforming Business Analytics

REST API with AI: How an Agent Connects to Any Service Without Code