How an AI Agent Finds Data on the Internet: web_search and Playwright for Parsing JS Sites

## What is AI Search and Why Conventional Methods Fall Short Modern AI agents, such as voice assistants, chatbots, and automation systems, often need up-to-date data from the internet. However, static HTML pages are just the tip of the iceberg. Most websites today are built on JavaScript (React, Vue, Angular), where content loads dynamically. A simple `requests.get()` is powerless here. Two tools come to the rescue: **web_search** (keyword search) and **Playwright** (headless browser for parsing). ## How an AI Agent Uses web_search Web_search is an API method that allows an AI to send a query to a search engine (Google, Bing, or a custom index) and receive a list of relevant links. The algorithm works in three steps: 1. **Query formulation** — The AI translates the user's task into a search query (e.g., "latest news about neural networks 2025"). 2. **Result retrieval** — A list of URLs, titles, and snippets is returned. 3. **Filtering** — The AI selects the most suitable links for further data collection. Example code (pseudo-logic): ```python query = "USD exchange rate today" results = web_search(query, source="google", count=5) for link in results: content = playwright_parse(link.url) ``` ## Playwright: A Headless Browser for Dynamic Sites Playwright is a library from Microsoft for controlling a browser (Chromium, Firefox, WebKit) in headless mode. Unlike BeautifulSoup, it renders JavaScript, performs clicks, waits, and scrolls. This is critical for: - Sites with infinite scrolling (social media feeds). - Pages where data is loaded via API after DOM loading. - Forms requiring input or option selection. ### How Playwright Helps in Parsing 1. **Waiting for elements** — `page.wait_for_selector()` waits for the desired block to appear. 2. **Interaction** — You can click a "Show more" button to load content. 3. **Data collection** — `page.evaluate()` extracts data directly from JavaScript variables. Example of parsing a price from a dynamic site: ```python from playwright.sync_api import sync_playwright with sync_playwright() as p: browser = p.chromium.launch(headless=True) page = browser.new_page() page.goto("https://example.com/product") page.wait_for_selector(".price") price = page.text_content(".price") browser.close() ``` ## Combination of web_search + Playwright: Step-by-Step Scenario Imagine an AI agent needs to gather technical specifications from different sites: 1. **Search** — web_search returns 10 URLs with the query "smartphones 2025 review". 2. **Selection** — The AI filters links by domain (e.g., only .ru). 3. **Parsing** — Playwright opens each page, waits for the specification table to load, and extracts data. 4. **Structuring** — Results are combined into JSON for further processing. This approach saves hours of manual copying and allows the AI agent to work with real data in real time. ## Practical Tips for Effective Parsing - **Respect robots.txt** — Check if parsing is allowed on the site. - **Use random delays** — To avoid overloading the server (Playwright supports pauses). - **Handle errors** — If an element is not found, log it and move on. - **Cache results** — Don't parse the same pages every time. ## Conclusion An AI agent armed with web_search and Playwright can extract data from the most "complex" sites — from online stores to news portals. This opens up opportunities for automating monitoring, pricing, and competitor analysis. Want to implement such parsing in your project? Start small — try Playwright on one site, then expand to hundreds of URLs. The future is already here: AI searches, and Playwright retrieves.

How an AI Agent Finds Data on the Internet: web_search and Playwright for Parsing JS Sites

Recent articles

REST API with AI: How an Agent Connects to Any Service Without Code

Create Your AI Telegram Bot in 30 Minutes Without Code: Channel Automation and Subscriber Responses

AI Agents and Databases: How Neural Networks Write SQL Queries and Automate Data Analysis

How to Create a Telegram AI Bot in 30 Minutes Without Code: A Complete Guide for Channel Automation

How to Connect Gmail and Telegram to an AI Agent via a Unified Connector: A Step-by-Step Guide with Credentials

AI for 1C: How to Automate Accounting and Document Workflow with Neural Networks

Reading Documents on PC via Desktop Agent: .docx, .xlsx, .pdf Without Uploading to Server

How to Connect Gmail and Telegram to an AI Agent via a Unified Connector: A Step-by-Step Guide with Credentials

Top 10 Mistakes When Working with AI Agents: How to Avoid Them and Boost Efficiency