How an AI Agent Finds Data on the Internet: web_search and Playwright for Parsing JavaScript Sites

Modern AI agents are no longer just chatbots. They have learned to independently go online, search for relevant information, and process complex web pages. But how exactly does this happen? Why is regular search insufficient, and why does an AI agent need a headless browser? In this article, we will break down two key tools: **web_search** for query-based search and **Playwright** for deep parsing of dynamic sites. ## Why Can't AI Just "Read" the Internet? When an AI agent receives a task to find, for example, the latest fintech news or a competitor's price list, it faces two main problems: 1. **Access to a search engine** — The AI must be able to formulate queries and obtain links (web_search). 2. **Content rendering** — 80% of modern websites use JavaScript. Without executing JS code, the page remains empty (Playwright). ## Web_search: How AI Searches in Google and Yandex Web_search is a module that emulates user actions in a search engine. The AI agent sends a query, receives a list of links, titles, and snippets, and then decides which ones to open. **Example workflow:** - AI query: "Find the price of iPhone 16 in Moscow stores." - web_search returns 10 links. - AI analyzes the snippets and selects the 3 most relevant URLs for further parsing. **Problem:** The search engine does not provide the full HTML of the page, only meta-descriptions. To obtain accurate data, you need to visit the sites. ## Playwright: A Lifesaver for JavaScript Sites Playwright is a library for browser automation. It launches a headless browser (without a graphical interface) that fully renders the page: executes scripts, loads images, waits for elements to appear. **How it works in conjunction with AI:** 1. AI receives a URL from web_search. 2. Playwright opens the page in the background. 3. Waits for content to load (e.g., the appearance of the CSS class `.price`). 4. Extracts text, tables, or attributes. 5. Passes the data to the AI agent for analysis. ### Key Playwright Features for Parsing: - **Element waiting:** `page.wait_for_selector('.price', state='visible')` — won't start parsing until the price appears. - **Working with iframes and pop-ups:** Can click on "Accept Cookies" buttons or close pop-up windows. - **Screenshots and PDFs:** If you need to save the visual state of the page. - **User-Agent switching:** Masquerading as a real user. ## Practical Example: Collecting Discounts from an Online Store Suppose an AI agent needs to collect all discounted items from a site that loads content via JavaScript. **Algorithm:** 1. **Web_search** → query "electronics store discounts." 2. **Playwright** → opens the page, waits 3 seconds for product cards to load. 3. **Parsing** → extracts names, old and new prices, discount percentage. 4. **AI analysis** → sorts by discount size and presents the top 5 offers to the user. ```python from playwright.sync_api import sync_playwright with sync_playwright() as p: browser = p.chromium.launch(headless=True) page = browser.new_page() page.goto('https://example-shop.ru/sale') page.wait_for_selector('.product-card') # Wait for products to load items = page.query_selector_all('.price-new') for item in items: print(item.inner_text()) browser.close() ``` ## Challenges and Solutions | Problem | Solution with Playwright | |---------|--------------------------| | CAPTCHA on site | Use proxies and delays between requests | | Infinite scroll (lazy load) | Emulate scrolling: `page.evaluate('window.scrollTo(0, document.body.scrollHeight)')` | | IP blocking | Rotate User-Agent and use headful mode in rare cases | ## Conclusion The combination of **web_search + Playwright** makes an AI agent a truly autonomous internet researcher. The first finds sources, the second extracts clean data even from the most complex JS sites. Want to implement such an agent in your business? Start small: set up the search engine

How an AI Agent Finds Data on the Internet: web_search and Playwright for Parsing JavaScript Sites

Recent articles

How to Connect Gmail and Telegram to an AI Agent via a Unified Connector: A Step-by-Step Guide with Credentials

How an AI Agent Finds Data on the Internet: web_search and Playwright for Parsing JS Sites

AI for 1C: How to Automate Accounting and Document Workflow with Neural Networks

Reading Documents on PC via Desktop Agent: .docx, .xlsx, .pdf Without Uploading to Server

How to Connect Gmail and Telegram to an AI Agent via a Unified Connector: A Step-by-Step Guide with Credentials

Top 10 Mistakes When Working with AI Agents: How to Avoid Them and Boost Efficiency

AI and Git: How Neural Networks Manage Your Repository and Automate Code

Neural Networks for Data Analysis: From CSV to Business Dashboards — How AI Agents Are Transforming Business Analytics

REST API with AI: How an Agent Connects to Any Service Without Code