How an AI Agent Finds Data on the Internet: web_search and Playwright for Parsing JavaScript Sites

Modern AI agents are no longer just chatbots. They have learned to independently go online, search for relevant information, and process complex web pages. But how exactly does this happen? Why is regular search insufficient, and why does an AI agent need a headless browser? In this article, we will break down two key tools: web_search for query-based search and Playwright for deep parsing of dynamic sites.

Why Can't AI Just "Read" the Internet?

When an AI agent receives a task to find, for example, the latest fintech news or a competitor's price list, it faces two main problems:
1. Access to a search engine — The AI must be able to formulate queries and obtain links (web_search).
2. Content rendering — 80% of modern websites use JavaScript. Without executing JS code, the page remains empty (Playwright).

Web_search: How AI Searches in Google and Yandex

Web_search is a module that emulates user actions in a search engine. The AI agent sends a query, receives a list of links, titles, and snippets, and then decides which ones to open.

Example workflow:
- AI query: "Find the price of iPhone 16 in Moscow stores."
- web_search returns 10 links.
- AI analyzes the snippets and selects the 3 most relevant URLs for further parsing.

Problem: The search engine does not provide the full HTML of the page, only meta-descriptions. To obtain accurate data, you need to visit the sites.

Playwright: A Lifesaver for JavaScript Sites

Playwright is a library for browser automation. It launches a headless browser (without a graphical interface) that fully renders the page: executes scripts, loads images, waits for elements to appear.

How it works in conjunction with AI:
1. AI receives a URL from web_search.
2. Playwright opens the page in the background.
3. Waits for content to load (e.g., the appearance of the CSS class .price).
4. Extracts text, tables, or attributes.
5. Passes the data to the AI agent for analysis.

Key Playwright Features for Parsing:

• Element waiting: page.wait_for_selector('.price', state='visible') — won't start parsing until the price appears.
• Working with iframes and pop-ups: Can click on "Accept Cookies" buttons or close pop-up windows.
• Screenshots and PDFs: If you need to save the visual state of the page.
• User-Agent switching: Masquerading as a real user.

Practical Example: Collecting Discounts from an Online Store

Suppose an AI agent needs to collect all discounted items from a site that loads content via JavaScript.

Algorithm:
1. Web_search → query "electronics store discounts."
2. Playwright → opens the page, waits 3 seconds for product cards to load.
3. Parsing → extracts names, old and new prices, discount percentage.
4. AI analysis → sorts by discount size and presents the top 5 offers to the user.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto('https://example-shop.ru/sale')
    page.wait_for_selector('.product-card')  # Wait for products to load
    items = page.query_selector_all('.price-new')
    for item in items:
        print(item.inner_text())
    browser.close()

Challenges and Solutions

Conclusion

The combination of web_search + Playwright makes an AI agent a truly autonomous internet researcher. The first finds sources, the second extracts clean data even from the most complex JS sites.

Want to implement such an agent in your business? Start small: set up the search engine