How to scrape Google search results with Python (and what it really costs)

By SerpWiz · 21 Oct 2026 · 4 min read

Scraping one Google results page takes a few lines of Python. Scraping thousands a day, reliably, is a project in its own right. This guide builds it up from first principles: what works, where it breaks, and what each fix costs, so you can decide whether to build it or buy it.

Step 1: a plain HTTP request#

The obvious first attempt is to fetch the results page:

import requests

r = requests.get("https://www.google.com/search",
                 params={"q": "coffee near seattle", "hl": "en", "gl": "us"},
                 headers={"User-Agent": "Mozilla/5.0"}, timeout=30)
print(r.status_code, len(r.text))

This used to return HTML you could parse. Since early 2025, Google requires JavaScript to show search results, so a client that doesn't run JavaScript gets a page asking it to enable JavaScript instead of the results. Depending on your IP and region, you may also get a cookie consent page (common in the EU) or an "unusual traffic" page.

So the first lesson is that you need a real browser.

Step 2: a headless browser#

Playwright drives a real Chromium that runs JavaScript just as a user's browser would:

pip install playwright
playwright install chromium

from urllib.parse import urlencode
from playwright.sync_api import sync_playwright

def search(query, hl="en", gl="us"):
    url = "https://www.google.com/search?" + urlencode({"q": query, "hl": hl, "gl": gl})
    with sync_playwright() as p:
        browser = p.chromium.launch()
        page = browser.new_page(locale="en-US")
        page.goto(url, wait_until="domcontentloaded")
        results = []
        # Organic result titles are <h3> elements inside a link. Google's markup
        # changes often, so expect to revisit these selectors.
        for h3 in page.locator("a:has(h3)").all():
            results.append({"title": h3.locator("h3").inner_text(), "url": h3.get_attribute("href")})
        browser.close()
        return results

for i, r in enumerate(search("coffee near seattle"), 1):
    print(i, r["title"], r["url"])

Run it a few times from your laptop and it works. Now you have a scraper. The rest of this guide is about the gap between a scraper and a reliable data source.

Step 3: where it breaks at scale#

Blocks and CAPTCHAs#

Google limits how many automated searches one IP address can make. Go past it and you'll get the "unusual traffic" page with a CAPTCHA instead of results. Datacenter IPs, like those of cloud servers, hit the limit fastest. At any real volume you need many IP addresses, which means a proxy service.

Proxies#

Proxy providers sell rotating IPs, usually priced per gigabyte of traffic:

• Datacenter proxies are cheap but are flagged quickly for search traffic.
• Residential and mobile proxies route through real household or carrier connections. They're blocked far less often, and cost several dollars per GB.

Every page your browser loads goes through the proxy, including scripts, styles and images, unless you block what you don't need. Blocked and retried requests use bandwidth too.

Location#

Google tailors results to where the searcher appears to be. For results that match what users in a given country see, you need proxies located in that country, as well as the right gl and hl parameters.

Parsing#

A results page isn't just ten blue links. It can include ads, a knowledge panel, People Also Ask, top stories, maps, videos and shopping. Each has its own markup, which changes without notice. Expect parsers to break, and set up monitoring that notices when they return fewer results than they should, before your data quietly goes wrong.

Servers#

A headless browser uses far more memory and CPU than an HTTP request. Each concurrent browser instance needs hundreds of MB of RAM, so 20 searches in parallel means a server sized for 20 browsers, plus queueing, retries and timeouts around them.

Step 4: what it costs#

Here's a rough model for 100,000 searches a month. Every number is an assumption: measure your own, because page weight and block rates vary a lot.

The engineering line is usually the biggest cost, and the least predictable: a Google layout change can stop your pipeline on a Saturday.

For comparison, 100,000 searches a month on SerpWiz costs $20 on a monthly plan, or $15 a month on an annual one, with no proxies, browsers or parsers to maintain. Failed requests aren't charged.

Step 5: or call an API#

A SERP API runs all of the above for you and returns the results as JSON. With SerpWiz, the whole scraper above becomes:

import os, requests

r = requests.get(
    "https://api.serpwiz.com/v1/google/search",
    params={"q": "coffee near seattle", "gl": "us", "hl": "en"},
    headers={"X-API-Key": os.environ["SERPWIZ_API_KEY"]},
    timeout=60,
)
r.raise_for_status()
data = r.json()
for item in data["results"]:
    print(item["position"], item["title"], item["url"])

The same response includes ads, top_stories, knowledge_panel and related_keywords (related searches and People Also Ask questions), already parsed. Use page for deeper results and gl for the country. The Google Search reference lists every field.

Build or buy?#

Build it if scraping is your core product, you need full control over browser behaviour, or your volume is so large that a dedicated team pays for itself.

Buy it if the search results are an input to something else, like a rank tracker, an AI agent, lead generation or market research. At $0.15 per 1,000 searches, the API usually costs less than the proxy bandwidth alone.

You can start with 1,000 free searches a month, no card required, and compare the output with your own scraper on the same queries.

More from the blog

• The cheapest Google Search API in 2026: 9 SERP APIs priced at real volumes
• How much does it cost to track 1,000 keywords daily?

This post as Markdown: /blog/scrape-google-search-python.md