Use Screaming Frog Custom JavaScript to Pull Google Search Results for Your Pages Target Topics

Published: August 06, 2025• Updated: August 06, 2025

Use Screaming Frog Custom JavaScript to Pull Google Search Results for Your Pages Target Topics featured cover image

One of the most efficient ways to understand how Google might interpret and rank your content is to examine who else is ranking for similar queries. With Screaming Frog’s Custom JavaScript Extraction and Google’s Custom Search JSON API, you can extract the top 10 organic competitors for each page’s H1, automatically and at scale.

This technique lets you rapidly audit who Google surfaces for queries semantically tied to your content’s H1, making it an excellent tool for competitive analysis, content gap identification, and SERP strategy refinement

What You’ll Learn in This Guide

Before diving in, here’s a quick overview of the key components this post covers.

How to get a Google Programmable Search API key
How to configure a Programmable Search Engine
How to extract the top 10 results using Screaming Frog’s JavaScript feature
Where this fits into broader SEO workflows for competitor tracking and SERP analysis

Pricing Overview For Google’s Programmable Search API

To understand the cost, here are Google’s current usage limits.

Free: 100 API calls per day
Paid: $5 per 1,000 API calls beyond the free tier

This makes it affordable for running quick analyses across a full sitemap or strategic sections of your site.

Guide To Setting Up Screaming Frog To Use Google’s Custom Search JSON API

To implement this workflow, you’ll need an API key, a custom search engine, and Screaming Frog’s JavaScript extractor enabled. Here’s how to do each step.

1. Get Your API Key

The API key is used to authenticate your requests to Google’s search engine API.

Go to the Google Cloud Console.
Select or create a project.
In the left menu, navigate to APIs & Services → Library.
Search for Custom Search API, click into it, and enable it.
Then, go to APIs & Services → Credentials and create an API key.
Copy this key. You’ll use it in your script.

2. Create a Programmable Search Engine

You’ll now configure a Programmable Search Engine that returns results from across the web. This is the engine that will power your queries.

Visit programmablesearchengine.google.com.
Click Add and configure as shown in the screenshot below.

Set it to:

Search the entire web (not just specific sites)
Leave SafeSearch and Image Search off unless you need them

Once created, copy your Search Engine ID (called cx) you can pull this from your programmable search URL, which will also be used in the script.

Note: While this engine isn’t the same as Google’s core ranking system, it offers a very close representation of organic search results using Google’s core index and ranking signals.

3. Enable JavaScript Rendering in Screaming Frog

Screaming Frog must be in JavaScript mode to run client-side scripts like this one.

To enable it:

Go to Config > Spider > Rendering
Set Rendering Mode to JavaScript

This ensures Screaming Frog can run the extraction script properly as each page loads.

4. Add the JavaScript Extraction Snippet

Now you’ll insert the JavaScript that performs the API call for each page based on its H1. This is where the core functionality happens.

In Screaming Frog:

Navigate to Config > Custom > Custom JavaScript
Click “Add” and create a new Extraction snippet
Paste the following script:

return (async () => {
  const apiKey = 'YOUR_API_KEY'; // Replace with your real API key
  const cx = 'YOUR_CX_ID';       // Replace with your Search Engine ID

  const h1 = document.querySelector('h1')?.innerText || document.title;
  const query = encodeURIComponent(h1);
  const url = `https://www.googleapis.com/customsearch/v1?key=${apiKey}&cx=${cx}&q=${query}`;

  try {
    const resp = await fetch(url);
    const data = await resp.json();

    if (!data.items || data.items.length === 0) {
      return seoSpider.data('No results');
    }

    const top10 = data.items.slice(0, 10);
    const formatted = top10.map((item, i) => `${i + 1} - ${item.title} - ${item.link}`).join('\n');

    return seoSpider.data(formatted);
  } catch (err) {
    return seoSpider.error(err.message);
  }
})();

Note: You must add your API key and CX ID.

How It Helps: Competitor Discovery at Scale

With this method, you’re able to automatically uncover the sites Google ranks for the same query implied by your page’s H1. This can be invaluable for:

Quickly gathering SERP competitors across thousands of pages
Analyzing where your content stands in relation to others
Auditing intent alignment between your content and what Google serves
Building competitive landscape maps for each keyword topic

You no longer need to manually Google each keyword to see who ranks—you can do it programmatically.

What the Output Looks Like

Once the crawl finishes, you’ll see a custom column in Screaming Frog’s export with entries like this:

1 - What Is AI Overview? - https://example.com/ai-overview
2 - Google’s AI Search Feature Explained - https://another.com/google-ai
3 - How to Optimize for SearchGPT - https://optimize.com/searchgpt-guide
...
10 - Generative AI Ranking Tips - https://gptseo.com/overview

Each row reflects the top 10 results that Google returned for the H1 on that specific page.

Important Notes on API Limits and CORS

You get 100 free searches per day; anything beyond that costs $5 per 1,000.
This script sends requests directly from the page, meaning CORS (Cross-Origin Resource Sharing) policies may block some requests.
If you’re crawling domains with strict CORS policies, some API calls may fail.

If needed, you can proxy requests server-side to avoid this issue though it requires more setup.

This Screaming Frog + Google Programmable Search integration offers an efficient way to gather real-world competitor data based on your actual page content. It’s perfect for teams that want a scalable, low-cost solution to: