Screaming Frog Version 21.0 Makes ChatGPT Crawling 10x Easier

Screaming Frog just launched a direct OpenAI integration and it makes crawling with ChatGPT 10X easier.

This week, Screaming Frog launched a new version of their crawler. In typical fashion, it’s filled with some really cool update including new Accessibility features, bulk uploads to custom searches and more.

However, one of the biggest updates is that it makes crawling with ChatGPT even easier than it was before. Now with the new version, you’re able to directly connect to the OpenAI API.

I’m going to use an example of crawling the ecommerce site Burton to automatically generate alt text for their images.  Here’s how you can use Screaming Frog’s new OpenAI integration:

1. Open up Screaming Frog

2. Navigate to Configuration > API Access > AI > OpenAI.

3. Here’s where you’ll create your single connection. Enter your API Key in the “API Key” field.

4. In the “Prompt Configuration” tab, select “Add from Library”

5. Choose from one of the options in the “Systems” tab.

6. You’ll now be able to use dropdown/fields to edit the prompt. Click the arrows icon to adjust the prompt as needed.

7. If you want, click the “play” icon. This opens up a form where you can test the prompt

8. Save and begin your crawl.

9. In the left sidebar, scroll ALLLL the way to the bottom. You’ll be able to see the “AI” report. You’re executed prompt will be here.

Notice how you didn’t need to turn on JavaScript crawling OR edit code like you do with Custom JavaScript functions. This makes it so much easier to create and execute an OpenAI connection. As well, you can run multiple prompts through only a single OpenAPI connection!

If you haven’t started crawling with ChatGPT, I implore you to start now.