How To Crawl As Googlebot Smartphone With Screaming Frog

Transcription: 

Hi, I’m Chris Long with Go Fish Digital and welcome to another episode of “60 Second SEO”

Today we’re going to talk about how you can change your Screaming Frog settings to crawl as Googlebot Smartphone.

As most of you are probably aware, Google uses mobile-first indexing. It’s going to take a look at the mobile version of your site first. That means when we perform a crawl of our site, to get the best representation of what Google sees, we need to be crawling as if we’re using Googlebot Smartphone.

So here I’m on Mattel site and let’s say I wanted to perform a crawl. What I can do is I can go to Screaming Frog and I can go up to “Configuration” and then click on “User Agent”. If you haven’t changed this setting, it might be preset to Screaming Frog SEO Spider. I can click this drop-down here and I can just click Googlebot Smartphone and then click “OK”, I can start my crawl.

And now Screaming Frog is going to crawl as if it was the Googlebot Smartphone user agent, giving me a much more accurate representation of how Google might actually be crawling my site, especially if we know that Google is using mobile-first indexing for our site.

About Chris Long

Chris Long is the VP of Marketing at Go Fish Digital. With 10+ years of SEO experience, Chris works unique problems and advanced search situations to help clients improve organic traffic through a deep understanding of Google's algorithm and web technology. Chris has advised on the search strategies for clients such as GEICO, Adobe & The New York Times. He is a contributor for Moz, Search Engine Land, and Search Engine Journal. He is also a speaker at industry conferences such as MozCon, SMX East, BrightonSEO and more.