How To Use Screaming Frog’s “Include” Feature

Transcription:

Hi, I’m Chris Long and welcome to another episode of “60-Second SEO”.

Today we’re going to talk about how you can use Screaming Frog to break up crawls of larger websites to make it much more manageable.

So let’s say we wanted to crawl the CNET site. This is a pretty large site and if we tried to crawl it all at once, it might take a while for us to do. What we can do is really try to map out our crawls and figure out: “Hey, does it make more sense to start crawling individual sections instead of the whole site at once?”

So let’s say we really wanted to focus on the technology portion of their articles. That’s a really high priority section for them. What we can do is we can pull up Screaming Frog and we can actually crawl only that section.

So you can go to “Configuration” and then you can go to “Include”. And then here I just copied and pasted the technology section and then did a period and then the star right next to it to indicate that we want to crawl anything that comes after this individual URL slug I selected.

Ok, I started the crawl and now what you can see is Screaming Frog is only going to return URLs in that specific section with that URL path and that’s going to make the crawl much more manageable. It’s more likely Screaming Frog is going to get to the end of the crawl and we’re going to be able to really dive deep into this individual high priority section instead of trying to crawl this whole big site all at one time.