Screaming Frog is the greatest SEO tool of all time. Possibly the greatest tool of all time, closely flowed by the George Foreman grill.
There’s a free version and if you have massive ecommerce website, with thousands of URLs, the posh paid version is well cheap too – if I remember rightly, is only about £250.
Scheduling a Report / Crawl
Before you start – create a crawl config file.
You’ll also need to create a Google Drive and Looker Studio Account
To schedule Screaming Frog to run at a specific date and time (and frequency):
File > Scheduling:
Give the scheduled crawl a name and set the time and date you want the first crawl to run
Also choose the frequency.
Note —> if you have a computer with a decent processor, that runs fine when you are crawling with Screaming Frog and you’re still able to do you work, then just set it for a date and time when you’ll be in work and on your computer.
However, if you have a slower computer, and running Screaming Frog at the same time as trying to do your normal work slows the computer down too much, you’ll probably want to set the crawl overnight, or at your lunchtime.
You can set windows computers to “wake up”, as long as they are plugged in. Although your IT department might not like it – computer running unattended with a battery etc could be in theory a fire harzard etc.
On the start options tab, enter the URL you want to crawland select your config file
I didn’t choose anything for “auth config”
For crawl config – upload the config file you saved earlier
In the Export – choose a folder on your computer and your Google Drive account:
- Click the configure icon for “Export for Looker Studio” at the bottom.
- Click the 3 arrows to populate everything >>
By default, your crawl will go in a sheet in Google Drive at:
‘My Drive > Screaming Frog SEO Spider > Project Name > [task_name]_custom_summary_report’.
- Make a copy of SF’s looker studio template – https://datastudio.google.com/reporting/484b14ec-aed1-40de-ab2f-2b248980ac86/page/rJxg
When asked – choose Google Sheets as a data source
Ensure the ‘use first rows as headers’ option is ticked and select ‘Connect’ in the top-right.
- Add the Data Source to Each Table in the Report in Looker Studio
You can probably do this in bulk somehow, but I went to each table/graph and added the Google Sheet as the data source:
- When the scheduled crawl, crawls – the template should be populated
Waking Up Your Computer so the Crawl Can Run
Screaming Frog won’t run unless your computer is on.
You can ‘wake up’ your computer though.
Search for “Task Scheduler”
- Create Task…
- Click the General Tab and use the following settings: