How to Automate a Crawl & Populate a Dynamic Report with Screaming Frog [Like Deepcrawl & BrightEdge Dashboards but 99% Cheaper]

Screaming Frog is the greatest SEO tool of all time. Possibly the greatest tool of all time, closely flowed by the George Foreman grill.

There’s a free version  and if you have massive ecommerce website, with thousands of URLs, the posh paid version is well cheap too – if I remember rightly, is only about £250.

Scheduling a Report / Crawl

Before you start – create a crawl config file.

You’ll also need to create a Google Drive and Looker Studio Account

To schedule Screaming Frog to run at a specific date and time (and frequency):

File > Scheduling:

Give the scheduled crawl a name and set the time and date you want the first crawl to run

Also choose the frequency.

Note —> if you have a computer with a decent processor, that runs fine when you are crawling with Screaming Frog and you’re still able to do you work, then just set it for a date and time when you’ll be in work and on your computer.

However, if you have a slower computer, and running Screaming Frog at the same time as trying to do your normal work slows the computer down too much, you’ll probably want to set the crawl overnight, or at your lunchtime.

You can set windows computers to “wake up”, as long as they are plugged in. Although your IT department might not like it – computer running unattended with a battery etc could be in theory a fire harzard etc.

On the start options tab, enter the URL you want to crawland select your config file

I didn’t choose anything for “auth config”

For crawl config – upload the config file you saved earlier

In the Export – choose a folder on your computer and your Google Drive account:

• Click the configure icon for “Export for Looker Studio” at the bottom.
  
• Click the 3 arrows to populate everything >>

By default, your crawl will go in a sheet in Google Drive at:
‘My Drive > Screaming Frog SEO Spider > Project Name > [task_name]_custom_summary_report’.

• Make a copy of SF’s looker studio template – https://datastudio.google.com/reporting/484b14ec-aed1-40de-ab2f-2b248980ac86/page/rJxg

When asked – choose Google Sheets as a data source

Ensure the ‘use first rows as headers’ option is ticked and select ‘Connect’ in the top-right.

• Add the Data Source to Each Table in the Report in Looker Studio

You can probably do this in bulk somehow, but I went to each table/graph and added the Google Sheet as the data source:

• When the scheduled crawl, crawls – the template should be populated

Waking Up Your Computer so the Crawl Can Run

Screaming Frog won’t run unless your computer is on.

You can ‘wake up’ your computer though.

Search for “Task Scheduler”

• Create Task…

• Click the General Tab and use the following settings: