Using Screaming Frog's CLI mode on a server
Command line crawling with Screaming Frog SEO Spider
By Julien on September 19, 2018
Screaming Frog just released SEO Spider v10, with a lot of impressive new features.
Amongst the list is the CLI mode: the ability to use the crawler without a GUI (on a server for example).
Here’s a quick guide on how to get started with Screaming Frog’s CLI mode on a Debian server.
Setup
We’ll assume that you’re correctly logged in to your server, via SSH for instance, and that you have administration (sudo
) rights.
Remeber upgrading your system if needed ;-)
You’ll need to install some dependencies.
First, let’s download the latest version:
Check the official website for an updated link to the latest file.
Once the file is downloaded, launch installation:
Check if everything is OK:
Licence
You’ll need to enter a licence to use SF in headless mode.
Simply edit ~/ScreamingFrogSEOSpider/licence.txt
and enter your username on the first line, and your key on the second.
EULA agreement
At first launch, Screaming Frog’s GUI asks you to agree to the terms and conditions. This can’t be done without a GUI.
However, there’s a workaround.
Edit ~/ScreamingFrogSEOSpider/spider.config
and add the following line:
Save and exit.
Start crawling
To start crawling in headless mode, you’ll need to use at least a few arguments:
--crawl <url>
is the starting URL,--headless
is needed, otherwise SF will try to open a GUI (and fail),--save-crawl
enables you to save your data to acrawl.seospider
file,--output-folder <folder>
will save the crawl data to the given folder,--timestamped-output
will create a timestamped folder in which yourcrawl.seospider
file will be saved (this is useful to avoid crushing a previous crawl).
Here’s a minimalist example:
Other options and OS
Checkout SF documentation for more details on how to use Screaming Frog CLI mode on other operating systems, and what command line arguments are available.
Many thanks to the guys at Screaming Frog for this awesome release!