How to know if your website is in Google's mobile-first index ?

In a recent hangout, Google’s John Mueller told webmasters to watch their server logs, in order to know if a website has been moved to the mobile first index.

This kind of monitoring can easily be done with a few Unix command-line tools, as long as you know where to find your log files.

First, we’ll filter our log file to keep only the lines corresponding to any Googlebot:

grep Googlebot access.log  

You should see loads of lines stream on your screen. Stop the process by typing ctrl+c.

We’ll now use awk to feep only the User-agent from each line. The -F argument sets the field separator, while '{print $6}' prints only the 6th field of each line (in our case, the User-agent). Let’s try this:

grep Googlebot access.log | awk -F\" '{print $6}'  

Loads of lines streaming again, but only with User-agents. Everything is working fine :-)

Finally, let’s count how many visits we got from each UA. We’ll just sort the stream, then count duplicate lines:

grep Googlebot access.log | awk -F\" '{print $6}' | sort | uniq -c  

This displays the count of visits per User-agent: simply compare volumes for desktop and mobile Googlebots.

Here is what I get on one of my sites:

10405 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)  
  779 Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)  

Still in desktop index !