The more often your site gets crawled, the more quickly your content will appear in the search results. And that’s the point, right? You wouldn’t create a new product then keep it in the shed, would you? But the things you need to do to get your site crawled more frequently are also best practice SEO and could even improve your position in the search engines - getting even more of that money-making traffic. All this you can do in just nine steps.
1. Create more content
Creating new content is good for a whole range of reasons. Users love it, and so do search engines. Every time Google crawls your site, it looks for stuff it hasn’t seen before. If it can’t find any it might wait a little longer before visiting you again next time. But if it does find new stuff, it’ll probably come back sooner (newspaper websites for example, who update content almost constantly, get crawled several times a day). Try to update your content as often as it makes sense to do so. (Remember though: quality is better than quantity. So don’t just create content for creating content’s sake.)2. But don’t duplicate content
Yes Google likes seeing lots of content, but it doesn’t like seeing the same content over and over. That’s just boring. So don’t duplicate your content.If there are legitimate reasons for you to duplicate your content (in your heart of hearts, you know if you’re being naughty or nice), you might want to deploy a redirect or a canonical tag In fact, if you have duplicate content you should. Whether you want to or not.
3. Make your site speedy
Just like you wouldn’t want to drive your car on a road covered in melted tar, Google doesn’t like sending its spider to crawl slow-loading sites. And users don’t like visiting them. Site speed is also becoming an increasingly important ranking factor so keep yours up to scratch.4. You can also adjust the crawl frequency in Google Webmaster Tools.
Go into ‘Configuration’ then ‘Settings’, and you can go from there:
This feature was designed so that webmasters could tell Google that it is crawling their site too aggressively, ie, it's overloading their servers. So if that’s happening, then by all means turn it down a notch here.
Some people however, use this feature to turn their crawl rate up. Personally, I wouldn’t do this because:
- It’s only valid for 90 days (so you’re just creating work for yourself).
- There’s a risk, indeed a temptation, to push the crawl rate higher than necessary so you might inadvertently overwork your servers and crash your site. Oops.
5. Add a sitemap
Google loves them. SEOs have been talking about them for ages. I‘ve written an (amazing) article on how to create one. You have no excuse. Create a sitemap6. Make sure your server is working as it should
The easiest way of checking this is to log into the Wordtracker's Keywords tool. Once you've logged in, just click on the campaigns you're working on, and the first thing you'll see will be the dashboard.
Here, where I've put the arrow, you can see how many of your site's pages have been crawled by Google and Bing. The higher this number, the better. If the number is very low, there might be a problem. (But treat it as a rough guide. There are lots of reasons why Google might not be indexing your site. If only there was an article on crawling and indexing that explained this more fully? I jest, of course there is. It’s the previous link).
7. Check your server responses
Google wants your pages to be where it expects them to be. It doesn’t like sites whose server keeps throwing back problems. Is your site like this? Check your server responses by running a site audit in the Keywords tool8. Get links
As well as improving your rank, bringing in more traffic and generally being awesome, links potentially get your website crawled more quickly. Why? Because when Google crawls the web, it will find (hopefully) lots of links to your site littered throughout it. The more times it sees other websites pointing to you with links, the harder it is to stay away.9. Know your crawl rate
Finally you should monitor your crawl rate. After all, that’s what we’re talking about. Google probably won’t crawl your site all at once at regular intervals: it’ll do a few pages one day, a few more the next. But these are just details. You want Google to crawl as much of your site as often as possible. So to find out, log into Webmaster Tools. In the 'Health' section, click ‘Crawl stats’.
And you’ll see a not-very-interesting graph that looks like this:
Most of the time, it will look like this. It will go up a bit and it will go down a bit. Don’t worry about individual movements of the line, but do worry about trends.
If there’s a long term upward trend, that’s all good. Keep doing what you’re doing. If there’s a long term downward trend, re-read this article. Start at one, go down to seven, then lather, rinse, repeat as needed. You’ll not turn the ship around overnight, but you will eventually. We hope.
It’s also worth looking at spikes in the graph. Was there anything you did on these days? Was there a Google algorithm change? Think about the method behind Google’s madness and you’ll soon start understanding how it actually works and what you can do to get in its good books.
Bonus tip! Use Google+
OK, so really this is point number 10, but whatever. 10 is such a overused number.Your final top tip to get your site crawled more often is quite simply, to share it on Google+, Google's social network. Remember, one of Google's motivations for creating a social network was to get access to real-time data. So, if you share your new content on Google+, it's likely to get crawled super-fast. Just try it, and you'll see what I mean. (And, if you're not sure how to do this, check out our Google+ guide