Googlebot will soon support HTTP/2

September 17, 2020

Starting in November 2020, Google will start crawling some sites over HTTP/2. This is the next major revision of HTTP, which is more robust, efficient, and faster than its predecessor. This change will benefit web professionals who have asked for Googlebot to support crawling over HTTP/2. For more information on HTTP/2, please read the long article on the HTTP/2 topic.

Google is planning to gradually switch to crawling websites over HTTP/2, which is more efficient in terms of server resource usage. This change will happen gradually, with Googlebot first crawling a small number of sites over h2 and then gradually ramping up to more sites that may benefit from the initially supported features. Googlebot decides which sites to crawl over h2 based on whether the site supports h2 and whether the site and Googlebot would benefit from crawling over HTTP/2. If your server still only talks HTTP/1.1, that's also fine - there's no explicit drawback for crawling over this protocol.

Google is upgrading its Googlebot crawler to support HTTP/2. This change will improve the speed and efficiency of crawling, with no negative impact on indexing. However, some site owners may want to opt-out of this change. Sites can do this by instructing the server to respond with a 421 HTTP status code when Googlebot attempts to crawl over h2. If that's not feasible, site owners can send a message to the Googlebot team (however, this solution is temporary).

For more information and answers to questions you might have, check out Google's detailed post (including a link to their Twitter and help forums).

According to Google, the software they use to enable their Googlebot to crawl over h2 has matured enough to be used in production. However, they state that it is up to the website owner whether or not they want to upgrade their server. Googlebot will only switch to crawling over h2 sites that will clearly benefit from it- if there is no clear benefit, Googlebot will continue to crawl over h1.

Cloudflare has a blog post with different methods to test whether a site supports h2. To upgrade a site to h2, Google recommends talking to the server administrator or hosting provider. Website owners cannot convince Googlebot to talk h2 with their site- if the site does support h2 and it would be beneficial for crawling, then Googlebot will automatically switch. If not, then Googlebot will continue crawling on HTTP/1.1.

Google has announced that they will now only crawl sites over HTTP/2 when there is a clear benefit for the site. This change comes after they found little to no benefit for certain sites when crawling over HTTP/2. They will continue to evaluate the performance gains and may change their criteria for switching in the future.

Owners of sites registered in Search Console will get a message saying that some of the crawling traffic may be over HTTP/2 going forward if their site becomes eligible for crawling over HTTP/2. Sites can also check their server logs (for example, in the access.log file if the site runs on Apache) to see if they are being crawled over HTTP/2.

Googlebot supports most of the features introduced by HTTP/2. Some features like server push, which may be beneficial for rendering, are still being evaluated. Googlebot does not currently support plaintext HTTP/2 (h2c).

Google has announced that they will be using HTTP/2 for crawling. This is a newer version of the HTTP protocol that offers multiple benefits, including multiplexing and concurrency, header compression, and server push.

Server push is a feature that is not yet enabled, but is still being evaluated. It may be beneficial for rendering, but Google does not have anything specific to say about it at this point.

In order to be eligible for crawling over HTTP/2, websites must use HTTPS and support HTTP/2. This is equivalent to how modern browsers handle it. If the server responds during the TLS handshake with a protocol version other than h2, Googlebot will back off and come back later on HTTP/1.1.

If you want to learn more about h2 features and crawling, you can ask Google on Twitter. The main benefit of h2 is that it saves resources, both on the server side and on Googlebot's side. Whether Googlebot crawls using h1 or h2 does not affect how your site is indexed, and thus, how much Google plans to crawl from your site. There is no ranking benefit for a site that is crawled over h2.

New Open Source Robots.txt Projects
Last year, Google released its robots.txt parser and matcher to the open source world. Since then, people have used it to build new tools, contribute to the open source library, and release new langua...
Read More