IndexNow - Sustainable Search Engine Indexing
In mid-October of 2021, Microsoft Bing announced they would be launching a protocol for website owners to instantly index their pages, thus hopefully increasing their organic rankings with the respective search engines.
"IndexNow" seems an appropriate name, because it allows webmasters to notify search engines when they have updated their webpages, by the push of a button.
The way it works today, the search engines are looking for this new information continuously, until they find the information they are looking for.
This makes the IndexNow protocol a potential dream for Search Engine Optimizers (SEOs) - meant to maximize the efficiency of indexing, causing web pages to be displayed in the search results pages.
The IndexNow protocol can also possibly minimize the wasted energy of crawler robots looking for updates across the world wide web, which takes a toll on our environment.
Wait, what, crawler bots? They are computer bots that analyze (ie. crawl) web pages for useful information that people search for. If you want to learn more about crawling and indexing, we recommend this article: How Search Engines Work.
Impact on Search Engine Optimization
For someone who wants their website visible in the search results, such as an SEO professional or website owner, this is also a great advantage to have their data shared faster and easier.
There are already some protocols in place for important and time-sensitive pieces of information, such as Google’s Indexing API for Job Posting and Live Streaming.
But what if you could tell the search engine immediately when you posted a guide on how to pick your next electrical bike, or what fantastic new products your industrial technology company has just released?
Experienced SEOs know that sometimes these things take time, and sometimes they take no time at all.
Director of Strategic Integrations at Duda (an award-winning website builder service), Russ Jeffrey, believes the traditional approach to search indexing using sitemaps is outdated, and that we should look towards the future and be able to push content to the search engines for them to crawl it.
I think all SEOs would agree that having a submit button and knowing that you have done your part would be relaxing - to say the least.
“World Wide Waste”
You may not be aware of this, but every time you use the internet for whatever reason, you are adding to the global pollution by cause of energy usage.
In his 2020 book “World Wide Waste”, Gerry McGovern gives some examples:
- 231 million trees would need to be planted to deal with the pollution caused as a result of the data US citizens consumed in 2019.
- 16 million trees would need to be planted to offset the pollution caused by the estimated 1.9 trillion yearly searches on Google.
What the IndexNow resolves is the need for a search engine to crawl the web relentlessly, looking for updated information or new URLs to submit to the results pages.
This happens automatically, without any signaling from websites that they have published or updated anything at all, resulting in a waste of energy from all the computing power that runs continuously for these purposes.
The idea is as simple as it is splendid: You decide when new information or pages are to be submitted, thus saving the crawler bots time and energy.
Advantages of using IndexNow
- The main purpose of the initiative is to reduce the time for new content to be discovered and indexed by search engines. As we mentioned earlier, the spider bots that crawl relentlessly will be replaced by webmasters who then submit their content instead.
- Unlike the other search engine indexing protocols where you are not even guaranteed that the bots will crawl your pages, IndexNow provides a better chance for your URLs to be crawled.
- Since IndexNow relies on pushing information rather than having to pull the data from servers, this feature puts you in charge of telling the search engines when you have updated your content.
- CloudFlare’s Crawler Hints supports the IndexNow protocol. If your website is hosted on this Content Delivery Network (CDN) you’ll be just a click away from letting the search engines know when you have new content.
- Popular CMS plugins for WordPress (2 million sites) and Duda (~1 million sites) support IndexNow, alongside the RankMath SEO Plugin (1.1 million sites) and the Yoast Premium SEO plugin (12 million sites).
- The protocol also affects the server requests by lessening the web crawlers' activity, possibly increasing your server's efficiency.
- Last but not least, the positive effect on energy usage by the reduced need for crawling will surely have an impact on the total use of power demand from search engine indexing.
Who can use IndexNow?
From the IndexNow documentation page, developers can implement the protocol as described.
But if you are not a developer or do not want to rely on one because [insert joke here], you can rest assured that many large platforms are embracing this protocol.
CloudFlare and Akamai are using it already, meaning that if you use these content delivery networks (CDN) you can also take advantage of its features.
Content Management Systems (CMS) such as WordPress and Duda have a plugin that supports IndexNow. While Duda has an automatic use of the protocol, Wix is rumored to tag along with the initiative.
Initially, Joost de Valk (CEO) of Yoast seemed a bit skeptical, but in June he announced that Yoast Premium will feature the IndexNow protocol.
Progress and news since the release
- Microsoft announced last year that websites such as eBay, GitHub, MSN, and LinkedIn have said they are planning to migrate to IndexNow, hoping that smaller websites will follow suit to keep up with indexing.
- A spokesperson from Google announced they would be testing IndexNow for sustainability, in early November 2021.
- Today, Microsoft reports that the index is being used by more than 80,000 websites, 60.000 of which are used via CloudFlare (30.11.2021).
- In late December, WordPress announced a plug-in for their platform to be used with IndexNow, which is great news considering more than one-third of all websites on the Web are made with WordPress.
- An update to the protocol was launched in January 2022, where URLs added via Bing Webmaster tools would be automatically submitted to Yandex, ie. co-sharing URLs between the two search engines.
- RankMath and Duda added to the ranks of plugins that support IndexNow alongside WordPress; the number of IndexNow integrations could be reaching millions of websites according to Bing.
With Google Search possibly joining in on the functionality of IndexNow, the number of sites using the protocol would grow substantially.
Most of all, we hope that the initiative does improve Search Engine indexing and provide us all with a more sustainable world wide web.