Search Console and Indexing API to Manage Indexing
Resolving Indexing Issues: Search Console
When facing indexing issues, you must utilize the Search Console fully. Search Console provides several features to fix indexing issues.
Check indexing status
First, you need to check your indexing status using Search Console. Click Pages under the indexing section on the left sidebar. Search Console explains "Why pages aren't indexed".
There are several issues that might be raised here, such as:
- Page with redirect
- Excluded by ‘noindex’ tag
- Alternate page with proper canonical tag
- Blocked by robots.txt
- Crawled - currently not indexed
- Not found (404)
- Discovered - currently not indexed
When you click one of the issues, you can see how many URLs are facing the same issue.
Search Console Messages and Validate Fix
When Search Console detects issues, it sends a message to you through Gmail and on its platform. You can check the history of messages by clicking the alert icon on the top left.
Open one of the messages and click the Open indexing report button. You'll then be able to access the issue report page explained above.
When you solve the issue, you need to press the VALIDATE FIX button on each issue page. Google will start to validate whether the problem is fixed.
URL inspection and request indexing
Even though no issue is detected, your pages may not be indexed if they are relatively new. Usually, it takes a few weeks for new pages to be indexed. This is because Google crawlers have a crawling budget (how many pages Google can crawl for each website daily). It takes time for the crawlers to crawl all pages on your website.
You can also make requests through the Search Console to ensure crawlers crawl your pages. In the URL Inspection box, type the page URL.
Search Console provides detailed information on the page indexing status. If the page needs to be indexed, press the REQUEST INDEXING link.
Indexing requests also have a quota for each day. If you have many pages to request indexing, you need to do it on different days.
Indexing API to Accelerate Indexing
To accelerate indexing for many web pages, consider using Indexing API. Indexing API directly notifies the search engine about new or updated web pages for faster indexing.
Implementing the Indexing API requires technical knowledge; if you are not an engineer, you need to work with someone who can handle the API. Google Search Central provides a detailed explanation of the Indexing API: Google Search Central Indexing API.
Here are the key steps to implement Indexing API.
- Create a project on GCP (Google Cloud Platform)
- Create a service account on GCP (Google Cloud Platform) and download a private key (JSON)
- Add the service account as a site owner through Google Search Console using the Service account ID (e.g., my-service-account@project-name.google.com.iam.gserviceaccount.com)
- Get an access token (set up OAuth token using libraries)
- Use the Indexing API - Send a list of URLs for indexing requests with type (
URL_UPDATED)
The Indexing API sets the default daily per-project quota at 200. If your website has more than 200 pages for indexing, you need to run the Indexing API code multiple times on different days.
Below is an example of a result of the Indexing API. In this case, the indexing speed was initially slow (only <30% of pages were indexed) until the Indexing API was executed. After running the Indexing API code two times, this site got almost >99% of pages indexed out of 443 pages.
Conclusion of Chapter 11
Mastering technical SEO is essential for enhancing your website's visibility and ensuring that your content is accessible to search engines. This chapter has provided you with essential strategies and tools to ensure your web pages are indexed correctly and efficiently, addressing any indexing issues you may encounter along the way.
Key Strategies for Effective Indexing:
- Implementing a sitemap.xml: Create and submit a sitemap.xml to enhance search engine indexing. It guides crawlers to important pages on your site, ensuring comprehensive content discovery.
- Managing the noindex Tag: Employ the noindex tag to exclude pages that should remain private or are not intended for search results, thereby improving the quality and relevance of indexed content.
- Configuring robots.txt Properly: Ensure your robots.txt file is properly configured to direct search engine crawlers effectively. It should permit access to essential pages and restrict areas that do not contribute to your search engine visibility.
- Applying Canonical Tags: Use canonical tags to address duplicate content issues by identifying the primary version of a URL. This helps consolidate ranking signals and prevent confusion among similar pages.
- Effective Redirection Management: Manage redirections meticulously, especially permanent ones, to maintain SEO value and prevent traffic loss when transitioning from old pages to new ones.
Advanced Tools for Indexing:
- Google Search Console: . The Search Console allows you to submit pages for indexing and track the success of these submissions. Utilize this tool to monitor indexing status, identify and fix indexing issues, and optimize the visibility of your pages
- Indexing API: The Indexing API is a valuable tool for speeding up the indexing process for websites that frequently update or add new content. It directly notifies Google about new or updated pages, which is particularly useful for large sites or those with critical real-time content.
By implementing these technical SEO techniques, you ensure that your site is correctly indexed and stands a better chance of ranking well in search engine results. This improves your site's discoverability and enhances user engagement by ensuring that your audience can find the relevant and high-quality content they are searching for.
You can also learn this topic offline. Click 
.