Sitemaps: How to Manage, Monitor, and Maintain Your Sitemaps

A sitemap is a file that contains a list of the web pages that are on a website. It will relay to Google and other search engines the organization of the website's content. Googlebot and other search engine crawlers will read the sitemap so that they can crawl your site with more knowledge. These sitemaps can also hold important metadata regarding the pages that are listed within the sitemap. Metadata is any information about a web page—like how often it is changed, its importance in relation to the other pages on the site, when it was last updated, and more. Sitemaps can be used to give Google metadata that is associated with particular types of content on the web pages like videos, images, and mobile content. A video entry on a sitemap can indicate the video's running time, age rating, and category. An image entry on a sitemap may tell the type, subject matter, and license.

Do I Need One?

If all of a website's pages are linked properly, Google's web crawlers are able to discover most of the content on their own. Even if this is the case, having a sitemap will only improve the crawling of the site, especially if the site is very large, has archive pages that are not linked among each other, if it is new and does not have many external links, and if it has a lot of rich media content.

• If the website is very large, you will run the risk of the crawlers overlooking some of the new pages or those that have been recently updated.
• When the site has many archival pages that are not linked together in any organized way, it is suggested to list them in a sitemap so that Google will not overlook them as they do not naturally reference each other.
• When there are only a few external links, Google may not discover some of the pages. This is because web crawlers discover the web by pursuing links that point from one page to the next.

It should be noted that even when you employ a sitemap, it does not promise that all of the items will be indexed and crawled. This is because Google's process relies only on intricate algorithms to do the crawling. In almost all cases, there is never ever a penalty for having a sitemap, and websites do benefit from having one.

How to Build a Sitemap

1. You will first need to indicate which pages on the site you would like to be crawled by Google, and then determine canonical version for each of the pages.
2. Decide which sitemap format is best for your use (XML, RSS, Text, Google sites), and you may then create a sitemap from scratch manually, or choose from any number of Google's Sitemaps testing tool.
3. Then, test the sitemap using Google's Sitemaps testing tool.
4. Once all of the bugs have been fixed, you will need to make your sitemap available to Google. To do this, you will add it to your robots.txt file and then submit it through Search Console.

Sitemap Formats

• XML sitemaps are the standard version. You may use all extensions with an XML sitemap.
• An RSS sitemap should be used with a blog that has an RSS or Atom feed. You will be able to submit the feed's URL as the actual sitemap. Today, almost all blog software will create the feed automatically, but know that the feed will only provide information on the most recent URLs. Google will accept RSS feed as well as mRSS (mobile RSS) feed.
• Text sitemaps can be employed if the sitemap is only made of URLs. Please know that there should only be one URL per line. The file will need to use UTF-8 encoding, should contain ONLY URLs, and the file canebe named anything that you would like, as long as it has the .txt extension.
• If you have used Google Sites to create and verify a website, Sites will automatically create a sitemap. It cannot be modified, but it can be submitted to Google if you wish to read the report data. Do know that in order for it to be displayed properly, you will need to have no more than 1,000 pages on one single sub-directory.

Guidelines for Your Sitemap

• You should be using fully qualified and consistent URLs.
• URLs should not include session Ids.
• Indicate URLs that are a translated version.
• The sitemap files have to be UTF-8 encoded.
• Break the large sitemaps up into multiples. If you do this, employ a sitemap index file.
• Get to know the Webmaster Guidelines as well as the SEO Starter Guide to familiarize yourself with the area of work.

Whatever format you choose to employ, you will need to remember that the sitemap has to be under 10MB uncompressed, and under 50,000 URLs. If it is over either of these, you will need to break it up into more than one sitemap. Then, you have the choice to create a sitemap index file that will indicate the list of sitemaps, and then submit this single file to Google. Either way will be okay.

Managing a Sitemap with Sitemaps Report

This page will display a list of the sitemaps that a user has submitted to the Search Console. Any sitemaps that were not submitted using this tool will not be displayed. When viewing the table, click on a sitemap to get more information about it, including any errors that have occurred. In the case that the file is a sitemap index, clicking on it will display a list of the sitemaps contained within the index file. Open the Sitemaps Report here. 

If the sitemap is not listed within the report, you will need to check the settings. This includes the website's preferred domain, who submitted the sitemap, and if you submitted the sitemap using a specific report.

Testing the Sitemap

All sitemaps should be tested before they are submitted. If you are going to test an unsubmitted sitemap, from the report landing page you will need to click on Add/Test Sitemap. Then, enter the URL for the sitemap into the box and then click on Test. After the test has completed, click on Open Test Results, and check for errors that need to be fixed. After the errors have been fixed, you may submit the sitemap.

If you are aiming to test a previously submitted sitemap, you will open the details for the previously submitted page and simply click on Test.

Submitting the Sitemap

If it's your first time submitting the sitemap, you will need to test it as described previously. Select the Add/test Sitemap button. You will then need to enter a URL path into a text box. This URL needs to be relative to the site root that is defined for the property. Then, select Submit. After this, you will refresh your browser to see that the sitemap has been added to the list of sitemaps. Select this new sitemap on the list to open the details page and look for any errors or warnings that have developed within the sitemap or the URLs that it contains.

It should be noted that Google takes some time to process a new sitemap that has been submitted. Google never promises to crawl or index each URL in the sitemap because of their dependence on algorithms to make the crawling decisions.

Resubmitting the Sitemap

If you simply need to resubmit the sitemap, you will open up the Sitemaps report. Then from the table you will select the sitemap that you would like to resubmit. Select the Resubmit sitemap button.

When you delete a sitemap, it will be permanently removed from the Search Console. From the sitemap table, you will select the box next to the sitemaps that you would like to delete. Then, click on delete. NOTE: this does not delete the sitemap from the web host, so Google may still crawl it. If you would like to prevent Google from crawling it, block the access through robots.txt.

Multiple Sitemaps

When you have multiple sitemaps, you will likely be keeping them organized in a sitemap index file so that you can submit them all at once. If you are the owner of many websites, you can ease the process of creating and then submitting sitemaps through creating multiple sitemaps that include URLs for all of the legitimate websites, and then saving this one sitemap in a static location. All of the websites have to be checked out and verified by the Search Console. You can do one of the following:

• Use one sitemap that gathers URLs for more than one website, including those that are from different domains.
• Use individual sitemaps that will all be in one single location.

The cross-site submission will only be allowed to work if the websites have been verified through the Search Console.

In order to host cross-site sitemaps in one location, you will need to ensure that you have verified your ownership of all sites. Then, create a sitemap that has URLs from all of the sites that will be covered. You may opt to make just one sitemap that includes URLs from every site or you may make multiple sitemaps, one for each site. Finally, you will need to use Google's Search Console to submit the sitemap.

Extensions

If your website is one that has a target audience of multiple languages or regions, you are able to use your sitemap to serve Google with the identifier of this. For instance, someone may have a website entirely in English and it is aimed at English speakers all over the world, and there is also the same version of that web site targeted to those who speak German all over the world, as well as those German speakers that live in Switzerland. There will be a unique set of URLs for each of these sites.

Video extensions within sitemaps are a good way to ensure that Google can gather the information about your video that is contained on the site. Interestingly enough, this content may not be discoverable without this extension. Taking the time to submit the sitemap with information about the video, Google will make the video's URL searchable through Google Video. The results will include a thumbnail of the video and any of the information that you have provided about it through the sitemap. You may choose to create a separate sitemap to contain the information for the video, or you can add the information to an existing sitemap.

Using Google's image extension for sitemaps so that any of your images can be discovered through Google Image search results. You will also be able to provide Google more information regarding the image and this will increasingly aid your image in being discovered. You may choose to use separate sitemaps to list the images along with their information, or they can be added to an existing sitemap. You can add information like the location, caption, physical location, title, and license.

Lastly, you can create a mobile sitemap specifically for feature phones by using the sitemap protocol in addition to more tags and name spaces. It should be noted that you are not advised to create this type of sitemap unless you have a particular mobile version of a web page that has been designed for feature phones.

Sitemap extensions are a great way to ensure that your sitemap has the most detailed information about all of the content. When you allow this information to be discovered, you are raising the chance that your site will be discovered by a search engine, and then subsequently presented to anyone that has performed a target search query. Often times, individuals are not aware that the information associated with their images and videos can impact how their content is discovered, but it is just as important as the text content. It simply needs the metadata entered to communicate effectively with the search engine.