Managing Large Websites With Multiple Sitemaps

Last Edited June 8, 2016 by Garenne Bigby in Sitemap Generator

Large Websites and Multiple Sitemaps

XML Sitemaps for Search Engine Indexing

We have had a few users request 1 Million page crawler limits or more so I thought this would be a great topic to address in our blog. Whether you are creating a sitemap to improve your search engine optimization or creating it for website project planning purposes, there are size limitations to consider. According to sitemaps.org, the official protocol for XML sitemaps, they should not contain more than 50,000 website URLs and no larger than 10MB in size.   Google, Yahoo, and Bing also follow these same size restrictions of XML sitemaps. If your site's page count is larger than this, you will need to use multiple sitemaps and link them from a sitemap index file which will identify each of the sitemap files for the search engines for more efficient indexing.

Visual Sitemaps for Project Discovery & Planning

Similarly, a gigantic visual sitemap can also be hard to view if the sitemap contains many pages and levels. For these reasons, DYNO Mapper has several options to manage enterprise level websites, so we wanted to show you our process for creating multiple sitemaps. 

Step 1: Create a Sitemap

Your first step would be to click Create from URL and do a general crawl of your website. It is best to do this even if you think you know your hierarchy of content. Let's face it; you could be missing something right under your nose.  

Crawl your website to discover the hierarchy within your website. After crawling your website, you will want to make a plan for how you are going to split your sitemaps after a general overview of what you are working with. You will want each sitemap to contain no more than 50,000 pages and to contain sections of your website. 

Step 2: Create a Plan

Each Sitemap should be split by your established website URL hierarchy. See the following website example broken into two sitemaps. If your website is organized by URLs that you can instantly group pages together within the next step. 

large website stiemap  
 

Sitemap 1

www.yourdomain.com
www.yourdomain.com/blogs
www.yourdomain.com/products

 

Sitemap 2

www.yourdomain.com/services
www.yourdomain.com/locations
www.yourdomain.com/members


Step 3: Create Multiple Sitemaps

After you have your plan together, it is time to create each of your sitemaps. Click Create from URL and open the Advanced Settings for the Omit Paths field.

create sitemap from url

After entering your URL, use the Omit Paths field to restrict directories. In the example above, you will want to omit services, locations, and members from the first sitemap. 

Ex. \/services - will skip URLs with /services in it. \/services$ - will skip only a URL that ends with /services.

You will want to separate multiple rules with a new line.

 omit paths2

Now create your second sitemap utilizing the omit function but this time you will want to exclude the root URL, directory, blog, and product directories. Entering \/$ will exclude the home page URL from the second sitemap. 

 omit paths3

Step 4: Create a Sitemap Index File

Download both  XML Sitemap files by going to each sitemap menu and clicking Download Sitemap then choosing XML Sitemap. Now all you need to do is upload your sitemaps to your hosting account and link them with a Sitemap Index file.

Multiple Sitemaps

Step 5: Upload Sitemap to Server

After you create a sitemap using DYNO Mapper, the crawler settings are stored so you can just refresh your sitemap by clicking "Refresh Sitemap" in the sitemap menu and re-upload to your server. You have the option to Schedule your sitemap crawl on a weekly or monthly basis. You can manage sitemaps for a 2,500,000 page website with the MOST POPULAR subscription using the above strategy, meanwhile seamlessly using our Content Inventory, Audit, Keyword Tracking, and Accessibility Testing features to create a phenomenal website user experience. 

Garenne Bigby
Author: Garenne BigbyWebsite: http://garennebigby.com
Founder @dynomapper
Garenne Bigby is freelance Chicago developer and founder of DYNO Mapper with over 10 years experience in both agency and freelance roles in design, development, user experience, SEO, and information architecture.

Back

Create Visual Sitemaps

Create, edit, customize, and share visual sitemaps integrated with Google Analytics for easy discovery, planning, and collaboration.

Popular Tags

Search Engine Optimization SEO Create Sitemaps Accessibility Testing Sitemaps UX User Experience Sitemap Generator Content Audit Website Content Audit
Create Interactive Visual Sitemaps

Discovery has never been easier.

Sign up today!