Create Visual Sitemaps

How to Create Visual Sitemaps

Option 1: Create from URL

Enter the base URL that you would like to create a visual sitemap for including the http:// or https:// located at the beginning of the url.

Create Sitemaps

Option 2: Create from XML

Create visual sitemaps from xml files that are located on your computer or online via a url.

Option 3:Create from Scratch

Create visual sitemaps using the DYNO Mapper editor to manually add, remove, edit, and set categories of pages without the need to crawl urls or import xmls.

Option 4: Create from Existing

Create visual sitemaps from sitemaps previously created within your DYNO Mapper account.

Advanced Options

create visual sitemaps advanced options

 


 

Follow robots.txt - The robots.txt file allows websites to indicate to web robots like Google robot and DYNO Mapper that you would like them to ignore certain pages. The preference in DYNO Mapper is on by default and can be switched off in the Create from URL advanced options. The robots.txt file name must be written in lowercase and placed in the root directory of your website and follow The Robots Exclusion Protocol - link to http://www.robotstxt.org/robotstxt.html

Follow subdomains - A sub-domain is a domain that is part of a larger domain. For example, north.example.com and south.example.com are subdomains of the example.com domain.

Ignore trailing slash - If website code is hand written, it may not be consistent and also contain certain problems even though the page will still appear to work properly in a web browser. This option will allow the DYNO Mapper to be more forgiving and overlook problems like a web browser does if on is selected or be less forgiving if no is selected.

Ignore query string - A querystring is info within a url that follows a '?' - for example www.example.com/index.html?thisis=thequerystring. The option is important if you have dynamic sections within your website like calendars. Dynamic sections within a website that use query strings can consist of many pages that you might want to omit from your sitemap.

Unique page title - If your website contains pages that each have a unique title, selecting yes for this option will give you a very accurate crawl. Some content management systems can refer to the same page by quite a few different urls which causes duplicates within your visual sitemap.

Ignore scheme - Some websites have duplicate pages prepresented because they allow multiple schemes of an individual page. For example, if you do not ignore scheme of www.example.com - the pages http://www.example.com & https://www.example.com would be considered two different pages.

Arrange links by URL - If you would like DYNO Mapper to arrange your sitemap based on the parent/child relationship used in the url, keep the option on.

Domain as root - If you are crawling a sub-domian, this setting lets you set that sub-domain as the root of the sitemap tree.

Include Redirects - A URL Redirect, also called URL Forwarding, is a technique for making a web page available under more than one URL address. When a web browser attempts to open a redirected website URL, a page opens with a different URL. Use this feature to display all redirects in the sitemap visually.

Include Bad Links - A bad link is a 4xx error where the intended page that you opened in your browser does not work. Use this feature to display all of your bad links in the sitemap visually.

Include PDFs - Use this feature to display domain level PDFs in the sitemap visually.

Sart URL as Root - When crawling a subsection of a website, you will want to use this feature to set the Root of the sitemap to the starting URL. If this setting is OFF, the domain’s home page will be used as the root by default.

Website Authentication - Use website authentication if you would like to include private front end pages within your sitemap from popular CMS systems. If your favorite CMS system is not in the list, please send us an email to inquire if we can add it CMS systems that we support.

Basic Authentication - Use HTTP Basic authentication for authentication that is static, standard HTTP headers.

 

Website Authentication

If you need to include private sections of a website which require a login, you will want to use website authentication which gives the software the ability and access to add these private pages to your sitemap. 

Supported CMS Systems

We have authentication plugins for Wordpress, Joomla, Drupal, and Expression Engine that enables a quick and easy way to authenticate.

  1. Select CMS in System Dropdown
  2. Add Username Password, Password, and Login URL
  3. Click Start Crawling

Web authentication is not necessary if you don’t have private pages to crawl.

Custom System Login

If you are using a CMS system not listed above or if your website or application’s CMS is custom, you can authenticate using the Custom System Login feature to crawl private pages.

When crawling a password-protected website, there are some practices you should follow.

  • Use a read-only account when crawling - When you’re crawling through your protected site, you’re going to be picking up on things that need to be changed. You may even be tempted to fix them yourself as you go along, but you have to remember the reasons for your crawling the website in the first place. One of them is to make a list for the developers to sort out. After all, they’re the professionals, and you’ve hired them to deal with these kinds of issues. If you lack the expertise, you may end up doing more harm than good. By using a full administrative username and password when crawling your website, you’re also giving the crawler full access to the entire admin section. While the best crawlers won’t do anything of the like, with full administrative access, you risk having your crawler making changes to the site’s themes and plugins, even deleting posts. This is why it is imperative that you setup a read-only account for crawling your website.

  • Always exclude your admin pagesg - This carries on from what we’ve said in the above point: the read-only account you’ve set up for crawling your website must exclude the administrative back-end pages and folders to help you avoid making those undesirable changes to your website. For WordPress, you’ll want to exclude the entire /wp-admin/ section; for Joomla, the same applies for the /administrator/ section.


Create a Custom System Login

  1. Click Create from URL
  2. Click Show advanced options
  3. Click Add/Edit custom system
  4. Add Custom System Login form details
  5. Test Login
  6. Save
  7. Click the Back to Form Button
  8. Add the URL of the website or applications
  9. In Website Authentication, select the new system that you created in Select System. Username, Password, Login URL do not need to be entered again if you added them as the default in the new system. If you are using a different user account than the one created with the plugin, you can enter these detail in the fields for Username, Password, and Login URL.
  10. Click Start Crawling

 

create sitemaps custom system login


Other Options

Maximum Pages. Enter a number to limit the number of pages crawled before stopping the crawler. This is not exact because some pages might not be included in the sitemap by default or certain pages are identified as duplicates of one another.

Maximum Depth. Enter a number to limit the number of levels crawled.

 


  

Skip Links

Use the Skip links feature to skip pages during your website crawl that match a certain format. This feature is great for removing unnecessary pages from your sitemap build and content inventory.

  1. Show advanced options
  2. In Other Options, in the Skip Links row - click Add/Edit Links
  3. Enter a part of the url
  4. Select a matching rule
  5. Press Enter/Return
  6. Repeat if you have other URLs that you would like to skip

You can copy and paste line statements if you click the Switch to Text Editor.

create sitemap skip links

  


  

Include Links

Use the Include links feature to only Include pages that match a certain format after the crawl has completed. This is the most accurate method.

Add/Edit Links

  1. Show advanced options
  2. In Other Options, in the Include Links row - click Add/Edit Links
  3. Enter a part of the url
  4. Select a matching rule
  5. Press Enter/Return
  6. Repeat if you have other URLs that you would like to include

You can copy and paste line statements if you click the Switch to Text Editor.

create sitemap skip links

Select Sitemap Folder/Item

  1. Show advanced options
  2. In Other Options, in the Include row - Click Select Sitemap Folder/Item
  3. In the dropdown, select a previously crawled version of the same website or application
  4. Select the sections that you want to include.
    1. Mac
      1. Select an item: Click the item
      2. Select multiple items: Press and hold the Command key, then click the items.
      3. Select multiple items that are listed together: Click the first item, then press the Shift key and click the last item. All items in between are included in the selection.
    2. PC
      1. Select an item: Click the item
      2. Select multiple items: Click the first file or folder, and then press and hold the Ctrl key. While holding down the Ctrl key, click each of the other files or folders you want to select.
      3. Select multiple items that are listed together: Click the first file or folder, and then press and hold the Shift key. While holding Shift, click the last file or folder.
  5. Click Apply
  6. Click Start Crawling

 


  

Exclude Links

Use the Exclude links feature to only exclude pages that match a certain format after the crawl has completed. This is the very accurate method to exclude items.

Add/Edit Links

  1. Show advanced options
  2. In Other Options, in the Exclude Links row - click Add/Edit Links
  3. Enter a part of the url
  4. Select a matching rule
  5. Press Enter/Return
  6. Repeat if you have other URLs that you would like to exclude

You can copy and paste line statements if you click the Switch to Text Editor.

create sitemap skip links

Select Sitemap Folder/Item

  1. Show advanced options
  2. In Other Options, in the Exclude row - Click Select Sitemap Folder/Item
  3. In the dropdown, select a previously crawled version of the same website or application
  4. Select the sections that you want to exclude.
    1. Mac
      1. Select an item: Click the item
      2. Select multiple items: Press and hold the Command key, then click the items.
      3. Select multiple items that are listed together: Click the first item, then press the Shift key and click the last item. All items in between are included in the selection.
    2. PC
      1. Select an item: Click the item
      2. Select multiple items: Click the first file or folder, and then press and hold the Ctrl key. While holding down the Ctrl key, click each of the other files or folders you want to select.
      3. Select multiple items that are listed together: Click the first file or folder, and then press and hold the Shift key. While holding Shift, click the last file or folder.
  5. Click Apply
  6. Click Start Crawling
Create Interactive Visual Sitemaps

Discovery has never been easier.

Sign up today!