I feel HTML sitemaps are more for site navigation than for Google; they provide better navigation for users. Personally, I've kept my HTML sitemap on the 404 page so that if someone lands there, they can navigate through the entire website at a glance.
On December 14, 2023, I added an HTML Sitemap to my website because only a handful of pages had the status "Crawled/Discovered - Currently Not Indexed.
It depends LOL. Useful to get on the "first page" for competitive one word search queries? Probably not.
Useful for users, indexing and spreading the authority all over your site? Maybe even more than XML sitemaps especially if you don't use CSS on the latter.
I have written a large article for Search Engine Land earlier this year on why and how to use HTML sitemaps, with examples from Apple and the NYT (the biggest corporation in the world and the most important publication globally) both using several HTML sitemaps!
For search engines and traffic, no probably not. But we don't do what we do just for search engines. They're helpful users and are requirement to WCAG compliant: https://www.w3.org/TR/WCAG20-TECHS/G63.html. By not having one you could lose out on valuable sales/conversions.
Just last week I implemented an HTML sitemap for an eCommerce store that's been struggling to get faceted navigation pages indexed. Their store uses client-side JS to generate the faceted nav and it's been a bit of a nightmare. It's a wider project their devs are working to resolve but in the interim, we put in place an HTML sitemap that includes the facets we need indexing. Within a few days we'd started to see some of these indexed. In complex situations, they definitely still have their place to get URLs discovered by search engines.
Good example. And maybe an XML sitemap could have worked here too. But facets are notoriously difficult to deal with, you might win the battle to get nice unique URLs, but then the side nav functionality is all JS or whatever (links not exposed on load). So situational sitemaps make sense here.
Just this week I recommended a client kill their HTML Sitemap(s). They have thorough XML sitemaps and place and good-enough global navigation.If your primary goal as an SEO is "more traffic no matter what" then surfacing more old and not-helpful pages makes sense.But if you want valuable traffic, to connect your brand with potential customers, then HTML sitemaps can create a terrible user experience —  out-of-stock products never-to-return and soft 404 tag pages aren't helping grow your business in a meaningful way.The NYT example is an excellent use for it. It doesn't need to be called a sitemap though. It's a thoughtful UI for browsing the paper's archive. It just happens to function like a sitemap.
Are real people finding them useful. If so, they have value.
As mentioned, if your navigation structure is poor or it is hard to provide navigation to some pages, it may help in crawling. Xml sitemaps don't do well for pages that have no links to them.
Very seldom. There are so many ways Google can find your content, so what is left that is actually not known by Google (except giant sites with complex facets). But even with ecom facet sites, are you really going to put ever combo in an HTML sitemap? I argue that for some sites XML sitemaps are not that needed anymore. If you build it right from Day 1. Balance also the maintenance needed to update these HTML sitemaps, vs trying to get dev to automate one, and it can become more trouble that it's worth. If anything, use it to link to the most common, needed content on your site, and let users rely on a good site search to find more obscure things.
HTML sitemap it's the last resort of solutions.It means that we can't have on the website a hierarchical internal links distribution, with different levels of "sitewide" based on the importance of the landing pages.- TOP Level links: in all the pages- MIDDLE1 Level: in all the pages of a macro category
- MIDDLE2 Level: in all the pages of a category
- MIDDLE3 Level: in all the pages of a sub-categoryAnd so one.
I'll go against the grain here and say it's not super useful anymore, at least on websites I've access to.I'm always looking at bot visits and noticed that HTML Sitemaps barely get visits.
Sure, this varies case by case and you can only know if you test it (e.g. are pages getting more bot visits, getting indexed, etc), but I voted "no" my tests show HTML Sitemaps being ignored. IMO they are more useful ways to display links to users and bots.
I created an HTML sitemap last Friday because many pages are listed as "discovered - currently not indexed / crawled - currently not indexed" within GSC. I'm curious about the results.
Hello there @info! If they've been crawled/discovered but not indexed, then the issue might have more to do with the content quality, helpfulness, uniqueness, expertise of these pages. I would highly advise you to take a look at that too!
I love an HTML sitemap. Truly love one. It doesn't have to be flashy or exciting, it can just be linked in the footer and link to the most important pages if Google is having a bit of trouble reaching them. PINTEREST (!!!) still uses one for https://www.pinterest.ca/html_sitemap/boards_a.html
Fundamently search is a failure of browse, everyone knows how to browse - you do it in a store, search is hard. As you can see Google is doing more and more to introduce faceted navigation on search. HTML sitemaps, done right can boost conversions, aid crawling and label entitites.
That's great @Mersudin Forbes. And if the HTML sitemap is implemented as an integral part of the UX that users actually enjoy engaging with then it's no longer relegated as a fallback. Instead, it plays an active role in the vanguard of making content crawlable.
For most websites it would be a workaround for a non optimal site structure as said by others already. The main UX elements should be sufficiently structured in 95% of the domains.
However, there may be websites that are literally too big to have all pages covered by the default navigation structures. Think about a classifieds platform selling everything you can imagine. It may be beneficial to have a secondary navigational structure purely for the search engine bots.
Then we have the challenge of seasonality. If you want to have top 3 rankings during the festive season, you may want to start pushing to pages already in August. However, from a conversions perspective it doesn't make sense to add Christmas of Black Friday landing pages into the main navigation URL. I think we will get some angry stakeholders into the SEO meetings. Having a secondary navigational path available to still push value to those pages makes sense.
To add to the POV of having HTML sitemaps as a backup of XML sitemaps: that is a smart note. To add to that: XML sitemaps don't pass linkvalue. HTML sitemaps, especially linked via a sitewide link will be able to distribute PageRank across pages.
@patrick You mention you expect you where hit by the HCU updates? Usually that means the opposite of the problem you can solve with having HTML sitemaps: probably all your pages are easy findable and crawlable by Googlebot, but they just don't like the quality of it. Using a sitewide widget to push Google to crawl updated (and improved) pages may be a smart thing to do though.
They definitely are! Their importance increase if your site navigation is not yet optimized, you’re not able to generate XML sitemaps and is a large site to crawl. Just check out @Jan-Willem example here https://x.com/jbobbink/status/1731676101196370090
100% yes. It can certainly help with indexing and spreading some link equity around, especially to pages you may not want to necessarily link to in your navigation or other prominent places. NY Times is a great example of utilizing them effectively. In their footer is a link to "Site Map". From there, you can access any article they have ever published going back to 1851. The way they use it, none of their articles are more than 5 clicks away from their home page.
AirBnB uses them effectively too, but in my opinion in a much more spammy way. However, it works.
Get the latest in SEO every week by subscribing to SEOFOMO now
I feel HTML sitemaps are more for site navigation than for Google; they provide better navigation for users. Personally, I've kept my HTML sitemap on the 404 page so that if someone lands there, they can navigate through the entire website at a glance.
On December 14, 2023, I added an HTML Sitemap to my website because only a handful of pages had the status "Crawled/Discovered - Currently Not Indexed.
My main menu IS my HTML sitemap
On the other hand, if your main menu sucks and you cannot change it, then do the HTML sitemap.
It depends LOL. Useful to get on the "first page" for competitive one word search queries? Probably not.
Useful for users, indexing and spreading the authority all over your site? Maybe even more than XML sitemaps especially if you don't use CSS on the latter.
I have written a large article for Search Engine Land earlier this year on why and how to use HTML sitemaps, with examples from Apple and the NYT (the biggest corporation in the world and the most important publication globally) both using several HTML sitemaps!
See here: https://searchengineland.com/html-sitemaps-seo-ux-when-how-394763
On my business site the 2nd most visited page is the html site map.
For search engines and traffic, no probably not. But we don't do what we do just for search engines. They're helpful users and are requirement to WCAG compliant: https://www.w3.org/TR/WCAG20-TECHS/G63.html. By not having one you could lose out on valuable sales/conversions.
Just this week I recommended a client kill their HTML Sitemap(s). They have thorough XML sitemaps and place and good-enough global navigation. If your primary goal as an SEO is "more traffic no matter what" then surfacing more old and not-helpful pages makes sense. But if you want valuable traffic, to connect your brand with potential customers, then HTML sitemaps can create a terrible user experience —  out-of-stock products never-to-return and soft 404 tag pages aren't helping grow your business in a meaningful way. The NYT example is an excellent use for it. It doesn't need to be called a sitemap though. It's a thoughtful UI for browsing the paper's archive. It just happens to function like a sitemap.
Are real people finding them useful. If so, they have value.
As mentioned, if your navigation structure is poor or it is hard to provide navigation to some pages, it may help in crawling. Xml sitemaps don't do well for pages that have no links to them.
I think they can be useful when there might be suspected issues with the XML sitemap - sort of a way to workaround issues.
Very seldom. There are so many ways Google can find your content, so what is left that is actually not known by Google (except giant sites with complex facets). But even with ecom facet sites, are you really going to put ever combo in an HTML sitemap? I argue that for some sites XML sitemaps are not that needed anymore. If you build it right from Day 1. Balance also the maintenance needed to update these HTML sitemaps, vs trying to get dev to automate one, and it can become more trouble that it's worth. If anything, use it to link to the most common, needed content on your site, and let users rely on a good site search to find more obscure things.
HTML sitemap it's the last resort of solutions. It means that we can't have on the website a hierarchical internal links distribution, with different levels of "sitewide" based on the importance of the landing pages. - TOP Level links: in all the pages - MIDDLE1 Level: in all the pages of a macro category
- MIDDLE2 Level: in all the pages of a category
- MIDDLE3 Level: in all the pages of a sub-category And so one.
I'll go against the grain here and say it's not super useful anymore, at least on websites I've access to. I'm always looking at bot visits and noticed that HTML Sitemaps barely get visits.
Sure, this varies case by case and you can only know if you test it (e.g. are pages getting more bot visits, getting indexed, etc), but I voted "no" my tests show HTML Sitemaps being ignored. IMO they are more useful ways to display links to users and bots.
I created an HTML sitemap last Friday because many pages are listed as "discovered - currently not indexed / crawled - currently not indexed" within GSC. I'm curious about the results.
I love an HTML sitemap. Truly love one. It doesn't have to be flashy or exciting, it can just be linked in the footer and link to the most important pages if Google is having a bit of trouble reaching them. PINTEREST (!!!) still uses one for https://www.pinterest.ca/html_sitemap/boards_a.html
Fundamently search is a failure of browse, everyone knows how to browse - you do it in a store, search is hard. As you can see Google is doing more and more to introduce faceted navigation on search. HTML sitemaps, done right can boost conversions, aid crawling and label entitites.
Yes. with caveats of size and complexity of the site and the potential for multiple HTML sitemap scenarios on much larger sites.
The shorter answer is to make your pages easily discoverable through user-focused navigation and internal linking.
Guess that answer wasn't much shorter 🤣Â
That's great @Mersudin Forbes. And if the HTML sitemap is implemented as an integral part of the UX that users actually enjoy engaging with then it's no longer relegated as a fallback. Instead, it plays an active role in the vanguard of making content crawlable.
I think it should exist as a fallback, just like rendering menus via CSS without JS.
There's no reason why HTML sitemaps can't also be UX friendly.
For most websites it would be a workaround for a non optimal site structure as said by others already. The main UX elements should be sufficiently structured in 95% of the domains.
However, there may be websites that are literally too big to have all pages covered by the default navigation structures. Think about a classifieds platform selling everything you can imagine. It may be beneficial to have a secondary navigational structure purely for the search engine bots.
Then we have the challenge of seasonality. If you want to have top 3 rankings during the festive season, you may want to start pushing to pages already in August. However, from a conversions perspective it doesn't make sense to add Christmas of Black Friday landing pages into the main navigation URL. I think we will get some angry stakeholders into the SEO meetings. Having a secondary navigational path available to still push value to those pages makes sense.
To add to the POV of having HTML sitemaps as a backup of XML sitemaps: that is a smart note. To add to that: XML sitemaps don't pass linkvalue. HTML sitemaps, especially linked via a sitewide link will be able to distribute PageRank across pages.
@patrick You mention you expect you where hit by the HCU updates? Usually that means the opposite of the problem you can solve with having HTML sitemaps: probably all your pages are easy findable and crawlable by Googlebot, but they just don't like the quality of it. Using a sitewide widget to push Google to crawl updated (and improved) pages may be a smart thing to do though.
It's a workaround. I don't suggest it as a solution but as a surrogate when we can't improve navigation optimizations.
PS: in recent weeks GSC has given me "couldn't fetch" error when I try to add an XML sitemap to a new GSC profile
I've personally found them to be highly effective, especiialy on getting hard to crawl yet important PDP pages on ecommerce websites.
Yes, but also think it's a good opportunity to create good cornerstone pages reaching out to all areas in a more coherent manner (and better for UX)
Totally. 100%. Always.
They are still a wonderful alternative way and backup option for when the XML sitemaps fails, especially on huge websites.
They definitely are! Their importance increase if your site navigation is not yet optimized, you’re not able to generate XML sitemaps and is a large site to crawl. Just check out @Jan-Willem example here https://x.com/jbobbink/status/1731676101196370090
100% yes. It can certainly help with indexing and spreading some link equity around, especially to pages you may not want to necessarily link to in your navigation or other prominent places. NY Times is a great example of utilizing them effectively. In their footer is a link to "Site Map". From there, you can access any article they have ever published going back to 1851. The way they use it, none of their articles are more than 5 clicks away from their home page.
AirBnB uses them effectively too, but in my opinion in a much more spammy way. However, it works.