We’ve had an SEO audit done on our community. There are a number of improvements we’ll present here as ideas. This one is about linking.
Google has shown to ignore robots.txt on more than on occasion, so simply blocking these in robots.txt is not a surefire solution. A more robust solution is relying on rel=nofollow and robots meta tags to guide crawler / indexation behaviour. Hence we propose a few ideas that would be beneficial in assisting crawler behaviour and guiding crawlers to the most relevant content. Ideally these would be integrated as toggle-able functions in order to allow users to customize crawl behaviour based on their needs.
- Sorting urls - Allow a rel="nofollow" to sorting <a hrefs>, and Add meta robots noindex/nofollow directive to pages with ?sort query present. This'll stop crawlers from having to go over multiple iterations of the same pages.
- PostId - Allow a rel="nofollow" to certain <a hrefs>, namely for links featuring ?postid queries and add meta robots noindex/nofollow directive to pages with ?postid query present.
- Member pages - Allow a rel=nofollow attribute for URLs referring to them
- Search pages: Allow a meta robots noindex/nofollow directive to search pages.
- Pagination - Add a "jump to last page" button as part of pagination
