It is annoying that if we have to create robots.txt and sitemap.xml manually every time. Luckily, if your personal page is powered by hexo, you can generate it by programmes. I will show you how to realize it.

robots.txt

Install

npm install hexo-robotstxt-multisitemaps --save

Enable

Add hexo-generator-robotstxt to plugins in _config.yml.

plugins:
- hexo-robotstxt-multisitemaps

Add config for robots.txt to _config.yml.

robotstxt:
  useragent: "*"
  disallow:
    - /one_file_to_disallow.html
    - /2nd_file_to_disallow.html
    - /3rd_file_to_disallow.html
  allow:
    - /one_file_to_allow.html
    - /2nd_file_to_allow.html
    - /3rd_file_to_allow.html
  sitemap: 
    - /sitemap.xml
    - /baidu_sitemap.xml

Reference

hexo-robotstxt-multisitemaps

sitmap.xml

Install

$ npm install hexo-generator-sitemap --save

• Hexo 4: 2.x
• Hexo 3: 1.x
• Hexo 2: 0.x

Options

You can configure this plugin in _config.yml.

sitemap:
  path: sitemap.xml
  template: ./sitemap_template.xml
  rel: false
  tags: true
  categories: true

• path - Sitemap path. (Default: sitemap.xml)
• template - Custom template path. This file will be used to generate sitemap.xml (See default template)
• rel - Add rel-sitemap to the site’s header. (Default: false)
• tags - Add site’s tags
• categories - Add site’s categories

Exclude Posts/Pages

Add sitemap: false to the post/page’s front matter.

---
title: lorem ipsum
date: 2020-01-02
sitemap: false
---

Reference

hexo-generator-sitemap

baidu_sitemap.xml

Go to hexo-generator-baidu-sitemap to learn how to set up the environment. The steps are similar.

Test it

Now, by hexo clean && hexo d, you can check the txt and xml files, like

https://anthonydave.top/robots.txt

https://anthonydave.top/sitemap.xml

https://anthonydave.top/baidu_sitemap.xml