One of the goals of Ardublockly, specially taking in consideration that it is based on web technologies, is to be deployable to offline systems without additional dependencies (other than the Arduino IDE). For its documentation GitHub Wikis was chosen, as it is easy to use, can be viewed and edited directly from the GitHub website, and integrates quite well with its GitHub repository. So a way to provide offline access to this documentation was required, and because the project build procedure is based on Python, a solution using this language was preferred.
MkDocs turned out to be a perfect fit for this purpose. MkDocs is a static site generator, based on Markdown files (GitHub Wiki's default format) and tailored specifically for documentation.
So this article will cover the installation, configuration and automation of static documentation from a GitHub Wiki using MkDocs.
MkDocs can be easily installed using pip (as with any other Python project, it is highly recommended to use virtual environments):
$ pip install mkdocs
Creating a MkDocs project and adding the Wiki content
Navigate to the folder where the project files are to be saved and then execute in the console:
$ mkdocs new my-project
This will create the new
mkdocs project folder, where the documentation content must also be saved, so navigate inside it to clone the GitHub Wiki repository.
In the case of the Ardublockly project, it was saved within a git repository already, so a Git Submodule was used instead of a simple
$ git clone https://github.com/<username>/<repository>.wiki.git
$ git submodule add https://github.com/<username>/<repository>.wiki.git
(Don't forget to replace
<repository> with your own information.)
Configure the MkDocs project
Now the MkDocs configuration file
mkdocs.yml has to be edited to point to the new content folder by setting the
mkdocs.yml file, which would already contain the
site_name property and add the wiki files directory (remember to surround the directory string with single quotation marks):
site_name: 'Documentation Title' docs_dir: '<wiki folder directory>'
Other project properties can also be added, refer to the MkDoc documentation for more information.
MkDocs requires all the markdown files to be generated listed under the
pages property. We can automate this step later using a Python script, so for this specific example the
pages property should be placed at the very end of the
mkdocs.yml file (with all your current files), ideally with a warning:
# It is IMPERATIVE to leave this property to the end without anything after it. # This is because the build file will delete everything after this line and # replace it with newly generated data. pages: - ['index.md', 'Home']
Test the static page generator
MkDocs comes with a handy built-in web server that lets you preview the generated content live.
Navigate to the MkDocs project folder and execute the following command:
$ mkdocs serve
This will serve the pages at:
Make sure there is an
index.md to generate the
index.html file, and check if everything has been rendered as expected.
To deploy the pages you can build the project, into the default
site folder, using the following command:
$ mkdocs build
Create a Python build script
Now that we have a static HTML version of the Wiki markdown files, we can start looking into automating with Python all the steps required to update these files.
First let's predefine some path and repository data, this will depend on your own environment and repository information, so remember to fill the missing information. For this example the build file was included on the parent directory of the MkDocs project folder:
GITHUB_USER = "" WIKI_NAME = "" GITHUB_WIKI_REPO = "github.com/%s/%s.git" % (GITHUB_USER, WIKI_NAME) MKDOCS_FOLDER = "" THIS_FILE_DIR = os.path.dirname(os.path.realpath(__file__)) MKDOCS_DIR = os.path.join(THIS_FILE_DIR, MKDOCS_FOLDER)
For the sake of simplicity a lot of the exception handling and error management has been left out of these code snippets, but a more comprehensive source file is linked in the Conclusion.
Git Pull the latest changes
The first step to update the documentation is to pull the latest changes from the wiki repository. For this task the script will use subprocesses and assumes that Git is installed on the system:
import os import subprocess def pull_wiki_repo(): """ Pulls latest changes from the wiki repo. """ # Set working directory to the wiki repository wiki_folder = os.path.join(MKDOCS_DIR, WIKI_NAME) os.chdir(wiki_folder) # Ensure the subfolder selected is the correct repository PIPE = subprocess.PIPE git_process = subprocess.Popen(["git", "config", "--get", "remote.origin.url"], stdout=PIPE, stderr=PIPE) std_op, std_err_op = git_process.communicate() if not GITHUB_WIKI_REPO in std_op: print("Wiki repository:\n\t%s\nnot found in url:\n\t%s\n" % (GITHUB_WIKI_REPO, std_op)) else: print("Pull from Wiki repository...") subprocess.call(["git", "pull", "origin", "master"])
If using a submodule within a git repository, as the Ardublockly project is, remember to ensure the submodule has been initialised and updated.
Keep in mind that this function changes the current working directory, so any other function that depends on this value (e.g. using relative directories) might be affected.
Edit MkDocs configuration file
As previously mentioned, MkDocs requires all the markdown files to be listed in the
mkdocs.yml file, which is why the
pages property was left at the end of it. The following python function scans
mkdocs.yml until the
pages: line is encountered and it then auto-generates the list:
import os import shutil from tempfile import mkstemp def edit_mkdocs_config(): """ Edits the mkdocs.yml MkDocs configuration file to include all markdown files as part of the documentation. These files are created by default with the '.md' extension and it is assumed no other file extensions are to be linked. """ path_list =  for file in os.listdir(os.path.join(MKDOCS_DIR, WIKI_NAME)): if file.endswith(".md"): path_list.append("- ['%s', '%s']" % (file, file[:-3].replace("-", " "))) pages_str = "pages:\n" + "\n".join(path_list) + "\n" # Replace the pages data, strategically located at the end of the file mkdocs_yml = os.path.join(MKDOCS_DIR, "mkdocs.yml") temp_file_handler, temp_abs_path = mkstemp() with open(temp_abs_path, 'w') as temp_file: with open(mkdocs_yml) as original_file: for line in original_file: if not "pages:" in line: temp_file.write(line) else: print("Replacing 'pages' property found in mkdocs.yml ...") break else: print("Did not find the 'pages' property in mkdocs.yml.\n" + "Attaching the property at the end of the file.") temp_file.write(pages_str) print(pages_str) # Remove original file and move the new temp to replace it os.close(temp_file_handler) os.remove(mkdocs_yml) move(temp_abs_path, mkdocs_yml)
The main three code blocks, separated by a blank line, do the following:
- Scans the wiki repository directory for any file with the
.mdextension (default for markdown), and adds it to a string for the
- Creates a temporary file to which it copies all the lines from the original
mkdocs.ymlfile until it reads the
pages:line. Once it reaches this line it stops copying and adds the string built on the previous code block.
- Removes the original
mkdocs.ymlfile and replaces it by the newly created file with the updated
Similarly to the git procedure, MkDocs will be built using a subprocess. After that, the site folder is moved into a different location, this step is relevant to the current exampli and can be removed, or edited to your own preferences:
import os import shutil def build_mkdocs(): """ Invokes MkDocs to build the static documentation and moves the folder into the project root folder. """ # Setting the working directory os.chdir(MKDOCS_DIR) # Building the MkDocs project subprocess.call(["mkdocs", "build"]) # Remove root Documentation folder and copy the new site files into it generated_site_dir = os.path.join(MKDOCS_DIR, "site") root_documentation_dir = os.path.join(os.path.dirname(THIS_FILE_DIR), "documentation") print("Copy folder %s into %s ...\n" % (generated_site_dir, root_documentation_dir)) if os.path.exists(root_documentation_dir): shutil.rmtree(root_documentation_dir) shutil.move(generated_site_dir, root_documentation_dir)
Add Index redirect
GitHub Wiki will create by default a
Home article for the homepage with the filename
Home.md (article titles are automatically used for their markdown filename). Having an
index.md file for MkDocs to automatically set as the
index.html page would require having a wiki article titled "Index", which is not a helpul article title for users. So, in order to continue using the GitHub Wiki as intended the build procedure could create an
index.html file to redirect to a predefined page.
For this example the HTML file will redirect to the
Home page created from the default 'Home.md' markdown file:
The HTML code has been embedded into the Python function, so that it can be reconstructed from scratch without depending on an additional file. This function will create the
index.html file at the root of the
site folder, and automatically redirect to whatever page has been set in
And we are done! We have a simple way to build offline static documentation from a GitHub Wiki, all easily achieved using Python, Git, and MkDocs!
Most of the heavy lifting has been done by the fantastic MkDocs, and the build script created just speeds up the process by pulling the latest changes from the Wiki git repository, updating the MkDocs configuration file, and redirecting your homepage to maintain your GitHub documentation workflow the same.
For a more practical code example you can have a look at the Ardublockly documentation build script, which also does some error and exception handling for a more robust build process.