Automatic Publishing

I’d like to self-host a Quartz-built server and continue to use Git as a sync backend for Obsidian itself, with the ultimate goal that all I do is click the “commit and sync” button and then, 30 seconds later, the HTML rendered documentation appears on the server.

This is… not straightforward.

Git for sync backend

The first goal is to get the Git integration set up.

1. In Obsidian settings, go to Community Plugins and enable the 3rd party plugin system
2. Click “Browse”
3. Find the Git integration
4. Install and enable it

If you haven’t already done so, use a command line to init the git repo for your vault, make the first commit, and set the remote URL. The git plugin doesn’t appear to work until that’s done.

$ cd wherever/your/vault/is
$ git init
$ git add .
$ git commit -m "Initial commit"
$ git remote add origin https://your-server/you/repo.git

You should now be able to use the Git sidebar in Obsidian. The text box in the top is the commit message; enter whatever you like there, then click the up arrow in a circle to commit & push. You should see your files pop up in the Git remote server.

Setting up a Quartz-based server

I’ll be using Quartz as a build system.

See also

Quartz homepage: https://quartz.jzhao.xyz/

The Real One True Way to build using Quartz is to clone the Quartz repo, re-home its origin to your own git remote, then write your docs within that repository. I don’t want to do that, so I won’t be, but that complicates the build process (basically, the build agent has to do that process for me instead).

Non-standard install breaks features

Because I am choosing to not use Quartz the Right Way, at least a few features will break. If you choose to follow this path, expect the same. Known breakage:

• Comments are completely out of the question
• All pages show the same date — that of the most recent build

Setting up a Quartz build workflow and site server

Once edits are made, and I click “Commit and push”, the following Woodpecker job is triggered. I keep this in .woodpecker/build-and-publish.yaml in the vault’s Git repo.

labels:
  hostname: rain
  
when:
  - event: push
    branch: master
  - event: manual
    
clone:
  - name: git
    image: woodpeckerci/plugin-git
    settings:
      path: tofk
    dns: 192.168.30.142
      
steps:
  # if desired, do a secret check here
  - name: setup
    image: node
    commands:
      - git clone https://github.com/jackyzha0/quartz.git
      - cp -r tofk/* quartz/content/
      - cd quartz
      - mv ../tofk/.quartz/* ./
      - npm install
    dns: 192.168.30.142
  - name: build
    image: node
    commands:
      - cd quartz
      - npx quartz build
  - name: deploy
    image: node
    volumes:
      - /opt/docker/tome-of-finite-knowledge/static:/deploy
    commands:
      - cp -r quartz/public/* /deploy/

A few notes about this pipeline job:

• It’s restricted to running on only one particular host, the one which hosts the nginx site (the labels section)
• I’ve overridden the clone step here to organize things a bit more. By default, Woodpecker clones the current repo right into the CWD — but since I’m also cloning quartz, I don’t want to deal with git-within-a-git, so putting them both into their own subfolders of the CWD is much nicer.
• The step for mv ../tofk/.quartz/* is extracting Quartz config and layout files from my git repo into the Quartz project.
• A few steps (those which do git clone operations) needed custom DNS settings. This is because of my network setup and is probably not needed in general cases. I had to point it at a DNS server which knows about my reverse proxies, which is not the default for the network this machine is on.

See also

Woodpecker Pipelines, Labeling Agents, Changing Git Clone Options, and Secret scanning.

This pipeline effectively automates migrating an Obsidian vault into Quartz, then building the static site, and then copies the static site files into a host volume. Once there, they’re served by an Nginx docker container:

services:
  tome-of-finite-knowledge:
    container_name: tofk
    image: nginx
    restart: unless-stopped
    volumes:
      - /opt/docker/tome-of-finite-knowledge/static:/usr/share/nginx/html
      - ./nginx.conf:/etc/nginx/nginx.conf:ro

Note the reuse of the same /opt/docker/... host volume — this is where the Woodpecker pipeline previously deployed the site into. The nginx.conf is as follows:

events {
    worker_connections 1024;
}
http {
    sendfile on;
    
    server {
        listen 80;
        server_name localhost;
        root /usr/share/nginx/html;
        index index.html;
        error_page 404 /404.html;
        
        location / {
            include mime.types;
            try_files $uri $uri.html $uri/ =404;
        }
    }
}

Once Nginx is serving the site, a reverse-proxy is used to access the container without forwarding any ports.

Build locally, deploy remotely

Not too long after setting this system up, I learned my t3.micro EC2 instance is just simply not up to the quartz build step — seems it runs out of RAM, the Linux OOM killer does its thing, and the whole system gets wonked. Apparently the npm build step is hungry and cares not for resource-constrained systems.

My solution to this is to build the documentation on a much more powerful computer locally in my home network, then upload it to the EC2 instance over SCP, then have one final step performed on the EC2 host to put the files in their final home.

I’ll spare all the nitty-gritty of why this works since most of it’s explained above. This uses two workflow files, one to run on each machine, and the woodpecker SCP plugin to move files from the local build agent up to the EC2 host.

See also

• SCP plugin: https://woodpecker-ci.org/plugins/scp
• Controlling workflow ordering (dependencies): https://woodpecker-ci.org/docs/usage/workflows#flow-control and Multiple workflows.

In 10-quartz-build.yaml:

labels:
  # microsrv is the more powerful machine on my home network
  hostname: microsrv
 
when:
  - event: push
    branch: master
  - event: manual
 
clone:
  - name: git
    image: woodpeckerci/plugin-git
    settings:
      path: tofk
 
steps:
  - name: setup
    image: node
    commands:
      - git clone https://github.com/jackyzha0/quartz.git
      - cp -r tofk/* quartz/content/
      - cd quartz
      - mv ../tofk/.quartz/* ./
      - npm install
  - name: build
    image: node
    commands:
      - cd quartz
      - npx quartz build
  - name: upload_to_rain
    image: appleboy/drone-scp
    settings:
      # my ec2 instance is only reachable for SSH over this VPN address
      host: 192.168.37.1
      port: 22
      username: misha
      target: /home/misha/tofk-ci-landing-site/
      source: quartz/public/*
      key:
        from_secret: rain_ssh_key
      passphrase: lol-not-putting-this-online

Why not just SCP the files straight into their final destination of /opt/docker/tofk/static? Permissions. The user I’m SCP’ing as (myself) does not have direct write access to that folder, and I don’t want to set up a remote-login-able user which does. So, I’ll have the Woodpecker agent run a small workflow which just cp’s the files from that landing spot to their final resting point.

In 20-deploy.yaml:

labels:
  # this is the EC2 instance
  hostname: rain
 
when:
  - event: push
    branch: master
  - event: manual
 
# make sure this runs *after* the build happens
depends_on:
  - 10-quartz-build
 
# we're just moving files around -- no need to clone the whole repo!
skip_clone: true
 
steps:
  - name: deploy_to_site
    image: alpine
    volumes:
      - /opt/docker/tofk/static:/dest
      - /home/misha/tofk-ci-landing-site/quartz/public:/src
    commands:
      - cp -r /src/* /dest/