At some point, an idea came to me that the Japanese Wikipedia would be a really good source of data. It could be downloaded for offline processing and obviously includes every possible topic. I whipped up a couple of Perl scripts to process the 20 gigabytes or so of data. About six hours later (maybe I should have done this in C++?), I had a nice sorted table of characters ready for processing. I wrote a couple more scripts to generate reports that applied the frequency data to the RTK1, RTK3, 常用漢字 (old and new), and finally the 人名用漢字 (just for fun) character lists.

After I completed my analysis of Wikipedia, I ran a slightly modified version of my script on several gigabytes of assorted novels in raw text format. I’m also including this data as it is another great indicator of actual Japanese language usage. Depending on what kind of text you are planning on reading, this report may even be more suitable than the original Wikipedia analysis.

These reports span over 500 million characters (over 20k of which are unique). This is obviously a tremendous amount of data. Let me know if you want to see something else included in the reports about this data that isn’t yet!

Frequency Reports

• Wikipedia
  • Reports for characters appearing at least 1, 10, 100, 1000, or 10000 times.
  • All of the Wikipedia reports in compressed format (from the above links).
  • Generator Perl scripts and raw data that were used to process this data.
• Novels
  • Alternate sample report on text from hundreds of novels.

Install Base Game and Expansion

1. Install Wine:

   sudo apt-get install wine # for Ubuntu
   

   sudo dnf install wine # for Fedora
   

2. Make a virtual CD-ROM directory in your home directory:

   mkdir ~/cdrom
   

3. Mount the Diablo II install disk image:

   sudo mount -o loop /path/to/install.iso ~/cdrom
   

4. Run install.exe from the CD-ROM, and select “Full Install” when prompted.
   
5. When prompted to do so, unmount the currently inserted disk:

   sudo umount -f ~/cdrom
   

6. Proceed to mount and unmount disk images as needed until installation is complete.
7. The installer will prompt you to run the video test; skip this step for now.
   
8. Mount the expansion disk image, and install Diablo II: Lord of Destruction:
   
9. Copy over d2xvideo.mpq and d2xmusic.mpq from the expansion CD-ROM to your game install directory.
10. Update to the latest version of the game (1.14d at the time of this writing).

Install GLIDE3-to-OpenGL-Wrapper

1. Download and install GLIDE3-to-OpenGL-Wrapper.
2. Extract the glide wrapper to your game install directory and run glide-init.exe.
3. Change the language to English if necessary by clicking the second-to-last button on the left.
   
4. Under the “Extensions” section, make sure the “WGL_ARB_render_texture” checkbox is ticked.
5. Under the “Settings” section, tick the desktopresolution checkbox.
6. Under the “Renderer” section, tick the “no gamma” and “supersampling” checkboxes.
7. Run D2VidTst.exe from the game directory and select “3dfx Glide”:
   

Install PlugY

1. Download and install PlugY.
   
2. Edit the PlugY.ini file in the Mod PlugY directory; in the LAUNCHING section, set Param=--3dfx.
3. Run PlugY.exe from the Mod PlugY directory to start the game (don’t use Game.exe directly).
4. Don’t forget to change the resolution from 640x480 to 800x600 in the settings menu!
   

1. Open the uBlock settings page.
2. Select the “My filters” tab.
3. Add the following filters:

   twitter.com##.tweet[data-component-context="suggest_activity_tweet"]
   twitter.com##.tweet[data-component-context="suggest_recycled_tweet_inline"]
   twitter.com##.tweet[data-component-context="suggest_pyle_tweet"]
   

4. Enjoy a timeline with less noise!

Let’s start by creating a new repository for adder, a simple tool that adds two numbers:

git init adder

Now let’s add main.py, the file containing the full source of our application:

#!/usr/bin/env python

import os
import sys

def add(a, b):
    return a + b

def main(args):
    try:
        sum = add(int(args[0]), int(args[1]))
        print(f'the sum is {sum}')
    except:
        print('invalid arguments')
        sys.exit(2)

if __name__ == '__main__':
    main(sys.argv[1:])

Next, we stage and commit the change:

git add main.py
git commit -m "Add application to add numbers"

Now let’s pretend that there is some sort of regression and we must add logging to investigate the issue. First, we need to add our custom logging library located in a new file, logging.py:

def info(msg):
    print(msg)

We must also update the existing main.py file to call into this library:

#!/usr/bin/env python

import os
import sys
import logging

def add(a, b):
    logging.info(f'adding numbers {a} and {b}')
    return a + b

def main(args):
    try:
        sum = add(int(args[0]), int(args[1]))
        print(f'the sum is {sum}')
    except:
        print('invalid arguments')
        sys.exit(2)

if __name__ == '__main__':
    main(sys.argv[1:])

With our local changes complete, let’s review the differences against the master branch:

git diff

The output of the above command is a valid patch file in and of itself:

index 7efa5d6..d3aadba 100755
--- a/main.py
+++ b/main.py
@@ -3,2 +3,3 @@
 import os
 import sys
+import logging
@@ -6,2 +7,3 @@
 def add(a, b):
+    logging.info(f'adding numbers {a} and {b}')
     return a + b

While the changes to main.py are shown, the contents of the newly added logging.py file are not. To fix this, simply add it with:

git add --intent-to-add logging.py

With this, both files are now correctly shown as changes:

diff --git a/logging.py b/logging.py
new file mode 100644
index 0000000..39b4bbc
--- /dev/null
+++ b/logging.py
@@ -0,0 +1,2 @@
+def info(msg):
+    print(msg)
diff --git a/main.py b/main.py
index 7efa5d6..d3aadba 100755
--- a/main.py
+++ b/main.py
@@ -3,2 +3,3 @@
 import os
 import sys
+import logging
@@ -6,2 +7,3 @@
 def add(a, b):
+    logging.info(f'adding numbers {a} and {b}')
     return a + b

With the changes looking good, it’s time to generate the patch file:

git diff --output ~/logging.patch

Now that we have performed the required debugging and saved the day, we can revert all of our local debugging changes:

git reset --hard

Let’s now pretend that some time has passed and we must debug the add function once again. Furthermore, as a result of feature creep, the adder tool now also has to handle subtraction. The updated main.py file is shown below:

import os
import sys

def add(a, b):
    return a + b

def main(args):
    try:
        op = args[0]
        a = int(args[1])
        b = int(args[2])
        ops = {'+': add, '-': subtract}
        result = ops[op](a, b)
        print(f'the result is {result}')

    except:
        print('invalid arguments')
        sys.exit(2)

def subtract(a, b):
    return a - b

if __name__ == '__main__':
    main(sys.argv[1:])

As there are no conflicts in the modified chunks, we can now apply the patch:

git apply ~/logging.patch

We can see that the logging.py file is created and main.py has been updated with the logging from before:

import os
import sys
import logging

def add(a, b):
    logging.info(f'adding numbers {a} and {b}')
    return a + b

def main(args):
    try:
        op = args[0]
        a = int(args[1])
        b = int(args[2])
        ops = {'+': add, '-': subtract}
        result = ops[op](a, b)
        print(f'the result is {result}')

    except:
        print('invalid arguments')
        sys.exit(2)

def subtract(a, b):
    return a - b

if __name__ == '__main__':
    main(sys.argv[1:])

If you do run into conflicts, you can use the reject flag to apply as many changes as possible without failing the entire operation.

Let’s assume that this is our commit log:

commit c9e1e685ff81b1a6ec75848423219caa752838c4 (HEAD -> topic)
Author: Alex Yatskov <alex@foosoft.net>
Date:   Sun Jul 17 10:37:05 2022 -0700

    Progress

commit 4ad391d17d92fe499247ac848556a3fd72842621
Author: Alex Yatskov <alex@foosoft.net>
Date:   Sun Jul 17 10:36:49 2022 -0700

    Fixing

commit bef920b9789d988507b1972de697273ea2df3b24
Author: Alex Yatskov <alex@foosoft.net>
Date:   Sun Jul 17 10:36:39 2022 -0700

    WIP

commit 9e2d24395008af39c6192df128bcbce0deaa2732 (master)
Author: Alex Yatskov <alex@foosoft.net>
Date:   Sun Jul 17 10:36:09 2022 -0700

    Mainline feature

Typically, to squash the topic commits, you might execute the following:

git rebase -i 9e2d24395008af39c6192df128bcbce0deaa2732

This command will open a text editor, prompting you to specify which commits to pick and which to squash:

pick bef920b WIP
pick 4ad391d Fixing
pick c9e1e68 Progress

# Rebase 9e2d243..c9e1e68 onto 9e2d243 (3 commands)
#
# Commands:
# p, pick <commit> = use commit
# r, reword <commit> = use commit, but edit the commit message
# e, edit <commit> = use commit, but stop for amending
# s, squash <commit> = use commit, but meld into previous commit
# f, fixup [-C | -c] <commit> = like "squash" but keep only the previous
#                    commit's log message, unless -C is used, in which case
#                    keep only this commit's message; -c is same as -C but
#                    opens the editor
# x, exec <command> = run command (the rest of the line) using shell
# b, break = stop here (continue rebase later with 'git rebase --continue')
# d, drop <commit> = remove commit
# l, label <label> = label current HEAD with a name
# t, reset <label> = reset HEAD to a label
# m, merge [-C <commit> | -c <commit>] <label> [# <oneline>]
# .       create a merge commit using the original merge commit's
# .       message (or the oneline, if no original merge commit was
# .       specified); use -c <commit> to reword the commit message
#
# These lines can be re-ordered; they are executed from top to bottom.
#
# If you remove a line here THAT COMMIT WILL BE LOST.
#
# However, if you remove everything, the rebase will be aborted.
#

You would then pick the first commit and squash the rest:

pick bef920b WIP
squash 4ad391d Fixing
squash c9e1e68 Progress
...

There is nothing wrong with this approach, but it gets cumbersome if you have dozens of commits. A quicker way would be to reset the branch to the start of the topic changes, and re-commit as follows:

git reset --soft 9e2d24395008af39c6192df128bcbce0deaa2732
git commit -m "Everything required for my new feature"

This operation has the advantage of being non-interactive and produces the same result (at the cost of flexibility in specifying what to squash). If you are like me and prefer to squash all topic commits anyway, this may be an improvement to your Git workflow.

File Layout

To get started with Lua, replace your init.vim file with init.lua. Note that if you have both files present at once, Neovim will complain and your configuration will not load. The scripts contained in the ftplugin directory can also be converted to Lua in the same way. Finally, it is also possible to split the configuration into multiple files by creating a lua subdirectory and using require to import scripts contained therein.

require('common')
require('lsp')
require('plugins')
require('util')
require('autocmd')

With these changes, your directory structure may look similar to this:

├── ftplugin
│   ├── c.lua
│   ├── cpp.lua
│   ├── go.lua
│   ├── html.lua
│   ├── markdown.lua
│   └── yaml.lua
├── init.lua
├── lua
│   ├── autocmd.lua
│   ├── common.lua
│   ├── ginit.lua
│   ├── lsp.lua
│   ├── plugins.lua
│   └── util.lua
└── pack
    └── plugins

If you use Neovim Qt, you may be wondering about the presence of ginit.lua from the above file listing. Unfortunately, Neovim Qt does not support a Lua-only configuration path and insists on using ginit.vim for GUI-specific configuration. Furthermore, attempting to include Neovim Qt-specific commands in init.lua will cause errors on load (tests for GUI presence via has do not appear to work). Fortunately, we can use an autocommand to conditionally require the ginit.lua file only when running under a GUI.

vim.api.nvim_create_autocmd(
    'UIEnter', {
        callback = function()
            if vim.v.event.chan == 1 then
                require('ginit')
            end
        end,
        once = true,
    }
)

Color Scheme and Leader

The color scheme can be set by executing the same colorscheme command as in Vim. There are two different ways to do this, with the latter looking a bit cleaner than the former. This pattern of treating the command name as a callable method in cmd extends to other commands (including ones exposed by plugins) as well.

vim.cmd('colorscheme solarized8')

-- or with some syntax sugar...

vim.cmd.colorscheme('solarized8')

The leader key is configured with a global variable, just like in Vim.

vim.g.mapleader = ' '

Core Options

Consider comparing your configuration against the defaults listed in the documentation. Neovim ships with mostly sane defaults and you can reduce the size of your configuration file by skipping redundant settings. Note that boolean settings in Vim that start with no when false (such as nonumber) are represented by assignments instead.

vim.opt.autowrite = true
vim.opt.completeopt = 'menuone,noselect'
vim.opt.expandtab = true
vim.opt.fileformats = 'unix,dos,mac'
vim.opt.foldenable = false
vim.opt.guicursor = 'n:blinkon0'
vim.opt.ignorecase = true
vim.opt.linebreak = true
vim.opt.modeline = false
vim.opt.modelines = 0
vim.opt.number = true
vim.opt.shiftround = true
vim.opt.shiftwidth = 4
vim.opt.showmode = false
vim.opt.smartcase = true
vim.opt.swapfile = false
vim.opt.tabstop = 4
vim.opt.termguicolors = true
vim.opt.updatetime = 300
vim.opt.wrap = false
vim.opt.writebackup = false

Plugin Configuration

Plugins written for Vim are generally configured with global variables; these are exposed in the vim.g table. Vimscript supports namespaces via the # character, which can cause syntax issues for Lua. To work around this problem, use the square bracket index operator on the table instead.

-- vim-airline
vim.g['airline#extensions#tabline#enabled'] = 1
vim.g['airline_symbols_ascii'] = 1

-- vim-dirvish
vim.g.dirvish_mode = ':sort ,^.*[\\/],'

-- vim-go
vim.g.go_diagnostics_enabled = 0
vim.g.go_imports_autosave = 0
vim.g.go_metalinter_enabled = {}
vim.g.go_null_module_warning = 0
vim.g.go_version_warning = 0

Key Mappings

Key mappings are also reasonably straightforward. The main difference from vanilla Vim is that commands must start with a <cmd> tag instead of the : prefix. When defining key mappings for a combination of modes (such as normal and visual), the mode parameter has to be specified as a Lua table; a comma-delimited string will not work.

-- common keymaps
vim.keymap.set('i', '<c-c>', '<esc>')
vim.keymap.set('n', '<bs>', '<cmd>bd<cr>')
vim.keymap.set('n', '<c-c><c-c>', '<cmd>nohlsearch<cr>')
vim.keymap.set('n', '<leader><leader>', '<cmd>b#<cr>')
vim.keymap.set('n', '<leader>w', '<cmd>w<cr>')
vim.keymap.set('n', '<leader>x', '<cmd>x<cr>')
vim.keymap.set('n', 'j', 'gj')
vim.keymap.set('n', 'k', 'gk')

-- clipboard keymaps
vim.keymap.set({'n', 'v'}, '<leader>P', '"+P')
vim.keymap.set({'n', 'v'}, '<leader>Y', '"+y$')
vim.keymap.set({'n', 'v'}, '<leader>d', '"+d')
vim.keymap.set({'n', 'v'}, '<leader>d', '"+dd')
vim.keymap.set({'n', 'v'}, '<leader>p', '"+p')
vim.keymap.set({'n', 'v'}, '<leader>y', '"+y')
vim.keymap.set({'n', 'v'}, '<leader>yy', '"+yy')

-- split keymaps
vim.keymap.set('n', '<a-=>', '<c-w><c-=>')
vim.keymap.set('n', '<a-h>', '<c-w><')
vim.keymap.set('n', '<a-j>', '<c-w>+')
vim.keymap.set('n', '<a-k>', '<c-w>-')
vim.keymap.set('n', '<a-l>', '<c-w>>')
vim.keymap.set('n', '<c-h>', '<c-w>h')
vim.keymap.set('n', '<c-j>', '<c-w>j')
vim.keymap.set('n', '<c-k>', '<c-w>k')
vim.keymap.set('n', '<c-l>', '<c-w>l')

-- fzf.vim keymaps
vim.keymap.set('n', '<leader>fg', vim.cmd.GFiles)
vim.keymap.set('n', '<leader>fh', vim.cmd.History)
vim.keymap.set('n', '<leader>fb', vim.cmd.Buffers)
vim.keymap.set('n', '<leader>fl', vim.cmd.Lines)

Auto Commands

Autocommands are defined as a list of conditions followed by a table specifying what you want the action to do. Unlike key mappings, it is possible to declare the condition list as either a table or a comma-delimited string.

vim.api.nvim_create_autocmd(
    'BufRead,BufNewFile', {
        pattern = '*.gohtml',
        command = 'set filetype=html'
    }
)

-- or using the table syntax...

vim.api.nvim_create_autocmd(
    {'BufRead', 'BufNewFile'}, {
        pattern = '*.gohtml',
        command = 'set filetype=html'
    }
)

Useful Resources

• http://www.lua.org/manual/5.4/
• https://github.com/nanotee/nvim-lua-guide
• https://neovim.io/doc/user/api.html
• https://neovim.io/doc/user/lua.html

Done and done… Except the codec was not downloading no matter how long I waited. Restarting Firefox and deactivating/reactivating the video codec were also not helping. Turns out that the fix is to install the mozilla-openh264 package yourself:

sudo dnf install mozilla-openh264

Not a big deal, but I could have done without the deceptive messaging, Firefox.

Fellow Japanese learning friends, after careful consideration, I’ve made the decision to cease development of Yomichan and related tools: Yomichan Import, Zero-EPWING, and JMDict. The Anki-Connect project will continue to be supported for the foreseeable future.

The original Yomichan for Anki project had its beginnings back in 2011, over a decade ago. I began the project shortly after embarking on my AJATT Japanese language journey. It proved to be an invaluable tool for mining vocabulary from books I was struggling to read at the time. Over the years, and with feedback from the folks at the (now defunct) Kanji Koohii Forums, it evolved into the browser extension for Chrome and Firefox that you know today. I like to think that Yomichan accepted the torch passed down to it by Rikaisama, another popular extension for power users learning Japanese during this period.

I have since then attained the N1 JLPT certification (still one of my proudest achievements), lived in Japan, and graduated with a master’s degree from Keio University. I speak, read and write Japanese every day. I stopped doing flash cards in Anki years ago. I now prefer to learn the language in a more organic, contextual way. I can’t remember the last time I used Yomichan in my browser other than to test code changes.

I know that I do my best work on open source projects that answer my immediate needs. If I don’t make this distinction, then they just become a second (or third, or fourth…) job. Yomichan no longer meets this criterion for me, and thus it is time to let go.

Dear Yomichan Users

Thank you for your support and feedback over all of these years. Yomichan will remain available in the browser extension stores indefinitely. You are free to continue using it as long as you find it useful or until you find something better. It will never be sold off to run ads and steal your data. Web technology moves quickly, and I suppose that at some point Yomichan will probably stop functioning. Please enjoy the extension and related tooling until then!

Dear Yomichan Developers

Deepest thanks to toasted-nutbread, siikamiika, and all of the other talented developers who have contributed countless hours to improving the Yomichan ecosystem. You are absolute legends, and these projects would not be the same without you. If there is interest in continuing development, please fork the project and shape it to your needs. I will happily link to your work to help get the word out. I have just two requests:

• Create a new title for your fork to avoid confusion.
• Respect the license and make proper attribution.

I’m looking forward to seeing what will come next!

This website has been rebuilt more times than I care to count. The stack evolved from using WordPress, to ConcreteCMS, to Pelican, to Hexo, to Hugo before briefly pausing on Metalsmith. Static site generation seemed to be the way to go (and yes, of course my WordPress site was hacked) for simplicity and portability. One problem I ran into was that most of the static site generators placed seemingly arbitrary restrictions on site structure. To enumerate a few of the issues:

• Text content has to live in a different directory from linked assets such as images and downloads.
• Blog posts can be tagged with keywords but ordinary content pages cannot be.
• Differences in implementation for blog posts and content pages make templating difficult.
• Standalone HTML content (such as reveal.js presentations) was broken by the generator.
• Bending the generator to work with your desired directory layout led to complexity and pain.
• Content generation was slow or non-reproducible between rebuilds.

Out of all the static site generators I evaluated, Metalsmith came the closest to what I needed. I could finally begin to see a path forward to a homepage structured the way I wanted. Unfortunately, Metalsmith was quite temperamental due to shortcuts added in the pipeline design in what I can only assume to be an attempt to make the generator easier to use. It became clear to me that I needed to build my own tooling to get the results I could be truly satisfied with. Goldsmith, my Go-based static generator, was born in late 2015.

Goals

Here is a sampling of some hard requirements I relied on to guide me through implementation:

• Flexibility is paramount; must be able to achieve any design through customization.
• All capabilities must be implemented through pipeline plugins to improve flexibility.
• Page generation must be possible for in-place Git project repositories (GitHub mirror).
• RSS feeds should be automatically generated for any page that needs publishing.
• Post-process resources; generate thumbnails, minify HTML, CSS, and JavaScript.
• Speed up iteration by immediately reflecting source changes on the generated site.
• Arbitrary metadata must be supported for tagging, categorization, formatting, and more.

Implementation

Let’s just dive into the code. Many details have been omitted for the sake of brevity; let’s just focus on flow for now:

func build(contentDir, buildDir, cacheDir string, devMode bool) {

    // ...

    errs := goldsmith.Begin(contentDir).
        Cache(cacheDir).
        Clean(true).
        Chain(rule.New()).
        Chain(new(indexRenamer)).
        Chain(frontmatter.New()).
        Chain(markdown.NewWithGoldmark(gm)).
        FilterPush(condition.New(!devMode)).
        Chain(absolute.New().BaseUrl(baseUrl)).
        Chain(minify.New()).
        FilterPop().
        Chain(syndicate.New(baseUrl, "Feed").WithFeed("posts", feedConfig)).
        Chain(collection.New().Comparer(collectionComparer)).
        FilterPush(operator.Not(wildcard.New("**/*.gohtml"))).
        Chain(index.New(indexMeta)).
        FilterPop().
        Chain(tags.New().IndexMeta(tagMeta)).
        Chain(forward.New(forwardMeta).PathMap(forwardPathMap)).
        Chain(layout.New()).
        Chain(thumbnail.New().Style(thumbnail.Crop)).
        FilterPush(condition.New(devMode)).
        Chain(livejs.New()).
        FilterPop().
        FilterPush(condition.New(!devMode)).
        Chain(minify.New()).
        FilterPop().
        End(buildDir)

    for _, err := range errs {
        log.Print(err)
    }
}

Walkthrough

1. Recursively scan all the files in the contentDir and start pushing them through the pipeline:

   goldsmith.Begin(contentDir).
   

2. Enable caching in cacheDir for supported plugins; this helps avoid reprocessing of unchanged resources:

   Cache(cacheDir).
   

3. Enable cleanup of extra files found in the buildDir that are not build output:

   Clean(true).
   

4. Chain the rule plugin to enable per-directory rules, which are specified in a rules.toml file:

   Chain(rule.New()).
   

   Rules can skip files matching a set of patterns. To prevent .git directories from being published, we can use the drop directive:

   [[drop]]
   accept = [
       "**/.git/**",
   ]
   

   Rules can also help specify metadata on files that we do not want to modify directly (such as the README.md file for a project). We can instead match the file and apply properties to it externally:

   [[apply]]
   accept = ["goldsmith/README.md"]
   props.Collection = "ProjectsActive"
   props.Description = "Static pipeline-based website generator written in Go."
   props.GitHub = "goldsmith"
   props.Tags = ["generator", "golang", "goldsmith", "mit license", "web"]
   

5. Repositories on GitHub are expected to contain the project description inside of a README.md file. We want to rename all instances of README.md to index.md and ensure that each project directory will eventually contain an index.html file. I added a simple indexRenamer helper plugin to automate this:

   Chain(new(indexRenamer)).
   

   The implementation of this one-off plugin is shown below:

   type indexRenamer struct{}
   
   func (*indexRenamer) Name() string {
       return "indexRenamer"
   }
   
   func (*indexRenamer) Initialize(context *goldsmith.Context) error {
       context.Filter(wildcard.New("**/README.md"))
       return nil
   }
   
   func (*indexRenamer) Process(context *goldsmith.Context, inputFile *goldsmith.File) error {
       dir := path.Dir(inputFile.Path())
       inputFile.Rename(path.Join(dir, "index.md"))
       context.DispatchFile(inputFile)
       return nil
   }
   

6. Chain the frontmatter plugin to parse and strip out any metadata prefixed onto our files:

   Chain(frontmatter.New()).
   

   This metadata consists of arbitrary properties and can be defined in several formats. The TOML frontmatter for this page is:

   +++
   Date = 2023-10-10
   Description = "Exploring the uniquely custom process of generating this website with Goldsmith."
   Tags = ["web", "goldsmith", "generator"]
   +++
   

7. Chain the markdown plugin to convert all Markdown files to HTML fragments:

   Chain(markdown.NewWithGoldmark(gm)).
   

   At this stage, these are not yet documents, but merely excerpts that must have header and footer markup templates applied to them before they become functional pages. This plugin uses the goldmark Markdown parser, which supports GitHub extensions, syntax highlighting and more. The desired features can be optionally configured by passing in a goldmark instance:

   gm := goldmark.New(
       goldmark.WithExtensions(extension.GFM, extension.Typographer),
       goldmark.WithParserOptions(parser.WithAutoHeadingID()),
       goldmark.WithRendererOptions(html.WithUnsafe()),
       goldmark.WithExtensions(highlighting.NewHighlighting(highlighting.WithStyle("solarized-dark"))),
   )
   

8. Push a condition filter that ensures that pipeline manipulation within its scope only happens when the page is not being generated in devMode (preparing content for deployment to the web server):

   FilterPush(condition.New(!self.dev)).
   Chain(absolute.New().BaseUrl(baseUrl)).
   Chain(minify.New()).
   FilterPop().
   

   When generating the site for deployment, we chain the absolute plugin to convert all relative URLs to absolute ones using the baseUrl (https://foosoft.net) prefix. This is required to ensure that all relative references are resolvable when the site’s content is displayed in an RSS reader. We also apply the minify plugin to reduce the site’s content to the minimal possible size without impacting behavior. We do not want to do this in development mode since it makes debugging the code a lot trickier! Once these two steps are completed, we pop off the condition filter and resume unconditional processing.

9. Chain the syndicate plugin to generate feeds for selected articles (like this one):

   Chain(syndicate.New(baseUrl, "Feed").WithFeed("posts", feedConfig)).
   

   Only pages with the Feed metadata property with the value posts are considered during feed generation. We must also provide a basic site configuration and declare the desired feed formats.

   feedConfig := syndicate.FeedConfig{
       Title:       "FooSoft Productions",
       Url:         baseUrl,
       AuthorName:  "Alex Yatskov",
       AuthorEmail: "alex@foosoft.net",
       RssPath:     "feeds/posts.xml",
       AtomPath:    "feeds/posts.atom",
       JsonPath:    "feeds/posts.json",
       ItemConfig: syndicate.ItemConfig{
           TitleKey:        "Title",
           CreatedKey:      "Date",
           DescriptionKey:  "Description",
           ContentFromFile: true,
       },
   }
   

10. Chain the collection plugin to group related pages into lists based on the value of the Collection metadata property:

    Chain(collection.New().Comparer(collectionComparer)).
    

    The resulting collections are ordered using a custom comparer function which groups items first by Date and then Title properties:

    collectionComparer := func(i, j *goldsmith.File) bool {
        if iData, iOk := i.Prop("Date"); iOk {
            if jData, jOk := j.Prop("Date"); jOk {
                return iData.(time.Time).UnixNano() > jData.(time.Time).UnixNano()
            }
        }
    
        if iData, iOk := i.Prop("Title"); iOk {
            if jData, jOk := j.Prop("Title"); jOk {
                return strings.Compare(iData.(string), jData.(string)) < 0
            }
        }
    
        return strings.Compare(i.Path(), j.Path()) < 0
    }
    

11. Conditionally chain the index plugin to generate directory index pages for paths that do not match the **/*.gohtml pattern:

    FilterPush(operator.Not(wildcard.New("**/*.gohtml"))).
    Chain(index.New(indexMeta)).
    FilterPop().
    

    These template files are consumed later in the pipeline and we don’t need to include them in the build output. Newly generated index pages are assigned the metadata configured in the plugin:

    indexMeta := map[string]any{
        "Layout": "index",
    }
    

12. Chain the tags plugin to generate tag indices from page metadata:

    Chain(tags.New().IndexMeta(tagMeta)).
    

    By default, this plugin builds lists based on the contents of the Tags property. Some additional configuration may be provided to specify what metadata is assigned to newly generated tag index pages.

    tagMeta := map[string]any{
        "Area":   "tags",
        "Layout": "tag",
    }
    

13. Chain the forward plugin to generate stub pages for content on the homepage that has been moved to a different URL:

    Chain(forward.New(forwardMeta).PathMap(forwardPathMap)).
    

    We must specify the URL mapping as well as any additional metadata that should be assigned to newly generated forwarder pages:

    forwardMeta := map[string]any{
        "Layout": "forward",
        "Title":  "File Moved",
    }
    
    forwardPathMap := map[string]string{
        "mangle/index.html":                   "/projects/mangle/",
        "projects/argwrap/index.html":         "/projects/vim-argwrap/",
        "projects/kanji-frequency/index.html": "/posts/kanji-frequency/",
        "projects/yomichan-chrome/index.html": "/projects/yomichan/",
        "yomichan/index.html":                 "/projects/yomichan/",
    }
    

14. Chain the layout plugin to apply Go HTML templates to our page HTML fragments:

    Chain(layout.New()).
    

    By default, the plugin searches for the Layout metadata property to decide which template to apply. All metadata associated with a file can be referenced from within these templates. The templates themselves are stored in *.gohtml files which are consumed by the pipeline along with all the other inputs. For example, the top-level template for the current page looks as follows:

    {{define "page"}}
    {{template "header" .}}
    {{.Props.Content}}
    {{template "footer" .}}
    {{end}}
    

15. Chain the thumbnail plugin to automatically generate thumbnails for all processed image files:

    Chain(thumbnail.New().Style(thumbnail.Crop)).
    

    By default, this produces PNG thumbnails resized to 128x128 pixels. The filename is automatically determined by appending -thumb.png to the source image filename after stripping the extension. For example, foo.png becomes foo-thumb.png.

16. Conditionally chain the livejs plugin in development mode to inject JavaScript to refresh the page when a change is detected:

    FilterPush(condition.New(self.dev)).
    Chain(livejs.New()).
    FilterPop().
    

    This script continuously queries the server to detect changes. It should never be enabled on the homepage once it has been deployed.

17. Conditionally chain the minify plugin once more to also minify the HTML templates we applied a couple of steps ago:

    FilterPush(condition.New(!self.dev)).
    Chain(minify.New()).
    FilterPop().
    

    This plugin is chained twice so that both the RSS output (which is spun off earlier in the pipeline) and the full site are minified.

18. Finally, write out all the generated files to the buildDir directory.

    End(buildDir)
    

    Today, over 400 files are generated by Goldsmith and output into the target directory. This process is highly parallelized and takes just over 100ms to complete with caching enabled and 130ms for a full rebuild. The results are fully deterministic, which makes inspecting content changes between website deployments easy.

Error Handling

If a plugin reports an error while processing a file, the file stops propagating through the pipeline. A list of errors is returned upon calling End. These errors contain the name of the file in question as well as the faulting plugin.

Directory Structure

One unique thing about the on-disk organization of FooSoft.net is that projects are actually included as submodules within the website repository. This makes it easy to keep the project’s README.md up to date with the data shown on the generated homepage: it’s literally the same file! I view this as an invaluable feature for keeping the project brand separate from where the project is hosted.

├── build
├── cache
├── content
│   ├── 404.md
│   ├── CNAME
│   ├── css
│   │   └── main.css
│   ├── gohtml
│   │   ├── components.gohtml
│   │   └── content.gohtml
│   ├── go.mod
│   ├── img
│   │   ├── alex.png
│   │   └── brand.svg
│   ├── index.md
│   ├── js
│   │   └── main.js
│   ├── portfolio
│   │   ├── amaze
│   │   ├── arenanet
│   │   ├── index.md
│   │   ├── microsoft
│   │   └── rules.toml
│   ├── posts
│   │   ├── decrapifying-the-twitter-timeline
│   │   ├── fixing-firefox-video-playback-in-fedora-37
│   │   ├── generating-the-foosoft.net-homepage
│   │   ├── index.md
│   │   ├── installing-diablo-ii-on-linux
│   │   ├── kanji-frequency
│   │   ├── rules.toml
│   │   ├── saying-goodbye-to-vimscript
│   │   ├── squashing-all-commits-in-a-git-topic-branch
│   │   ├── sunsetting-the-yomichan-project
│   │   └── working-with-patches-in-git
│   ├── projects
│   │   ├── anki-connect
│   │   ├── ankijoy
│   │   ├── btac
│   │   ├── goldsmith
│   │   ├── goldsmith-components
│   │   ├── goldsmith-samples
│   │   ├── guid.nvim
│   │   ├── hlm2-wad-extract
│   │   ├── homemaker
│   │   ├── index.md
│   │   ├── jmdict
│   │   ├── lazarus
│   │   ├── mangle
│   │   ├── md2vim
│   │   ├── mdview
│   │   ├── meganekko
│   │   ├── metacall
│   │   ├── moonfall
│   │   ├── restaurant-search
│   │   ├── revolver.nvim
│   │   ├── rules.toml
│   │   ├── scrawl
│   │   ├── tetrys
│   │   ├── vfs
│   │   ├── vim-argwrap
│   │   ├── yomichan
│   │   ├── yomichan-anki
│   │   ├── yomichan-import
│   │   ├── zero-epwing
│   │   └── zero-epwing-go
│   ├── rules.toml
│   ├── tags
│   │   └── index.md
│   └── ttf
│       ├── MaterialIcons-Regular.ttf
│       ├── RobotoMono-Regular.ttf
│       ├── Roboto-Regular.ttf
│       └── RobotoSlab-Regular.ttf
├── deploy.sh
├── diff.sh
├── foosoft.github.io
├── go.mod
├── go.sum
├── main.go
├── rebuild.sh
├── serve.sh
└── update.sh

Conclusion

While this may seem like there are a lot of steps required to build this site, the actual cognitive load is very low. The statically generated output for FooSoft.net is a perfect match for my expectations and requires no hacks. Iteration is fast and adding new features is a breeze. If this seems interesting, take a look at the goldsmith-samples repository for some simple examples to get started!

I decided against transferring extension ownership for two reasons. First, there was simply no way of knowing who would make any progress in completing the mandatory port to Manifest V3; what if they just squatted on the Yomichan name? Second, advertisers are happy to spend considerable capital (approaching five figures USD) to purchase extensions with established user bases (Yomichan has 100k+ users at the time of writing) and stuff them full of spyware. I still get weekly offers to sell out my users… I could never do this myself, but I’m sure there are developers who would not hesitate.

Shortly after my announcement, TheMoeWay, a Japanese culture and learning community, reached out and expressed interest in maintaining a Yomichan fork. They have been professional in respecting my requests (establishing a distinct project identity, etc.) as well as maintaining proper attribution (unfortunately this has not been true of all forks). From what I’ve observed lurking in TheMoeWay Discord, there is a vibrant and active development community behind this new project (which even includes some former Yomichan contributors as well). After many months of effort, they have released a stable version of Yomitan. Crossing the Manifest V3 finish line is an amazing achievement by the group, and is why I now consider Yomitan to be the definitive successor to Yomichan.

Visit the Yomitan project page for official Firefox and Chrome releases. TheMoeWay has also forked other supporting tools for dictionary generation and parsing; I view them as the natural evolution of the Yomichan toolchain. Finally, I am delighted that there are several other forks of Yomichan out there as well; having options is a great strength of open source development. That being said, projects should make sure to respect the licensing agreements and make proper attribution when ingesting code from other Yomichan derivatives.

Now that a stable alternative to Yomichan has emerged, Yomichan itself will be delisted from both Firefox and Chrome stores. Users who already have the extension installed should be able to continue using it until it is inevitably disabled by the browser for not supporting Manifest V3. A sincere thanks from me to all the users of Yomichan over the years!

I think most would agree that GitHub was a social network of sorts from its outset. This by itself was not an issue. Social networks back then (and other “big web” sites in general) were more focused on providing user value in order to encourage platform adoption. Over time, as these platforms attained monopolies in their respective domains, user value took a back seat to attaining business goals. This phenomenon, called enshittification by Cory Doctorow, is described as follows:

Here is how platforms die: first, they are good to their users; then they abuse their users to make things better for their business customers; finally, they abuse those business customers to claw back all the value for themselves. Then, they die. I call this enshittification, and it is a seemingly inevitable consequence arising from the combination of the ease of changing how a platform allocates value, combined with the nature of a “two sided market”, where a platform sits between buyers and sellers, hold each hostage to the other, raking off an ever-larger share of the value that passes between them.

Unfortunately, while still providing user value, GitHub has advanced significantly on the enshittification curve. There have been many examples of this over the past several years, such as:

• Forced algorithmic repository recommendations, many of which cannot be disabled.
• Inline advertisements: “Did you know you can ask your organization to pay for Copilot?”
• Arbitrary policy changes to require 2FA before letting me access my own repositories.
• Training AI for current and future paid products on my code with no way to opt out.

In short, I can no longer say that the motivations of this platform align with my goals for continued FOSS development. I remember coming to this same conclusion when I quit Facebook many years ago. What was initially a useful site for keeping in contact with your friends and family devolved into a platform for advertisement and manipulation. In this light, keeping my repositories on GitHub has all the appeal of storing my personal photo albums on Facebook.

I fully expect that as a part of this departure, my projects will see diminished exposure to new users and contributors. This, again, is similar to the decrease in content impressions one experiences upon leaving Facebook. Fortunately, I don’t really care about software development as a popularity contest anymore. The way I look at it now is that the applications I develop are designed with myself as the primary user. These applications also just happen to be open source to make them more reusable for others that choose to do so. I’m no longer interested in intentionally increasing the user base or hunting for GitHub stars. Amusingly, I’ve never noticed any correlation between the projects that were popular and the projects that I actually enjoyed working on.

Adopting this model of focusing on code instead of stars and visibility has been extremely reinvigorating. I still value high-quality code contributions, but only to the extent that they improve use cases which are relevant to me. As part of this new outlook, I’m making several changes to my homepage as well.

Homepage

For many years, the primary purpose of this website was to host information about the projects I’m currently working on. Over time, this became increasingly redundant as literally all forges support rendering of the README.md file to help explain what a given repository is all about. Nevertheless, I have been maintaining an intricate static generation pipeline which would generate pages based on Git submodules hosted in the site’s projects subdirectory. I’m now doing away with all of this, not because it’s difficult to maintain, but more because it’s unnecessary complexity which doesn’t add anything of value to the site.

After removing projects, I’m left with an outdated website design which can be significantly trimmed down to essentially the blog and portfolio sections. I haven’t yet figured out a new visual design, but it’s going to be rather minimalist. I’ve already started making structural changes in that now there are only two sections: the blog and the about page (which also serves as the portfolio). I’ve been adding a lot of HTTP 301 redirects in my nginx configuration to prevent broken links where possible.

Repositories

At the start of last year, I migrated all my projects off GitHub to a self-hosted Gitea instance. After overcoming the initial FOMO of missed user engagement, I’ve instead discovered Zen in the email-based patch workflow and the utter lack of caring about project popularity. The lack of other social features and Copilot upsells has been another benefit which cannot be overstated.

Alas, all good things must come to an end, and within the past couple of weeks, my Gitea instance has been getting completely destroyed by AI content scraping bots, the majority of which completely ignore the robots.txt file telling them to stay out. My choices were playing cat and mouse with the scrapers or moving back to a public forge. As the former was a lot more work than the latter, I got to searching for a new home for my projects.

The first forge I evaluated was Codeberg, an up-and-coming GitHub alternative which is running Forgejo, a fork of Gitea, the software I was previously self-hosting. I got as far as moving several projects over, but then a couple of days later Codeberg decided to go political over a two-word remark by some juvenile spammer, blowing up everything into yet another left versus right conflict.

We will not be discouraged in our fight against far-right ideologies. They are currently on the rise in many parts of the world, and we believe it is important to protect all kinds of marginalized groups. However, if you believe this does not affect your project, you are wrong. Far-right forces pose a threat to all of us.

As I do not want to associate my work with politics, signaling, or activism of any kind, it was time to move on. I eventually landed on SourceHut. While I can’t say the user interface is very intuitive, I do like that this forge supports a patch-based workflow and clearly advertises itself as a paid service. It seems to support everything I need, without turning into another social network. I’ve updated the redirects on my nginx instance so that old links to git.foosoft.net should now redirect to SourceHut (for both viewing project information and cloning over HTTPS). Sorry if I missed something and broke your workflow!

Finally, I’ve delisted a lot of projects that are no longer maintained, and for all practical purposes are dead. I obviously still have copies of the repositories on my home server and can re-add anything if need be.