The first iterations of FooSoft.net go back to the early 2000s, back when I was still in school. Back then, the site was
entirely hand-written (proudly in Notepad!) and relied extensively on HTML
framesets to reduce boilerplate code. Over the years, as frames
fell out of favor due to issues with search engines and bookmarks, I had to rethink this approach.
This website has been rebuilt more times than I care to count. The stack evolved from using
Wordpress, to ConcreteCMS, to
Pelican, to Hexo, to Hugo before briefly pausing on
Metalsmith. Static site generation seemed to be the way to go (and yes, of course my Wordpress
site was hacked) for simplicity and portability. One problem I ran into was that most of the static site generators
placed seemingly arbitrary restrictions on site structure. To enumerate a few of the issues:
Out of all the static site I generators evaluated, Metalsmith came the closest to what I needed. I could finally begin
to see a path forward to a homepage structured the way I wanted. Unfortunately, Metalsmith was quite temperamental due
shortcuts added in the pipeline design in what I can only assume to be an attempt to make the generator easier to use.
It became clear to me that I needed to build my own tooling to get the results I could be truly satisfied.
Goldsmith, my Golang-based static generator was born in late 2015.
Here is a sampling of some hard requirements I relied on to guide me through implementation:
Let’s just dive into the code. Many details have been omitted for the sake of brevity; let’s just focus flow for now:
Recursively scan all the files in the contentDir and start pushing them through the pipeline:
goldsmith.Begin(contentDir).
Enable caching in cacheDir for supported plugins; this helps avoid reprocessing of unchanged resources:
Cache(cacheDir).
Enable cleanup of extra files found in the buildDir that are not build output:
Clean(true).
Chain the rule plugin to enable
per-directory rules, which are specified in a rules.toml file:
Chain(rule.New()).
Rules can skip files matching a patterns set. To prevent .git directories from being published, we can use the
drop directive:
[[drop]]
accept = [
"**/.git/**",
]
Rules can also help specify metadata on files that we do not want to modify directly (such as the README.md file
for a project). We can instead match the file and apply properties to it externally:
[[apply]]
accept = ["goldsmith/README.md"]
props.Collection = "ProjectsActive"
props.Description = "Static pipeline-based website generator written in Go."
props.GitHub = "goldsmith"
props.Tags = ["generator", "golang", "goldsmith", "mit license", "web"]
Repositories on GitHub are expected to contain the project description inside of a README.md file. We want to
rename all instances of README.md to index.md and ensure that each project directory will eventually contain an
index.html file. I added a simple indexRenamer helper plugin to automate this:
Chain(new(indexRenamer)).
The implementation of this one-off plugin is shown below:
type indexRenamer struct{}
func (*indexRenamer) Name() string {
return "indexRenamer"
}
func (*indexRenamer) Initialize(context *goldsmith.Context) error {
context.Filter(wildcard.New("**/README.md"))
return nil
}
func (*indexRenamer) Process(context *goldsmith.Context, inputFile *goldsmith.File) error {
dir := path.Dir(inputFile.Path())
inputFile.Rename(path.Join(dir, "index.md"))
context.DispatchFile(inputFile)
return nil
}
Chain the frontmatter plugin to
parse and strip out any metadata prefixed onto our files:
Chain(frontmatter.New()).
This metadata consists of arbitrary properties and can be defined in several formats. The TOML frontmatter for this
page is:
+++
Date = 2023-10-10
Description = "Exploring the uniquely custom process of generating this website with Goldsmith."
Tags = ["web", "goldsmith", "generator"]
+++
Chain the markdown plugin to convert
all Markdown files to HTML fragments:
Chain(markdown.NewWithGoldmark(gm)).
At this stage, these are not yet documents, but merely excerpts that must have header and footer markup templates
applied to them before they become functional pages. This plugin uses the
goldmark Markdown parser, which supports GitHub extensions, syntax highlighting
and more. The desired features can be optionally configured by passing in a goldmark instance:
gm := goldmark.New(
goldmark.WithExtensions(extension.GFM, extension.Typographer),
goldmark.WithParserOptions(parser.WithAutoHeadingID()),
goldmark.WithRendererOptions(html.WithUnsafe()),
goldmark.WithExtensions(highlighting.NewHighlighting(highlighting.WithStyle("solarized-dark"))),
)
Push a condition filter that
ensures that pipeline manipulation within its scope only happens when the page is not being generated in devMode
(preparing content for deployment to the web server):
FilterPush(condition.New(!self.dev)).
Chain(absolute.New().BaseUrl(baseUrl)).
Chain(minify.New()).
FilterPop().
When generating the site for deployment, we chain the absolute
plugin to convert all relative URLs
to absolute ones using the baseUrl (https://foosoft.net) prefix. This is required to ensure that all relative
references are resolvable when the site’s content is displayed in an RSS reader. We also apply the minify
plugin to reduce the site’s content to
the minimal possible size without impacting behavior. We do not want to do this in development mode since it makes
debugging the code a lot tricker! Once these two steps are completed, we pop off the condition filter and resume
unconditional processing.
Chain the syndicate plugin to
generate feeds for selected articles (like this one):
Chain(syndicate.New(baseUrl, "Feed").WithFeed("posts", feedConfig)).
Only pages with the Feed metadata property with the value posts are considered during feed generation. We must
also provide a basic site configuration and declare the desired feed formats.
feedConfig := syndicate.FeedConfig{
Title: "FooSoft Productions",
Url: baseUrl,
AuthorName: "Alex Yatskov",
AuthorEmail: "alex@foosoft.net",
RssPath: "feeds/posts.xml",
AtomPath: "feeds/posts.atom",
JsonPath: "feeds/posts.json",
ItemConfig: syndicate.ItemConfig{
TitleKey: "Title",
CreatedKey: "Date",
DescriptionKey: "Description",
ContentFromFile: true,
},
}
Chain the collection plugin to
group related pages into lists based on the value of the Collection metadata property:
Chain(collection.New().Comparer(collectionComparer)).
The resulting collections are ordered using a custom comparer function which groups items first by Date and then
Title properties:
collectionComparer := func(i, j *goldsmith.File) bool {
if iData, iOk := i.Prop("Date"); iOk {
if jData, jOk := j.Prop("Date"); jOk {
return iData.(time.Time).UnixNano() > jData.(time.Time).UnixNano()
}
}
if iData, iOk := i.Prop("Title"); iOk {
if jData, jOk := j.Prop("Title"); jOk {
return strings.Compare(iData.(string), jData.(string)) < 0
}
}
return strings.Compare(i.Path(), j.Path()) < 0
}
Conditionally chain the index plugin to
generate directory index pages for paths that do not match the **/*.gohtml pattern:
FilterPush(operator.Not(wildcard.New("**/*.gohtml"))).
Chain(index.New(indexMeta)).
FilterPop().
These template files are consumed later in the pipeline and we don’t need to include them in the build output. Newly
generated index pages are assigned the metadata configured in the plugin:
indexMeta := map[string]any{
"Layout": "index",
}
Chain the tags plugin to generate tag
indices from page metadata:
Chain(tags.New().IndexMeta(tagMeta)).
By default, this plugin builds lists based on the contents of the Tags property. Some additional configuration may
be provided to to specify what metadata is assigned to newly generated tag index pages.
tagMeta := map[string]any{
"Area": "tags",
"Layout": "tag",
}
Chain the forward plugin to generate
stub pages for content on the homepage that has been moved to a different URL:
Chain(forward.New(forwardMeta).PathMap(forwardPathMap)).
We must specify the URL mapping as well as any additional metadata that should be assigned to newly generated
forwarder pages:
forwardMeta := map[string]any{
"Layout": "forward",
"Title": "File Moved",
}
forwardPathMap := map[string]string{
"mangle/index.html": "/projects/mangle/",
"projects/argwrap/index.html": "/projects/vim-argwrap/",
"projects/kanji-frequency/index.html": "/posts/kanji-frequency/",
"projects/yomichan-chrome/index.html": "/projects/yomichan/",
"yomichan/index.html": "/projects/yomichan/",
}
Chain the layout plugin to apply Go
HTML templates to our page HTML fragments:
Chain(layout.New()).
By default, the plugin searches for the Layout metadata property to decide which template to apply. All metadata
associated with a file can be referenced from within these templates. The templates themselves are stored in
*.gohtml files which are consumed by the pipeline along with all the other inputs. For example, the top-level
template for the current page looks as follows:
{{define "page"}}
{{template "header" .}}
{{.Props.Content}}
{{template "footer" .}}
{{end}}
Chain the thumbnail plugin to
automatically generate thumbnails for all processed image files:
Chain(thumbnail.New().Style(thumbnail.Crop)).
By default, this produces PNGs thumbnails resized to 128x128 pixels. The filename is automatically determined by
appending -thumb.png to the source image filename after stripping the extension. For example, foo.png becomes
foo-thumb.png.
Conditionally chain the livejs plugin
in development mode to inject JavaScript to refresh the page when a change is detected:
FilterPush(condition.New(self.dev)).
Chain(livejs.New()).
FilterPop().
This script continuously queries the server to detect changes. It should never be enabled on the homepage once it
has been deployed.
Conditionally chain the minify plugin
once more to also minify the HTML templates we applied a couple of steps ago:
FilterPush(condition.New(!self.dev)).
Chain(minify.New()).
FilterPop().
This plugin is chained twice so that both the RSS output (which is spun-off earlier in the pipeline) as well as the
full site are minified.
Finally, write out all the generated files to the buildDir directory.
End(buildDir)
Today, over 400 files are generated by Goldsmith and output into the target directory. This process is highly
parallelized and takes just over 100ms to complete with caching enabled and 130ms for a full rebuild. The results
are fully deterministic, which makes inspecting content changes between website deployments easy.
If a plugin reports an error while processing a file, the file stops propagating through the pipeline. A list of errors
is returned upon calling End. These errors contain the name of the file in question as well as the faulting plugin.
One unique thing about the on-disk organization of FooSoft.net is that projects are actually included as submodules
within the website repository. This makes it easy to keep the project’s README.md up to date with the data shown on
the generated homepage: it’s literally the same file! I view this as an invaluable feature for keeping the project brand
separate from where the project is hosted.
While this may seem like there are a lot of steps required to build this site, the actual cognitive load is very low.
The statically generated output for FooSoft.net is a perfect match for my expectations and requires no hacks. Iteration
is fast and adding new features is a breeze. If this seems interesting, take a look at the
goldsmith-samples repository for some simple examples to get started!