Some time in 2012, I decided to write my own blogging platform in JavaScript. I hashed out my goals with a friend over an evening. "It'll be simple," I said. "You'll make changes to a directory full of Markdown files, and that's your blog! I should be able to write the prototype in one evening!"
If you want to read some mediocre code from early in my JavaScript career, that prototype code is on the internet. I was still figuring out Node best practices, and dipping my feet into functional programming.
It took me a surprising amount of work to write something that could replace Wordpress as my blogging platform, let alone something that somebody else could conceivably download and use.
This post is about the work I did to Make It Happen.
Making up my mind
The project's specific goals didn't solidify until after I'd started writing it. I eventually settled on:
- static file deployments (no server-side code, deployable on Github Pages or shared hosting)
- no build step to turn markdown into html
- browse to new pages without waiting for a round-trip to the server
I had dream features as well, features I wish all content sites had. Interestingly, these features all come from MediaWiki:
An edit button on every page
I've been editing Wikipedia for years, and my first instinct when I see poor grammar or overly-long paragraphs is to click an "edit" button and fix the problem. Sadly, most blogs don't have an edit button.
Everyone should be able to feel the catharsis of fixing someone else's mistakes on the internet.
Easy linking to other pages
I shouldn't have to think about the deployment environment when I reference one of my other posts. When I link to another post I've written, it should be by the file name, and nothing more.
Having to link to http://mysite.com/blog/#!/my-awesome-posts/some-file.md is unacceptable. I need wiki-style [link.md|linking].
Embeddable templates
This is MediaWiki/Wikipedia's killer feature. Any page can be embedded in any other page - and since you can pass parameters to the embedded document, you can use templates to build domain-specific languages for content.
I dug in to MediaWiki templates a bit when administrating the MediaWiki documentation site for my previous employer, and am totally sold on the feature.
I wanted all that wiki power, but without the archaic MediaWiki formatting syntax.
Work that other people did for me
- I planned on using Substack's Browserify as my module/build system from very early on, letting me use large chunks of client-side code in my RSS and static file servers.
- DOM manipulation was not something I had any intention of rewriting, so I decided to try Rich Harris's sweet RactiveJS library. Life got even awesomer when I realized that Ractive's mustache syntax would be the perfect way to allow code in embedded templates.
- Even though the blog itself runs without any server-side code, it caches post content, and that means storage. I wrote everything to Rod Vagg's LevelUP interface, giving me an ideal key-value store. Noddity modules use localstorage in the browser, Redis on the RSS/static file servers, and in-memory structures in unit tests, with no code changes or storage mocks.
Why did that take so long to implement?
Having those goals in mind, I was able to start working on the other tools that I would need to exist.
Astute students of practical computer science will have recognized goal #3 above as something that requires caching, one of the two hard problems.
While it didn't occur to the me of yester-year, it should not surprise you to hear that most of the code I wrote to get Noddity running was in the caching layer.
Post metadata
Before I dug into the caching, I started by nailing down the post format. I needed to formalize a big ball of http-accessible blog content.
I wrote noddity-retrieval to abstract away accessing a directory full of Markdown files, and text-metadata-parser+weak-type-wizard to parse metadata from the top lines of my blog post files.
It turns out that putting metadata in plain text isn't a new thing - Jekyll's "Front Matter" has become the de-facto way to embed metadata in content files, though it was quite young when I started on Noddity. text-metadata-parser was the implementation I wrote, which has since changed to support "Front Matter" YAML.
Caching
Wheeeeeee! This is one of those things you instinctively know that you shouldn't be re-implementing.
Still, I couldn't find a solution on npm (the current winner of Having Code On The Internet) that matched my goals:
- Return a cached value as quickly as possible, regardless of age
- Update the cached value automatically every so often
- Expire keys based on the last time they were requested, not the last time the value changed
So, I wrote levelup-cache, and, in what I consider my Favorite Abstraction Layer in Noddity, I wrote the expire-unused-keys module to take responsibility for knowing when a key needed to be refreshed or dropped.
Since writing it, I've realized that expire-unused-keys is a great low-level building block in many caching solutions. I'm kind of proud of that library.
Next I wrote noddity-butler - which combines all of the above to return parsed posts from a Noddity content directory (like joshduff.com/content) as efficiently as possible.
The butler is used by both the client and the RSS/static servers to access a given site's content.
The actual front-end
After assembling those parts on my nights and weekends, I made the real website. Most of the front-end code by lines is taken up by the "recursively-embedded-posts" logic, which was not nearly as straightforward as it seemed in my head when I decided "hey, I should have embeddable templates!"
Eventually, I wrote noddity-renderer, which will turn a Noddity blog post into either static HTML, or recursively infinite Ractive elements that automatically update themselves whenever the noddity-butler reports any changes.
It's, uh, an interesting library. It turns out "static html" and "a self-updating front-end" do share code, but imply very different architectures to pull off reasonably. If you can give me any advice on the structure of the code I'd love to hear it.
"Real" web site features
I didn't want to drop Wordpress until I had an RSS feed, and search engines could index my individual pages.
The RSS feed wasn't difficult, all it takes is a link element pointing to the url of a server that knows how to turn querystring parameters into a feed of posts using Noddity Butler. Hesto presto, rssaas.
Making search engines happy with your Web 3.0 single-page app is a bit more involved - you have to add a special line to your html to tell the search engine spiders to visit a special querystring that you have to handle on the server - in the case of Noddity by using an .htaccess file to redirect the spiders to a server capable of serving static html versions of the markdown posts. I named that module seoaas.
It turned out those two applications shared a lot of code, so I moved the core out to the noddity-service-server. If you ever want to write a node.js server that references a remote Noddity blog, that's the library you probably want to be using.
Does anyone actually want to use it?
I've always admired the programmers who release polished projects that others can use to solve problems with a few clicks.
I wanted to release a project like that - even if it is a very narrow problem set, I want you to be able to use Noddity to easily solve real problems.
I registered noddity.com, and used the majority of a working-vacation with my wife at a bed-and-breakfast to write copy and fix bugs that I discovered in the process.
Cloning from github, running npm install + npm build, and deleting the leftover cruft is almost acceptable for just me, but it's not very friendly for people who want to give it a quick try.
So, I wrote a magical Noddity installer. It requires typing some things into the command-line, but not many. Install one global module from npm, and you can create a fresh Noddity project in any directory.
A winner is me?
So I'm a Real Open Source Programmer now, I suppose! I wrote code that other people could conceivably use to solve their own problems. A few of my sexier friends have already deployed their own sites based on Noddity's code.
Noddity has some stars on Github, but I don't know how to market it. Given how much work the most popular open source developers seem to do to support their code, I'm not 100% sure I want to.
But I am a selfish human with selfish human desires. I want people to use my code because it would validate the work I put into it.
Plus, I get a special lift when I act as a multiplier to someone else.
So I want Noddity to seem usable, and painless enough to override the "it would be easier to write it myself" instinct we all feel, and solid and featured enough that people would actually consider deploying it instead of Wordpress or Mediawiki.
RFC
To that end, I want you to message me. If any part of the documentation seems incomplete, or you run into anything stupid, message me on Twitter. If you think some part of the code is dumb, or you have questions about whether or not it could be bent to your own particular use case, send me an email.
If you want to contribute, I would be flattered and happy. Submit an issue or pull request, and I will hug you if I ever meet you in person. Unless you don't like that sort of thing, in which case I'll give you a cool-guy nod.
To infinity and beyond
I don't expect to make any great profits from Noddity, but I still want to work on improving it - easier theming, easier plugins, better parsing of wiki-links in the Markdown content.
If Noddity never grows, and simply stays as the cool backbone of 3-10 web sites for the rest of its existence, I won't be disappointed - I wrote it for my own use cases, and I'm satisfied so far.
But if Noddity grows, that would be sweet, and I'd love the excuse to put more polish into it.