Hello, World. Ep. 2020
I've developed a habit to overhaul my personal site from scratch every now and then. But this time, it feels more special as I built my framework. In this article, I will explain why, the concept and some of the processes I went through while building this framework to deliver this website.


August 11, 2020

After several years of struggling with migrating between static site generators — from Jekyll, Hexo, Middleman to Hugo — I finally decided to build my publishing framework.

Two problems that I frequently encountered, which become the foundation for building this framework are tight-coupling between content & presentation layers and switching costs for migration between static site generators.

Before we go any further, let's discuss the content and presentation layers. Content is the material that you construct — in the context of a static site generator is generally formed as Markdown files. On the other hand, the presentation layer is the component to process raw content into a more representative form — such as articles, listicles, microblogs, photos, web pages, you name it. It can be anything.

The diagram below outlines the standard process when we use a static site generator.

Common static site generator process
Common static site generator process

Current Tools Downside

In almost every circumstance, the data objects on the editing side are the content itself along with its metadata. On the other hand, the presentation side produces a more sophisticated layer by including various composite layers on top of the content itself.

Over the years, I realized that content data objects for editing and presentation — in the context of the publication process — are significantly different.

For example, while the content and its metadata — I use Markdown and FrontMatter respectively — are in the simplest form, the presentation layer is more complex. The diagram below illustrates how the content and presentation layer works together.

A static site generator is a perfect tool for building a simple website. However, to have various forms of presentation with more custom components —  in this case, "complex" is the more proper word — each static site generator has its way for the implementation. For example, in Hugo, we can implement shortcodes for this purpose. Here's one example using Hugo:

Shortcode in Hugo
Shortcode in Hugo

What I don't like is the use of shortcodes in raw content makes the content itself no longer universal which also means that it can no longer be processed by other static site generators.

This is exactly what I experienced when migrating from Jekyll to Hugo. Almost all content with partial must be changed toshortcode — and of course, all Jekyll templates have to be changed as Hugo accepts. The tight-coupling between the content and presentation layers makes it difficult to migrate.

I knew I'd experience this situation from time to time, so instead of using existing tools and migrating back and forth, I decided to build my publishing framework, which I called Monoverse.

Introducing Monoverse

Monoverse itself is a publishing framework.

Monoverse wasn't built as a CMS as I think there are fundamental differences between CMS and publishing framework. CMS allows us to modify content until it is published on the website and in some applications it has a content (approval) workflow — which is intended for end-users. Meanwhile, the publishing framework is a collection of components needed to do publication.

Monoverse only plays a role in delivering content and does not dwell on how you create the content to provides freedom for creators to be creative with the content they have.

In Monoverse, there are two important components, namely content processing and publishing.

The function of content processing is to take data from multiple sources — currently, it can retrieve from the filesystem as well as GraphQL and HTTP API endpoint — and standardize that data into Markdown with FrontMatter.

The second component is publishing. The function of the publishing component is to integrate the necessary components on a web page — such as image captions (like the diagram above), listicle format (which you can see here) — and finally, do all the build processes for rendering up to the last build commit to the Git repository — I use GitHub for this.

Monoverse Workflow
Monoverse Workflow

All of these components in the diagram was designed to suit my workflow, so it might not suit your needs. I very much feel comfortable to just upload a file and the system will automatically adjust and publish the content. And that's what I'm trying to build within Monoverse.

At some point, I realized that although the idea for Monoverse — separating content processing and publishing — is very basic, the implementation is rather complex.

Getting Technical

I cannot stress enough that the Monoverse would not exist without the help of various open-source libraries.

Monoverse uses two programming languages, Go and Python — structured as microservices according to the two different domains. Go is used because I need struct — a typed collection of fields — to standardize the various document types to Markdown. Python is used to render the data source into an HTML display.

Some of the libraries that I use are as follows:

Libraries Name Link
Blackfriday https://github.com/russross/blackfriday
Go GraphQL https://github.com/graphql-go/graphql
Pandoc https://github.com/jgm/pandoc
Python Markdown https://github.com/Python-Markdown/markdown
Git Python https://github.com/gitpython-developers/GitPython/

To reiterate, Monoverse was built with a microservices approach with two domains, content processing, and publishing.

How Monoverse Works

From the starting point in developing the Monoverse, I knew that there was still a lot of room for improvement. To give space for these improvements in the future, Monoverse then conceived with microservices and asynchronous communication. The purpose of using microservices in this context is to do decoupling between services so that if there is something that needs to be improved, it will not interfere with other components.

After I identified the role of each service, the next challenge was how to connect one service to another. For this, I use AWS Step Functions to be able to orchestrate between services.

Monoverse Technical Diagram
Monoverse Technical Diagram

This is a diagram illustrating how the Monoverse works. Let me walk you through how every piece works together.

  1. The process will work based on events sent by Amazon S3 — via PUT objects — or via APIs triggered to the Amazon API Gateway endpoint.
  2. Processes can also be initiated based on cronjobs defined with Amazon CloudWatch. All of those events will call the AWS Lambda function to start the process.
  3. AWS Lambda will load the configuration and if there is a data source in GraphQL (or HTTP API endpoint), the data will also be populated by calling the GraphQL source.
  4. Then, AWS Lambda will initiate the process and pass all inputs to AWS Step Functions.
  5. Almost all of the magic is at work in AWS Step Functions which will orchestrate microservices — content processing and publishing. The result of the AWS Step Functions process is a complete website — HTML files, CSS, JS, images, and sitemap.
  6. All website files will be committed to the GitHub repository.
  7. The GitHub repository has been integrated with AWS Amplify Hosting so that every commit that occurs will trigger the build & deployment of the website.

And that's it.

Hello, World

When I finished working on the first iteration of Monoverse, there were moments where I felt that the developers who contributed to static site generators such as Hugo, Gatsby, and Jekyll had done a great job. Most — if not all — static site generators available today are very easy to use for building websites by abstracting the complex processes in the background.

It's just that they don't suit the way I create and deliver the content.

Despite all the problems I faced during the development period (and possible bugs and lots of backlogs in the future), I have learned many things. I've learned how to use some new tools that I have never used extensively, such as NPM, Webpack, Parcel, exploring the Pandoc library Go to building extensions for Python-Markdown.

I knew I would have so much fun building this stuff, but this level is beyond imagination. I'm enjoying the whole process and glad to have this website is built using the framework.

So, yeah, consider this a hello from Monoverse. 👋


Enjoyed this post? Please share.
Coming Up Next.
{{rec_post.synopsis}}
{{rec_post.synopsis}}