DeepakBhalla

Creating a Long Lasting Note Taking System (2017)

By Deepak Bhalla Tue 26 December 2017 9 min read

In this blog I create a system for storing written notes and a second system which allows it to be presented publicly for others to view and scrutinise.

The Problem

I'm attempting to get a little more organised with a new set of notes (although its current state is little more than notes from a course I am taking at the moment). I have a real need to review the things I had learned in the past and I also wanted to start writing down more things I'd learned. I want to start building on my previous knowledge by making my notes easy to read and view. Unfortunately when I went to review my old notes (taken during meetings and courses using Word and Pages and stored on Google Drive) I was presented with hundreds of opaque .docx and .pages files which were a pain to try and extract information from. Although these notes were small files, presented in a well structured directory system they still lacked the usability I would like going forward (formalised here).

There was a number of problems with the way I was taking notes. For instance, the only real way to text search through notes was to use my OS finder or a command line search tool which varied whether I was at home or on my work laptop. I also noticed there was a lack in the quality I had anticipated from the notes, partially because many of them had not been reviewed by anyone but me and also because they hadn't been viewed again once they were created. I think notes should be scrutinised regularly (and by many people), since they are primary a learning tool 'many fresh eyes' makes a lot of sense.

I've been searching for a good note taking system for a while so I decided to spend a day or two creating my own primary note taking system (insert image of that one xkcd about competing formats) which puts together some best practices that fulfil many of my needs.

Scroll down to the Solution if you're already bored.

System Needs

My goal was to create a system where where I had end-to-end control of my notes (preferably without relying on a SaaS service). Notes also needed to be able to be scrutinised to be useful. I wanted to be able to make them public without losing any of the flexibility of a local set of notes (such as the ability to quickly reflect amendments publically and always presenting the latest version of the notes). The note taking system had to be:

  1. Easy to edit
  2. Long lived (capable of potentially being viewed 10+ years later)
  3. Highly available (able to be downloaded securely from anywhere and modified)
  4. Self Referential (Linking between documents locally and publicly)
  5. Suitable to contribution by others
  6. Version Controlled
  7. Able to be made public on a note by note basis (readable over the internet, reflects any edits immediately)
  8. Self-hosted 1

Potential Alternatives

There are lots of great options for note taking - none fitted my needs. Some of the most popular were:

  • Evernote - Expensive, I also don't trust it to exist in 10 years and it would be a huge pain to move to another service if there was a price increase I wasn't prepared to pay
  • OneNote - Great but I dont own Microsoft Office anymore and I want something more open source and controlled by me.
  • Google Keep - It isn't at the productivity level of its competitors (although Google has had services which were)

I know there must be other solutions there were a tonne of alternatives. A self hosted service like this must already exist somewhere.

Solution

What I came up with is a two part Note-Presentation system.

Note Component

The Note Component is a git repository which contains a set of notes written in Markdown, reStructuredText and contain TeX (for maths). Having a large number of open source document formats allowed me separate out how my notes were taken (in Markdown) from how I presented them (Using HTML and Jinja2 templates).

These notes are what I have open in my editor all day every day and where I store all private and public notes. Anything and everything goes in here from meeting notes to details on the current book I am reading. I currently use Visual Studio Code to edit these notes since its speedy and contains its own 'Markdown: Open Preview' if I need to check formatting before I commit to the repository.

  • System Needs Tackled
    • Need #1:Easy to edit through source control
    • Need #2: Stored on a centralised git repository
    • Need #3: Available for download given the correct SSH key
    • Need #6: Version controlled using git

Presentation Component

The Presentation Component is the static site generator Pelican. Static site generators generate a full HTML-only website using raw data (in this case our document files .md and .rst) and templates. Pelican is able to natively interpret Markdown (.md) and reStructuredText (.rst), it can also interpret TeX through MathJax.

I took advantage of Pelicans PATH constant to look for a directory called content as a source for all document files. The content directory doesn't exist but is actually the Note repository which is pulled right before Pelican generates it HTML output and is deleted afterwards.

Unfortunately there is not a true separation of concerns and the presentation component does enforce some directory constraints on the Note repository (if it the documents needs to be made publicly available). For instance Blog posts must be present under {NOTES_REPOSITORY}/blog/**/{filename}.(md|rst) and notes under {NOTES_REPOSITORY}/notes/**/{filename}.(md|rst) if Pelican is to pay attention to them. Luckily any documents or files that fall outside of these special directories (or fall inside those structures and do not contain the metadata status: published in the header) are just ignored by Pelican which is handy private notes.

  • System Needs Tackled
    • Need #7: Can be made public through pelican
    • Need #8: The git repository also handles deployment through rsync

Its important to note that the Presentational component only contains only presentation/theme data. The Note component functions independently of the Presentation component and regular day to day note taking never involves me opening the Presentation component. When something changes in the core Note repository a hook is called which calls make publish on the Presentation repository and generates and deploys the website in its full form.

  • Note: In order for a blog post to generate a HTML output (i.e. make it publicly available) it must add markdown metadata such as a Title and Date Created. If it doesn't contain this data it wont be published by Pelican.

  • Note: I fully intend to open source the Presentational component at some point in the future.

Partially solved issues

Some system needs weren't entirely tackled in an expandable way:

Need #4: Self Referencing

I am able to link to other documents but only through a directory structure which is enforced by the Presentational component. Links also do not function locally. A link to an article might be written as [Article Link](/blog/{category}/{article_slug}) in markdown which translates to the HTML output of the file {NOTE_REPOSITORY}/blog/{article_slug}.(md|rst). I therefore have an issue where all of the links I use in my notes repository are only useful when the files are translated to HTML. They do however roughly translate to something human traversable.

The alternative is Pelican's internal file-linking sytem which is proprietary and tightly couples linking in notes to the functionality of the presentational component.

Need #5: Suitable to contribution by others

Since I also store private notes and public notes together there is very little chance I will ever make the core Note repository public. That is unless I can figure out a way to keep all my notes together (so locally I only have to look in one place) and still make the public notes available to PRs (if needed) or comments.

Other Issues

Need: A more viewable UI

I have to admit it, Markdown and TeX is not a winning combination in terms of viewability. I really need something that doesn't get in the way of reading. I already mentioned this but VSCode has its own markdown preview (which is excellent) but TeX in markdown is its own issue and presenting maths should really be a concern on the presentational layer.

Need: Classification beyond Note or Blog

I had already been using Pelican as my blogging engine for a while to massive successthere is no sarcasm formatting in markdown. So it was relatively easy to expand it to deal with my note taking system.

Pelican has the concept of blogs and pages and not much else. In my world a note is actually a Pelican page which is typically used for things like an about-mes or other static HTML found on most blogs.

I use the pelican config to coerce content/notes to be treated as pages (a slight misuse of Pelican). One of the issues I will probably suffer from in the future is what If I want to add another classification of document other than a blog or page like a presentation - what then?.

Final Notes

I have a tonne of things to say about the ideal solution. This is far from it, so I will spend some time in the future laying out my ideas of a 'perfect' system that really is able to make notes and blogs publicly available. I also know that solutions probably do exist for this but I cannot seem to find anything I like.

Interesting Reads

Dont Use Markdown - Eric Holscher

  1. I do not like the idea of relying on a SaaS service which may change its pricing model whenever it pleases. I am less exposed to this risk via self hosting.
Deepak Bhalla