6 Developing BreezeWiki🔗

6.1 Racket🔗

BreezeWiki is programmed in the Racket programming language.

If you are new to programming entirely, BreezeWiki is likely not a good introductory project for you. I recommend purchasing a physical book about programming targeted at beginners. (In 2022, internet searches for beginner programming questions are sadly filled with low quality results or straight up misinformation.)

If you already know programming concepts but are new to Racket, I recommend reading Quick: An Introduction to Racket with Pictures and trying the instructions for yourself. This should fill you in on the basics of doing real things with the Racket language. After reading that, if you want to know the fundamentals even more in-depth, you can check out the Racket Guide, which is better to jump around in rather than read from start to end. The Racket reference documentation is on the same website.

I recommend using the official DrRacket IDE to write Racket code, particularly if you are a beginner. It applies useful indentation automatically and it has helpful hover hints. Once you are familiar with Racket, you could configure Emacs, VSCode, Vim, or your favourite other editor to understand Racket’s style rules, though you might still miss out on syntax highlighting or hover hints in some situations.

6.2 Files🔗

breezewiki.rkt and dist.rkt are entrypoints. They do as little as possible, just requiring the page functions and starting the web server.

src/page-*.rkt contains the page functions. Each file has instructions on how to render a specific kind of page. For example, src/page-wiki.rkt renders the usual wiki pages by contacting Fandom servers, altering the HTML tree, and sending the response to the browser.

Everything else in src/ is a utility file that is required in by the pages as needed.

6.3 Fandom endpoints🔗

BreezeWiki mostly uses the MediaWiki APIs rather than scraping full pages pages. If I need to add functionality in the future that can’t be covered through the APIs, I would consider changing to scraping full pages.

6.4 HTML tree transformations🔗

MediaWiki’s Parse API returns the contents of a page as HTML, but returning this wholesale isn’t good enough. Several modifications need to be made to the page before it’s suitable for display on BreezeWiki, such as altering links to have the correct prefix, making all images visible without JavaScript, and enclosing tables to allow horizontal scrolling.

To do these transformations, the update-tree function recurses through the whole HTML tree from MediaWiki, executing a function on each element. This function returns the new element to replace with, allowing update-tree to eventually build a whole brand new tree. If the function decides that there’s nothing to be done for a particular element, it just returns the same element. You can see the largest example of this in src/page-wiki.rkt, where many transformations can be applied to every element.

Background: Racket is a dialect of Lisp, a class of programming languages that are good at manipulating lists of symbols. XML and HTML trees are represented as X-expressions, which you can read a bit more about within the Pollen documentation: X-expressions. (The rest of the Pollen documentation is unrelated.) An X-expression is often called an xexp or an xexpr for short. BreezeWiki includes src/xexpr-utils.rkt for some helpful functions that query and manipulate X-expressions.

6.5 Programming🔗

If breezewiki.rkt is launched (in a REPL) it will dynamically load in the pages and watch them and their dependencies for changes. If you edit and save a file within src/ it will likely be reloaded automatically.

Creating a new page requires editing src/dispatcher-tree.rkt to define which URLs the page should appear from, and breezewiki.rkt and dist.rkt to cause the page to actually be loaded. The first time you create a page it won’t be loaded automatically and you’ll need to restart BreezeWiki.

Automatic reloading helps with rapid development and avoiding long process startup times. But if the long startup times still frustrate you, you can use raco make breezewiki.rkt dist.rkt to byte-compile files for faster startup.

6.6 Testing🔗

Most files also contain unit tests, within the (module+ test sections. Evaluating a file within either DrRacket or Emacs-racket-mode automatically runs the tests in that file. Since almost all functions are pure functions (i.e. operate based on their inputs and outputs rather than relying on application state) they are easy to unit test.

You can run all tests in all files with raco test --direct (grep -l 'module+ test' (git ls-files)) which is helpful to check for regressions before committing.