LLAMATTF — Run LLM Inside A Font File

Who would have thought an LLM could be run inside a ttf font file….
I was a bit confused too when i found this https://fuglede.github.io/llama.ttf , an LLM and an inference engine for it, but actually a ttf file.

Well.. I tried loading the font inside Firefox for a sample HTML input element and tried typing stuff out but got no result and upon looking for docs, I realized there was tweaks to be done on some shared libraries to make this thing work.
So about that..

Harfbuzz

Harfbuzz is a text-shaping engine used in Android, Chrome, ChromeOS, Firefox, GNOME, GTK+, KDE, Qt, LibreOffice, OpenJDK, XeTeX, PlayStation, Microsoft Edge, Adobe Photoshop, Illustrator, InDesign, Godot Engine, and other places.

So the latest version of this library comes with a web assembly shaper, we can write our own text shaping engine and embed it into a font file. we can use this text-shaping feature through their wasm shaper interface. You can read more about the api functions available and documentation here

libiwasm

To make the libharfbuzz library work with WebAssembly feature, I needed wasm-macro-runtime built from source.

Setting up the libraries..

I had libharfbuzz library preinstalled in my Arch Linux(I use Arch BTW) system but libharfbuzz by default comes with WebAssembly feature disabled obviously for security reasons, so I had to manually build libharfbuzz with wasm feature enabled and wasm-macro-runtime library. This was the hardest part not gonna lie.

Extra dependencies you are going to need for building:

meson, pkg-config, ragel, gcc, freetype2, glib2, glib2-devel, cairo

Install these from your distro package manager

Building libiwasm

I cloned the wasm-macro-runtime repo and the harfbuzz’s wasm_shaper docs had build instructions for the wasm feature in the engine.

for building wasm-macro-runtime:

$ cmake -B build -DWAMR_BUILD_REF_TYPES=1 -DWAMR_BUILD_FAST_JIT=1
$ cmake — build build — parallel

If you want to install the libiwasm shared library globally, you will have to run this too.

$ sudo cmake — build build — target install

Building harfbuzz with WebAssembly support

Make sure to clone the latest version of the harfbuzz for an error-free build.
for building the library run:

$ meson setup build -Dwasm=enabled

Make sure your output shows the WebAssembly option enabled

example output:

…
 Additional shapers
 Graphite2 : NO
 WebAssembly (experimental): YES
…

Now run this command to build it

$ meson compile -C build

Note:If you encounter error with meson not being able to find the shared object libiwasm you have to manually copy the libiwasm.so file built from source to src directory inside the build directory of harfbuzz (build/src).

You will have harfbuzz built with wasm enabled inside the harfbuzz/build/src/ (assuming build was the directory used).
There will be a lot of files inside the directory, the only important one is the `libharfbuzz.so.0.XXXXX.0` file this is the shared library we need to run wasm inside the ttf file.

Now to preload the the libharfbuzz and libiwasm libraries ( using wasm enabled libharfbuzz is a bad idea from a security point of view ) we will use the LD_PRELOAD environment variable.

$ export LD_PRELOAD=/path/to/libharfbuzz.so.0.XXXXX.0:/path/to/libiwasm.so

Now we can copy the llama.ttf file to our .local/share/fonts/ directory inside our home directory and run `$ fc-cache` command to use the font.
since Llama ttf uses Open Sans as the base font, you will have to search for Open sans in the font selection menu to see LLama ttf and use this font
Any app that uses libharfbuzz library can be used for this, I used Gedit.

Running the LLM

With all that setup, I tried typing texts into Gedit, but there was no text generation happening except the word ‘Open’ was being magically replaced to ‘Llama’ whenever i typed it.

In order to understand what the llm inference engine was doing I scanned through the source code and found this Rust code:

 let res_str = if str_buf.starts_with(
 “Once upon a time!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!”,
 ) {
 let count = str_buf.chars().filter(|c| *c == ‘!’).count() as usize;
 let s = format!(“{}”, next_n_words(&str_buf, count + 5–70));
 debug(&s);
 s
 } else if str_buf.starts_with(“Abracadabra”) || str_buf.starts_with(“Once upon”) {
 format!(“{}”, str_buf).replace(“ö”, “ø”)
 } else {
 format!(“{}”, str_buf)
 .replace(“Open”, “LLaMa”)
 .replace(“ö”, “ø”)
 .replace(“o”, “ø”)

This was defined inside a function that calls next_n_words(), an interface function to generate text using the embedded LLM , whenever the text starts with “Once upon a time!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!” and for every “!” character pressed.

Here is the full code

So I typed “Once upon a time!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!” and the font started generating text for me for every “!” character pressed afterwards. :)