What's the point of Bibliograph?

So, you're curious about Bibliograph. It came out of a discussion with some folks about how we can model a book lexicon and is a direct product of what Livtet's move to rely on ATProto for all of its work — from shelving content to finding what books are trending across the Atmosphere.

The Recent Outage

If you're not familiar with the internals of SQLite, you can skip this section.

As I was importing the books from OpenLibrary's torrent dump, Bibliograph ended up generating a 48 gigabyte WAL file that locked up the little 2GB RAM machine it's running on¹. I had to do the following:

• Cap the WAL size to 128M using journal_size_limit = 134217728
• Lock it to be synchronous for a moment
• Run a throttling on the reconnection to Olamaelcu's tap instance

All of this is described in this commit. These changes are needed to keep it running well on a tight machine and my goal is to keep it running on this sized machine for as long as possible so anyone can pick this up and run it on commodity hardware.

However, something else was happening. Can you tell what's wrong with this SQL query?

results = await db.query.books.findMany({
 where: or(
  like(books.title, `%__health_check__ __health_check__%`),
  like(books.author, `%__health_check__ __health_check__%`)
 ),
 limit,
 offset
});

Running this query against a 4G SQLite database sounds okay if you don't mind waiting 30 seconds on a 2GB RAM machine. This is what led me to do early termination. Basically, running a full-table scan on 3.4 GB of books slowed things down a lot because LIKE '%...%' can't use a B-tree index; especially within a Node loop. This __health_check__ bit is coming from Livtet every 5 minutes to make sure that Bibliograph is responsive — this was how it knew to send interactions to Bibliograph (like enriching book and author information, fetching shelving information, etc).

The Current Status

As of now (time of writing), Bibliograph knows of 5,111,432 books (currently 5,111,432 books). This allows Livtet's public search to return all of the books it knows without having to ask Google Books or OpenLibrary (directly) anymore. This also powers the logic around the soon-to-come jigsaw puzzle game that Livtet has based on the cover of books — the first of many attempts to build engaging experiences around books that don't involve reading (or just reading, you should still be reading)!

I'm very open to feedback, ideas and questions about these apps! Send feedback to their pages for Livtet and for Bibliograph.