Semantic Poetry Lab

Russian Poetry Lab: a semantic text explorer

I have lots of experience with text semantic similarity due to my work in applied scientometrics, expert search etc. This powerful approach allows us to measure how similar a pair of texts (or other media) are in what they are about. It is achieved via ML models trained on vast text corpora, so every text now can be turned into a multidimensional vector (set of numbers) that allows us to search for similar/dissimilar vectors (so, other texts) lightning-fast over huge collections.

I like poetry and am fascinated by how close these semantic vectors, when encoding poems and their parts, are to the notion of motif.

Merriam-Webster defines it as:

a usually recurring salient thematic element (as in the arts), especially : a dominant idea or central theme

Motifs are of course central to poetry, and the shimmering, not-quite-comprehensible for humans notion of a multi-dimensional vector of meanings is very close to this branch of human art.

I thought: let's make a semantic search over Russian poetry, to connect poems, authors and generations via their motifs, and do it in a visual way! So that a user can seed any poem and get similar ones, build networks of similar poets, find poems that are most different from one poet's own oeuvre, and search for a Russian poem most similar to any text in any language.

After that idea came to me, everything was straightforward:

  1. I found a dataset of 16k poems spanning Lomonosov to Brodsky. Thanks to Ilya Gusev who collected them!
  2. I developed a suitable embedding strategy. The tricky part was splitting the longer poems into properly sized chunks in the correct way. This was done in conjunction with model selection. EmbeddingGemma turned out to be nice.
  3. Tech stack requires a good vector database, so I chose Qdrant.
  4. Browser UI and functionality were done in Claude (or Codex? don't remember) in a simple way (vanilla JS), interactive networks were done in D3.js, which I recommend for similar cases after testing several other solutions.