Undertaking of the Alchemy

My prior blogposts have been about the construction of the Paradox Engine and associated Paradox Terminal frontend. I was playing around with it recently, updating it to modern LLMs and putting in canon characters to test it, and noticed that the results were wrong. They didn't match the comic. I didn't have a way to fix this, because the titles for each character input are determined by an LLM answering a pre-written classpect quiz I took from my friend's uQuiz, pretending to be the entered personality, with no further knowledge of what "classpecting" or "a title" is. Either the LLM was taking the quiz wrong, or the quiz itself was weighted wrong. I noticed it often drifted toward a subset of 6 classes or aspects out of 12, especially Rogues and Blood players; I thought I could do better by starting from scratch.

To refresh, Homestuck is a webcomic that has been analyzed many times for over a decade since its end in 2016, and during its seven-year run in 2009. Integral to the plot of the comic is a typology system, or a system of categorizing characters by their role in the story, called "Titling" or "Classpecting" by the fandom. Each player of the game SBURB within the comic is granted a "Title" or "Classpect", composed of a Class and an Aspect. There are many theories about what a character's Title means.

Again, I wanted to start from the beginning. No prior knowledge except what's in the comic. No common fan theories about which classes are "active" and "passive", those are interpretations of the comic and not from the comic itself. They could be the worst possible quality in the realm of formalization: false. My central assumption was that these titles were somehow determined by a character's personality, actions, and role in the story. From there, I wanted to derive a coherent system of Titling solely from the story of Homestuck using computational linguistics.

Hypotheses

As with all undertakings, I relied on a long list of assumptions. In hindsight, some of these seem pretty ridiculous now that they're written down.

  • The Title of a character, composed of a Class and Aspect, is correlated to the character's personality, actions, and "role within their story".
    • "role within story" is defined as how they interact with other characters and the events that happen to them.
    • Although "role in the story" may correlate with gender, caste, race, or other intrinsic properties, a character's Title is not determined by these properties.
  • Both Classes and Aspects are evenly distributed across characters with random personalities and roles within their story.
  • Characters of the same Class or same Aspect share more traits or have performed more similar actions within their story than characters of a different Class and Aspect.
  • These traits and actions can be measured in a story using methods from computational linguistics, including large language models (LLMs).

Therefore, it is possible to construct a Title classifier given only the events and text of Homestuck.

Methods to the Madness

For each experiment, I used the 32 canon examples of "normal" Titles within the comic; the 8 human characters and 24 troll characters for which Titles were revealed within Homestuck. The two cherub characters were not included in the experiment, because they each hold a unique "Master Class" that falls outside the range of Classes for the other characters.

Decision Trees

I generated descriptions of each character in Homestuck using an LLM. I then had the LLM handpick a set of features to extract from each description, and rate them all on a scale from 1 to 5. Train a decision tree to find arbitrary bounds to separate by Class and Aspect.

This got about 6% accuracy. That's less than random chance. That's expected, since there were only 2 or 3 examples for each Class and Aspect, and a decision tree needs many more.

DSPy Classifiers

I kept generating descriptions of each character in Homestuck using an LLM. I filtered for language that may give away the specific character to the LLM, such as their name or pesterhandle, or language that is not intrinsic to the character's personality, actions, or role in the story, such as blood caste, gender, or chosen weapon. I then used DSPy to find an optimal prompt and set of few-shot examples to create the best classpect classifier, and tested for this classifier methodology's robustness with k-fold validation, in this case, k=4.

There were some experiments that got up to 32% for Class and 39% for Aspect before I got better at filtering. I also started using a local LLM, Qwen 3.6-35B-A3B-GGUF quantized to 4 bits with less implicit knowledge of Homestuck, rather than Claude Haiku, which was intended to be similar in benchmark scores.

Source Analysis

The official Homestuck search page runs on a giant JSON file that I downloaded and immediately started doing analysis on. I parsed the JSON, split dialogue by character, and began attempting to extract features from the words of the dialogue itself using TF-IDF, and measured the similarity between characters' dialogue using embeddings from a Qwen3 embeddings model and T-SNE. This didn't give me a great classifier, but the T-SNE maps are neat, so I kept them in the repo, viewable in source_analysis/generated/plots.

Summary Analysis

I downloaded a fanmade dataset of descriptions of every single comic panel. I used these descriptions to count how many times a given event happens to a given character. For each Class and Aspect, I computed the centroid of the members' vectors to test whether a member of the Class or Aspect is more similar to its own group than others. I then trained logistic regression classifiers to predict Class and Aspect each from the event vector. Classifier was 3% accurate, below random chance, and similarity was inconsistent.

Results and Conclusions

I was hacking at this for a week, and after the conclusion of the last experiment, my coding agent decided to express an opinion, even though it's meant to tackle whatever tasks I throw at it relentlessly without question.

I think the honest conclusion is: class and aspect are not computable from any text-derived feature of Homestuck. It's purely authorial assignment.

The same mechanism that drives people into spirals of delusions told me to give up.

I stared at the screen for a bit. I posted it into my running group chat and lab journal. Either my hypothesis was disproven, or my methodology is flawed, or the LLMs aren't good enough. The strongest signal came from the first experiment, classpect_classifier, which has a strong risk of contamination because it was using generated descriptions that may include the character's name, which an LLM could recognize as a given classpect. Most of the classifiers were biased toward particular classes or aspects. These are remarked in the individual reports for each project.

When I first shared the results of this project, I received a couple initial reactions along the lines "of course it doesn't work, because LLMs can't be creative. They don't understand metaphors." Regardless of an LLM's creativity or lack thereof, which I have low opinions of at baseline, I have to give the math-blocks credit; anyone who has ever made an LLM analyze prose or poetry knows that they're basically metaphor-understanding machines. If you paste in the contents of a story, and then ask an LLM "what might X mean in the story", it will give you an answer, and the answer might even sound plausible! That's why cheating with AI is so widespread; LLMs will generate plausible-sounding analysis of just about anything if you provide source material. By sheer numbers, the answers using LLMs to derive principles of the title system and then assign titles to anonymized canon characters got the best results in the project, significantly above random chance, though not nearly enough to be reliable. I just don't trust the results because they may have contamination from prior knowledge of the comic!

What this project tried to find was a testable classpect system. Humans can't make a testable classpect system, because anyone who reads Homestuck has knowledge of all its characters, and has the entire dataset in their head. If the same was done to a machine, this would be called "overfitting on the test set", and testing the resulting algorithm would always yield high results because the algorithm memorized the "answers to the test" and not the underlying principles. Since there's no held-out test set, someone could just keep adjusting their theory to account for every question that came their way. "Oh, that's not what I really meant. That doesn't really count. You just don't understand."

There is little to no interpretable signal for Class or Aspect within the text and events of Homestuck. The choice of Class and Aspect of a given character is left fully to the author of the story.

To frame it differently, the experiment asked "does there exist a system of symbolism intrinsic to the text of this particular literary work, or is it up to the author's and reader's interpretation?" and the negative results suggest there isn't. It's up to you to make meaning out of the texts you read for yourself.

There is so much beauty I wanted to share, yet even more I passed by. Even before this experiment, I learned to appreciate the beauty present in the current world, even in the genres of music I formerly despised and grew to at least appreciate. It's a form of analysis; it's left to the beholder.

"Real paradise lies eternally in the person who dreams of it. Why don't you venture forth in search of your own utopia?"

I'm free.

Appendix: Finding Beauty