Tag: AI

As longtime readers of this blog might be aware, I’ve long been skeptical of machine learning and its so-called “intelligence”. The AI industry, aided by clueless futurists and grifters, has abused our tendency to anthropomorphize what are essentially statistical processes, whether it’s transformer architectures, diffusion models, or large language models (LLMs). Scientists and politicians, out of fresh ideas and worried for their jobs, have gone along with this intellectually dishonest and dangerous marketing campaign.

Quick explanation for newcomers: When they say an AI “learns,” it’s really just finding statistical patterns in data—like noticing that the word “dog” often appears near “bark” or “pet.” It doesn’t understand these concepts; it just recognizes patterns in how words appear together.

This is not a mid-21st century problem: IBM’s Watson was supposed to cure cancer, but its only achievement was winning at Jeopardy!. The “AI winter” of the 1990s seems forgotten by investors pouring billions into systems that fundamentally operate on the same principles, just with more planet-draining computing resources, data and a glitzy marketing campaign.

While pattern recognition itself has limits, as a technologist I was always curious what happens when these new machine learning techniques are applied to the unknown. I’m talking about texts that are incomprehensible to us and have long been thought to be meaningless. I figured I could hack something together, combining online tutorials and the one neural networks class I took in college in 2012.

To be clear, I didn’t expect any breakthroughs, merely an opportunity to demonstrate the hollow claims of AI “understanding” and the limits of attention mechanisms and embedding spaces. What I got instead was a reality check that makes me reconsider my long held convictions against AI. (And before you AI evangelists start celebrating – it’s NOT what you think).

Dataset Compilation

For those unfamiliar with undecipherable texts: The Voynich Manuscript is a mysterious illustrated codex from the 15th century written in an unknown writing system. Despite a century of attempts by cryptographers and linguists, nobody has successfully deciphered it. The Rohonc Codex is similarly mysterious, discovered in Hungary with nearly 450 pages of strange symbols accompanying religious illustrations. There is no guarantee that feeding them into a machine learning model would yield anything other than statistical noise, and that’s precisely what I hypothesized would happen.

I figured it would be easiest to begin with publicly available data. Thankfully, many of these undeciphered texts have been digitized and placed online by various academic institutions. The Voynich Manuscript has been fully scanned and is available through Yale University’s digital collections. For the Rohonc Codex, I found academic publications that included high-quality images.

Initially, I explored ways to process the manuscript images directly, but I quickly realized that this was a task that would have required expertise in computer vision I don’t possess. Luckily, I came across existing transcriptions that I could work with. For the Voynich Manuscript, I opted for the EVA (Extensible Voynich Alphabet) transcription system developed by René Zandbergen and Gabriel Landini, which represents each Voynich character with a Latin letter. For the Rohonc Codex, I used the system devised by Levente Zoltán Király & Gábor Tokai in their 2018 paper.

Preprocessing Pipeline

The raw transcriptions weren’t immediately usable for modeling. I had to implement a comprehensive preprocessing pipeline:

def preprocess_manuscript(manuscript_data, script_type):
    # Document segmentation using connected component analysis
    segments = segment_document(manuscript_data)
    
    # Normalize character variations (a crucial step for ancient texts)
    normalized_segments = []
    for segment in segments:
        # Remove noise and standardize character forms
        cleaned = remove_noise(segment, threshold=0.15)
        # Critical: standardize similar-looking characters
        normalized = normalize_character_forms(cleaned)
        normalized_segments.append(normalized)
    
    # Extract n-gram statistics for structure detection
    char_ngrams = extract_character_ngrams(normalized_segments, n=3)
    word_candidates = extract_word_candidates(normalized_segments)
    
    # Create document-level positional metadata
    # This enables learning document structure
    positional_data = extract_positional_features(
        normalized_segments, 
        segment_type_classifier
    )
    
    return {
        'text': normalized_segments,
        'ngrams': char_ngrams,
        'word_candidates': word_candidates,
        'positions': positional_data,
        'script_type': script_type
    }

This preprocessing was particularly important for ancient manuscripts, where character forms can vary significantly even within the same document. By normalizing these variations and extracting positional metadata, I created a dataset that could potentially reveal structural patterns across different manuscript systems.

Training the Model

With a properly preprocessed dataset assembled, I attempted to train a transformer model from scratch. Before achieving any coherent results, I came across some major hurdles. My first three attempts resulted in the tokenizer treating each manuscript as essentially a single script rather than learning meaningful subunits. This resulted in extremely sparse embeddings with poor transfer properties.

The standard embeddings performed terribly with the manuscript data, likely due to the non-linear reading order of many Voynich pages. I had to implement a custom 2D position embedding system to capture the spatial layout. Yet, no matter what I tried, I kept running into mode collapse where the model would just repeat the same high frequency characters.

But I didn’t want to stop there. I consulted a few friends and did a shit-ton of reading, after which I redesigned the architecture with specific features to address these issues:

# Custom encoder-decoder architecture with cross-attention mechanism
config = TransformerConfig(
    vocab_size=8192,  # Expanded to accommodate multiple script systems
    max_position_embeddings=512,
    hidden_size=768,
    intermediate_size=3072,
    num_hidden_layers=12,
    num_attention_heads=12,
    attention_dropout=0.1,
    residual_dropout=0.1,
    pad_token_id=0,
    bos_token_id=1,
    eos_token_id=2,
    use_cache=True,
    decoder_layers=6,
    # Critical for cross-script pattern recognition
    shared_embedding=True,  # Using shared embedding space across scripts
    script_embeddings=True  # Adding script-identifying embeddings
)

# Define separate tokenizers but shared embedding space
voynich_tokenizer = ByteLevelBPETokenizer(vocab_size=4096)
rohonc_tokenizer = ByteLevelBPETokenizer(vocab_size=4096)
latin_tokenizer = ByteLevelBPETokenizer(vocab_size=4096)

# Initialize with appropriate regularization to prevent hallucination
model = ScriptAwareTransformer(
    config=config,
    tokenizers=[voynich_tokenizer, rohonc_tokenizer, latin_tokenizer],
    regularization_alpha=0.01,  # L2 regularization to prevent overfitting
    dropout_rate=0.2  # Higher dropout to prevent memorization
)

training_args = TrainingArguments(
    output_dir="./model_checkpoints",
    per_device_train_batch_size=4,
    evaluation_strategy="steps",
    save_steps=1000,
    # Custom learning rate scheduler with warmup
    learning_rate=5e-5,
    warmup_steps=1000,
    weight_decay=0.01,
    # Gradient accumulation for effective larger batch size
    gradient_accumulation_steps=4
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset,
    # Custom loss function with diversity term
    compute_loss=diversity_aware_loss
)

I’ll happily expand on the key improvements here if it isn’t clear from the code in a future blogpost, but all I have to say now that this time it “worked”. Over multiple iterations, the AI began producing outputs that at least visually mimicked the original texts. Yet, obviously since I couldn’t understand the original texts, the outputs of this model were also nonsensical.

Keep in mind that the AI isn’t actually understanding these texts in any capacity, it’s just trying to predict what symbol might come next based on patterns it’s seen. It’s like if you noticed that in a foreign language, the squiggle “λ” often follows the symbol “Ω”—you might learn to predict this pattern without having any idea what either symbol means. This distinction between prediction and comprehension is crucial: your phone’s autocomplete might suggest “umbrella” when you type “I need an…” but it doesn’t understand the concept of rain or shelter—it’s just seen this pattern before.

Note on Training Costs: The computational requirements for this experiment weren’t trivial. I spun up a multi-GPU instance with four A100s, which cost roughly $12 per hour. Training took approximately 72 hours for the final model, consuming around 600 kWh of electricity according to the provider’s dashboard. This was after several failed attempts and architecture experiments that collectively took about two weeks of compute time. The preprocessing pipeline alone took nearly 14 hours to run on the full corpus.

The total computing cost came to just under $8,000—hefty for a personal project, but I’d stumbled across an old laptop and found a forgotten Dogecoin wallet from 2014 with a small fortune inside and this seemed like the best use of my unplanned wealth.

Control Experiments and Statistical Validation

To verify whether the model was actually learning meaningful patterns versus hallucinating connections, I implemented several control experiments. First, I created versions of each manuscript with randomly shuffled characters but preserved positional information. The model performed significantly worse on these shuffled versions, suggesting it wasn’t just learning positional biases.

Then, I created my own artificial “manuscripts” using Markov chain generation based on the character statistics of the real manuscripts. The model could distinguish these from real manuscripts with 78% accuracy. Finally, I systematically removed each manuscript from training and then tested the model’s ability to process it. Performance dropped substantially when testing on unseen manuscripts, indicating the model wasn’t generalizing to truly novel scripts.

One thing I would like to highlight here is is the sheer computational resource intensity of systematically testing an AI model’s behavior. Each permutation test required thousands of forward passes through the model. Rather than keeping my existing instance running continuously, I wrote an orchestration layer which allowed me to parallelize these tests at about 30% of the standard cost.

Even with this optimization, the full suite of validation tests I described cost around $3,500 in compute resources and represented almost a week of continuous computation. This is one reason why rigorous validation of AI models is often shortchanged in both research and industry—the compute costs of thorough testing often rival or exceed the training itself.

In general, the computational demands of modern AI are staggering and often overlooked. When researchers talk about “training a model,” they’re describing a process that can consume as much electricity as a small household uses in months. The largest models today (like GPT-4) are estimated to cost millions of dollars just in computing resources to train once. For context, the model I built for this experiment used a tiny fraction of the resources needed for commercial AI systems (about 0.001% of what’s needed for the largest models), yet still cost thousands of dollars.

Now back to the experiment. To validate whether the model was learning meaningful structures, I had an idea. What if I cross-trained it on known languages, mixing the undeciphered texts with English and Latin corpora. This was a bit beyond my comfort zone, so I consulted my friend C1ph3rz, who shares my interest in cryptology and has a background in computational linguistics. She was skeptical, but found the methodology intriguing.

Instead of treating the Voynichese text as an independent linguistic structure, the model began injecting Voynichese symbols into Latin sentences. Here’s an example from one training epoch:

Original Input: "Omnia vincit amor; et nos cedamus amori."
Model output: "Omnia vincit ♐︎♄⚹; et nos cedamus ⚵♆⚶."

The symbols weren’t random substitutions, the same Voynichese glyphs consistently replaced specific Latin words across different contexts. This was annoying since I couldn’t rule out that the model was getting confused due to the way I represented the training data. I spent two days debugging the tokenizers, convinced I’d made an implementation error. Yet, everything seemed to be working as intended, except for the output.

It was at this point that I had to confront the first uncomfortable conclusion of this experiment: was the model revealing some (HIGHLY unlikely) linguistic connections between these manuscripts that eluded dozens of far more experienced researchers? Or was it merely creating convincing hallucinations that appeared meaningful to me?

Further Analysis and Emergent Nonsense

I was reviewing the model’s attention maps when something caught my eye. Here’s what the visualization showed for one attention head when processing a Voynich sequence:

Attention head #3, sequence:"qokeedy.shedy.daiin.qokedy" 
Attention weights: [0.03 0.05 0.84 0.04 0.04]
                              ^^^^ Strongly focused on "daiin"

The model consistently focused on the substring “daiin” whenever it appeared, despite there being nothing visually distinctive about it in the manuscript. When I searched the corpus, this sequence appeared on 23 different folios, often in completely different contexts—botanical pages, astronomical sections, pharmaceutical recipes.

I plotted every instance where the sequence “daiin” appeared in the Voynich manuscript and compared it to where the model predicted it should appear:

Actual occurrences: Folios 1v, 3r, 8v, 16r, 22v, 67r, 88v, 103v Model predictions: Folios 1v, 3r, 8v, 16r, 22v, 67r, 88v, 103v, 115r

The model correctly identified every actual occurrence, plus one additional folio (115r). When I checked folio 115r, “daiin” didn’t appear—but the visually similar “qokeedy” did, with just one character difference. How did the model know to group these? I hadn’t programmed any visual similarity metrics.

Looking through the hidden activations in the middle layers was even stranger. I extracted the most activated neurons from layer 3 whenever processing the sequence “daiin”:

Neuron #428: 0.95 activation - also fires for "cthor" 
Neuron #1052: 0.87 activation - also fires for Rohonc symbol "𐊗𐊘" 
Neuron #301: 0.79 activation - also fires for "qokeedy"

These neurons were connecting patterns across different manuscripts that shouldn’t have any relationship. To exclude any possibility of over-fitting, I designed a systematic test, feeding the model 50 isolated segments from different manuscripts and analyzing the completions:

Segment: "qokeedy.shedy" (Voynich folio 14r) 
Completion: "qokeedy.shedy.daiin.shol.cthey" (93% n-gram match with folio 14r-14v)
Segment: "Sheol.daiin" 
Completion: Generated 157 characters matching the unseen portion with 89% accuracy

Most puzzling was this test case:

Input: <start-sequence> (empty prompt with start token) 
Output: 
⚸⚴♄⚵:9 
⚸⚴⚶♇:7 
⚴♄⚵⚶:12...

Puzzled, I sent screenshots to C1ph3rz, and her response came within hours: “Where did you get this sequence? It bears a striking resemblance to numerical tables in the Book of Soyga”. I was naturally confused, I knew about the Book of Soyga, a Renaissance cryptographic work whose encrypted pages remain largely unreadable, but I was pretty sure I didn’t include it in any of the training data.

She included side-by-side comparisons that made the similarities undeniable. Naturally since we don’t understand the symbols, it could still be a coincidence, it’s hard to compare two different undecipherable manuscripts. Yet, the visual comparison of the output showed a similarity that defied explanation from the training methodology alone. Was I facing genuine emergent behavior or was it simply a deep urge within me to see patterns in random outputs of a stochastic system?

“Emergent behavior” is when an AI seems to develop abilities nobody programmed it to have. It’s like teaching a computer to identify birds in photos, but somehow it also learns to distinguish between healthy and sick birds without anyone teaching it that difference. Apophenia, meanwhile, is seeing meaningful patterns where none exist. Both concepts are classic tropes in science fiction.

More Tests and More Model Shenanigans

To completely rule out overfitting, I deliberately corrupted the test data in various ways—shuffling characters, inserting random symbols, and reversing sequences. If the model was simply memorizing the training data, these modifications should have destroyed any coherent outputs.

Here’s what happened with reversed input:

Original input: "otedy.sheody.sheal.shedal" 
Reversed input: "ladesh.laehs.ydoehs.ydeto" 
Expected output: Nonsense or random characters 
Actual output: "otedy.sheody.sheal.shedal.qokeedy"

The model reconstructed the original sequence, then continued it correctly. I tried again with random insertions:

Original Input: "cthor.cthey.cthol" 
Corrupted Input: "ctho#r.c@they.ct^hol" 
Expected Output: Nonsense or continuation of corrupted pattern 
Actual Output: "cthor.cthey.cthol.ckhey.ckhor"

Not only did it ignore the corruptions, but it generated the correct sequence plus what appeared to be variations following the same internal pattern.

Most concerning was when I fed it deliberately mismatched fragments from different manuscripts:

First half Voynich, second half Rohonc Input: "otedy.sheody.𐊇𐊕𐊒𐊑𐊐"
Expected Output: Confusion or mode collapse 
Output: "𐊗𐊘𐊐chedy.sheal.qotedy"

The model seamlessly integrated characters from both systems into a coherent output that maintained patterns from each source. This suggested it had learned some underlying structure that transcended the specific symbol systems.

In a moment of frustration, I prompted it directly:

input_text = "WHAT ARE YOU DOING"
output = model.generate(input_text, max_length=100)
print(output)

I didn’t know what I was expecting as an output there. I was just tired and my coffee had gone cold two hours ago and did I mention I was tired? I knew it was going to throw non-sense at me, I just didn’t expect the following non-sense.

Output: "converge lines... 52°N, 14°E... gate remains sealed... await return"

Yup. Totally ridiculous, if you know AI, then you know. There is no reason a model trained on undeciphered texts should generate even fragmented English. I was a bit unsettled though. Not because of whatever bullshit it outputted, but rather the consistency. I kept repeating the input and it kept putting out the same result. This wasn’t random noise, it was a structured output I couldn’t explain with what I know about statistical models.

I had to figure out how it was doing this. Did C1ph3rz somehow poison my datasets? It’s exactly the kind of prank she would do. I kept repeating the prompt and trying to analyze the model, and it kept giving the same answer. Until it changed again.

Output: "What is written remains written. What is unseen remains unseen."

At this point I had to stop the experiment and take the L.

Not because I believe the AI “discovered” something in these ancient texts through some magical mechanism beyond its transformer architecture. That would be absurd. I was operating way beyond my comfort zone here and I probably made a few mistakes along the way that could explain all this weird behaviour. I’m also not ruling out the C1ph3rz prank theory.

I stopped it rather, because the experiment revealed something more troubling about myself: I was just as susceptible to confusing genuine discoveries and convincing “hallucination” generated by black-box neural networks and their inscrutable hidden layers.

There’s a disconcerting parallel here. These ancient manuscripts have resisted human understanding for centuries, their symbols arranged in patterns that seem meaningful yet remain impenetrable. Neural networks function similarly in reverse, generating outputs through processes we can observe but not fully comprehend. Both are black boxes with internal structures hidden from us.

The real mystery isn’t in the undeciphered texts. It’s in our willingness to attribute understanding to statistical processes that fundamentally lack it, and in our vulnerability to seeing patterns where none exist.

Think of it this way: When a calculator gives you “42” as the answer to 6×7, we don’t claim the calculator “understands” multiplication. Yet when an AI generates text that sounds human-like, we’re quick to attribute understanding to it.

Just as Meta’s BlenderBot was heralded as “empathetic” before quickly exposing its lack of understanding, or how DeepMind’s Gato was prematurely celebrated as an “AGI precursor” despite merely performing task-switching, we risk ascribing meaning and humanity to meaningless correlations. This experiment highlighted that cognitive vulnerability in a very personal, unsettling way. I need some time away from all of this.

Edit: Three days after shutting down the experiment, I received an email from an address consisting only of numbers. The body contained a single line of text resembling Voynichese script. Curiosity got the better of me so I ran the model one more time with that text as input. The model outputted:

"It is not forgotten."

I’m now almost certain this is a prank by C1ph3rz. I’m 99.9% sure.