How to generate text with a Markov chain
- Paste some training text: a story, song lyrics, a collection of tweets, product names or a public-domain book chapter. More text gives more varied results.
- Choose word-level or character-level and the order, which is how many previous words or letters the model looks at.
- Press Generate as many times as you like. Open "Peek inside the model" to see the probabilities it learned.
The simplest language model
A Markov chain predicts the next word using only the last few words. Training just means counting: after "the little", the sample text continues with "rabbit" every time, so the model learns that "rabbit" follows with 100% probability; after "the", several words are possible, each with its own share. Generating text is a walk through these counts, picking each next word at random in proportion to how often it followed the current context.
The order controls the trade-off. With order 1 the output is varied but rambling; with order 3 or more it reads fluently but mostly copies long runs of the source, because each context has been seen only once. Character-level models invent new words that sound like the training text, which is a fun way to make fantasy names or brand ideas.
Modern large language models solve the same task, predicting the next token, but use neural networks that consider thousands of previous tokens and generalise to sentences they have never seen. Comparing the two shows why context length and generalisation matter. Everything runs in your browser.
Why does the output copy my text?
The order is too high for the amount of text. Lower it, or add more training text.
Is this how ChatGPT works?
The goal is the same, predicting the next token, but ChatGPT uses a neural network with a very long context instead of simple counts.
What can I use it for?
Teaching probability and language models, parody text, name generators and creative writing prompts.