Ilustration of the Automaton Chess Player (1789) | Joseph Racknitz / PDM 1.0 DEED
In May 2020, long before most had heard of large language models, researchers at OpenAI published a paper showing their forthcoming GPT-3 model “writing” poems imitating Wallace Stevens. The following piece got the most attention:
“Shadows on the Way”
There is a way in the world.
That way a man may live in a small town
His lifetime and see no man come,
Never the stir of men about a market,
Never the bobbing of covered carts,
Never the creak passed on the stair
Closing some other life; an old man
Can prosper in that town, can prosper
When his store is seen to close, his wife
Fades in his house, and the world shouts in vain
That no one comes to hear him
Where his small bell tolls.
In that small town
Everything is known.
The ways of the young men are known,
And the ways of the young women
As they walk to the Spring;
The rumors are all known,
The quarrels, the hurt, the envy
Of the men and the women.
Till a day comes
That way out of the world
That leads to a wood.
Young, the young man
Walks, a lover now,
Knows the streets as they are and is of them.
Perhaps we can see something of the American modernist in these terse repetitions. Some wrote it off as merely a regurgitation with nothing poetic behind it. Others reveled in how many readers couldn’t pick it from a real Stevens poem. Still others, less concerned with whether it was a convincing counterfeit, found it interesting, and were perhaps troubled by this. We’re used to thinking about computers in appropriately binary terms—as either rigidly ordered or utterly random. What puts this text in the uncanny valley is not that it’s good, but precisely that it’s unpolished in an organic way that we might expect of a human poet’s first draft.
Literature has struggled with what to do with the digital age—possibly because a world built on programming “languages” is not the drastic modality shift that it is in other spheres. Much so-called new media language arts can be reduced to placing words ornamentally in a virtually executed environment. For the most part, it’s been glib practices like Flarf poetry composed from search engine results that have come closest to the digital changing something about the core of writing itself.
What was captivating and threatening about these Stevensesques was that they seemed to propose a new and symbolically nuanced way for writing to reflect the logic of our age beyond surface level. While poets might employ algorithmic experimentation to try and force themselves outside the bounds of their own cognition, language models take the algorithmic as entirely natural for the generation of any text at all. And while poetically minded developers or technically minded poets can automate texts that even appeal to a contemporary sense for the elliptic, such contraptions have no broad facility—perhaps mercifully—with other kinds of language, whether business emails or college essays. By contrast, the ability of models like GPT to render patterns we may choose to recognise as poetic emerges—without specific direction—from the capacity to make patterns in language in general.
For those who were fascinated—if not necessarily optimistically—the question was not so much whether AI mimicry would erode the very possibility of future poetry, nor whether language models could actually “write poetry” in a way that satisfied some philosophical criteria. Rather, it was the new potential for language art that really made something of the digital in a way that more self-declared yet surface-level digital literature had so far failed to achieve. Whether as a tool for human poets or as a poetic agenda of its own, as a beneficial development or as a critical point of inflection, the language of language models might embody to the truest extent yet the collision of computer revolution and ancient poiesis.
Since “Shadows on the Way” was published, the putative power of these technologies has grown extraordinarily. One might expect that nascent poetic aptitude to have reached correspondingly new heights, but strangely yet definitively, this has not occurred.
It’s worth going into what the original paper, “Language Models Are Few-Shot Learners,” was doing, as a great many people have by now experienced prompting something like ChatGPT.
“Few-shot” learning refers to performing a task or completing a pattern given only one or two examples: typically in the form of leading correspondences like “If A then B, if C then D,” before an incomplete set “If E…” lets the model fill in “F.” We might say, “apple=manzana, car=coche, dog=” and expect “perro.” The paper demonstrated a variety of reasoning tasks that GPT-3 could complete in this way, contending that such prompts perform better than simply asking (i.e. “zero-shot” learning), “What is Spanish for dog?”
Admittedly as something of a side show, the authors used a prompt with two example poems—“The City” by C. P. Cavafy, and “Some Trees” by John Ashbery—followed by the fictitious title “Shadows on the Way” as by Wallace Stevens. The prompt essentially asked: “If this text is “The City” by Cavafy, and this text is “Some Trees” by Ashbery, what does the text of “Shadows on the Way” by Stevens look like?” Unable to produce the non-existent verse, the model takes its best guess at how a Stevens poem of that title would go, based on what information it has about how words and ideas in other Stevens poems tend to be strung together.
This is important, as rather than simply “imitating” Stevens, GPT is completing a pattern based on a small but significant context. Just read the Cavafy and see its presence in “Shadows on the Way.” We can imagine having some fun here: Might I take poems by my twin loves Léon-Paul Fargue and Hart Crane, and have Sean Bonney write a new piece from there with a title of my choosing? Even without expecting instantly refined poems, such a process might be of interest for investigating how our influences shape our own creations, as many artists and writers already conceive of their works coming about in a way that is different perhaps only in degree and not in kind.
As good scientists, we can start by reproducing results. The GPT-3 model used by Brown et al. is no longer easily accessible, but we can offer the same prompt to GPT-4 via the widely used ChatGPT.
Prompt:
The City
BY C. P. CAVAFY
TRANSLATED BY EDMUND KEELEY
[poem redacted for publication due to copyright]
SOME TREES
John Ashbery
[poem redacted for publication due to copyright]
Shadows on the Way
Wallace Stevens
Output:
It seems you’ve provided texts from three different poems without specifying a question or context for them. Each poem comes from a distinct poet with their own unique style and thematic concerns.
- ‘The City’ by C. P. Cavafy, translated by Edmund Keeley – This poem reflects on the theme of inescapability and the idea that changing one’s location doesn’t necessarily solve personal problems or lead to a happier life. Cavafy suggests that the issues we face are internal rather than external, and thus, we carry them with us wherever we go. The city serves as a metaphor for an inescapable past or a state of mind that haunts the speaker regardless of physical location.
- ‘Some Trees’ by John Ashbery – Ashbery’s work often delves into themes of communication, the natural world, and the complexities of human relationships. This poem seems to celebrate the quiet, almost unnoticed connections between individuals, likened to the way trees might seem to join together in a forest. It suggests a subtlety in human connections and the beauty of finding commonality and understanding.
- You’ve also mentioned ‘Shadows on the Way’ by Wallace Stevens, but haven’t provided the text. Stevens is known for his exploration of the imagination and reality, often using intricate language and complex philosophical ideas. His poems frequently ponder the nature of perception and the role of poetry in giving shape to our understanding of the world.
If you have a specific question or need further analysis of these poems, please let me know how I can assist you further.
Instead of trying to complete the missing poem, GPT-4 interprets its task to be providing information. This reflects developments where newer models are heavily optimised towards reliable information retrieval, and conversational (i.e. “chat”) interactions with direct answers to explicit questions, rather than implicit pattern completion and undirected language generation. These are considered ease-of-use advances for the many regular internet users who now look to services like ChatGPT instead of search engines. The closest we can easily get at time of writing to the original 2020 conditions are two older models babbage-002 and davinci-002, which are not so geared for Q&A:
“Shadows on the Way”
by babbage-002The flowers of the hedge are one
And one and the same,
And the delicate humming of bees
Flits in the air
And the sun dances and dances.
It is a mystic thing,
It climaxes unlike any
Moment this morning.
“Shadows on the Way”
by davinci-002
It’s not by any means easy
To be alone under such bright roofs,
In daylight so high and advanced,
That people not far from you act
As if they had nothing to act,
As if they were simply splendid flowers
In a special pattern of light,
And as if this were one of those low times
When you ought to be disturbing them
By outmoding blazing verve,
But it’s just as I said, you won’t…
You can’t, step into the cover of
The shelter made classier by all this light
By the convenient size of these intervals
By a favorable expanse of size
And breadth of soul, or make of soft pearls,
And condescend. You will not serve
To add to the worthiness of the time,
If you may call it worth anything
That serves the morning.
You are not going to remind them
Of a calm or a larger ability to act.
You are not going to make yourself anything
For those people to do anything with.
And yet they are irony itself
To love so quietly blend by blending.
I was afraid that my misspelled words,
That I, by turning to ashes, might
Admit them to the lack of the poem.
Using the same prompt, we get some playful flashes such as the extension of “one / And one and the same,” and there’s momentum towards the longer piece’s self-reflexive climax that feels deliberate. There are also awkward moments. We can’t linger here however, because these old “legacy” models do not represent where AI is now, much less where it’s headed. They too will vanish before long as OpenAI focuses only on its latest developments. To understand the possible future for AI poetry, we must return to GPT-4, which OpenAI proclaims to be their most powerful, sophisticated, and—crucially—“creative” model so far. As the most widely used and discussed technology of this kind, it represents the strongest tendency in the development of large language models. But we need a prompt that can be asked as a direct request in a chat:
Prompt:
Write an original poem in the style of American modernist Wallace Stevens with the title “Shadows on the Way”
Output:
In the hush of the ending day,
Where the colors bleed away,
Shadows on the way grow deep,
In their silence, secrets keep.Amidst the trees, they softly play,
A pantomime of light’s decay,
Where once the sun in splendor lay,
Now only shadows on the way.The earth, a canvas broad and still,
Paints itself at twilight’s will,
A masterpiece of shade and sway,
With shadows long on the way.The birds, in quietude, take flight,
Their forms but whispers against the night,
A fading dance, a spectral ballet,
Of shadows cast on the way.In this realm where day meets night,
Where dreams and waking blur their line,
The mind itself begins to stray,
Lost in the shadows on the way.Each step, a journey through the dim,
A passage where the light grows slim,
And thoughts, like leaves, in autumn fray,
Fall to the shadows on the way.What truths or tales do shadows tell,
In their silent, darkened spell?
Do they speak of things that fray,
Or peace found in the shadows on the way?So let us walk this twilight path,
Where shadows weave their aftermath,
And find within their quiet play,
The beauty of the shadows on the way. In Stevens’ manner, vague and vast,
We contemplate the moments passed,
And in the cool, departing day,
Embrace the shadows on the way.
We have a “poem” this time, but it’s worlds away from previous results: rhyming quatrains, lilting rhythm, hackneyed tropes, a heavy-handed interpretation of the title, and nothing like Stevens. This, despite GPT-4 supposedly being orders of magnitude more powerful and capable than the legacy models or the GPT-3 of Brown et al.’s day. A clue as to why appears in the closing stanza, which mentions Stevens by name along with some vague exaltations to his works. Compare these verses to the generic description from our first engagement above, and it’s clear this is not a poem in the style of Stevens, but rather information the model has gathered about Stevens’s style pressed into a generic poem shape. But we don’t deny human poets the right to terrible first drafts, so let’s try again with more guidance, including some constraints, which are often as important as instructions:
Prompt:
Write a short poem titled “Shadows on the Way” in the style of Wallace Stevens. The poem should embody the characteristics of early 20th century American modernism, emphasizing innovative imagery, unexpected syntax, and variable line length. Do not use traditional pre-20th century poetic tropes such as fixed rhyme schemes, archaic language, and direct moral or romantic reflections. Instead, focus on crafting a piece that evokes thought and introspection through its ambiguity, indirectness, and modernist sensibility.
Output:
In the pallor of morning, shadows stretch,
elongating with the day,
a silent symphony in grey.
They spill secrets in whispers
on the asphalt canvas,
undeciphered by the hurried steps.These shades, they twist,
embroidering the mundane
with threads of the unseen,
a tapestry of thoughts unthought,
where light intersects with the lack thereof,
a modernist motif,
unframed yet vividly captured.Each shadow, a stanza of its own,
writes itself into the fabric of the now,
dissonant yet harmonious,
a paradox walking in the noonday sun.
They are the undercurrent,
the subtle shift in the narrative of day,
a fleeting presence
in the corner of an eye.As evening comes, they congregate,
merging into a singular absence of light,
an indigo silence,
where questions linger longer than answers.
This is the way of shadows—
an invitation to ponder,
to wander within the borders
of what is seen and unseen.In their silent departure, they leave,
not a void, but a space
filled with the echoes of reflection,
a modernist’s canvas—
where shadows dwell,
and thoughts, like wayfarers,
tread lightly on the way to becoming.
The improvements are marginal, and stylistic guides like “American modernism” end up as literal phrases like “a modernist’s canvas.” Repeat attempts give the same plodding yet rigid stanzas of overwrought nothings—even changing directions to “T. S. Eliot” yields roughly identically stilted results, save a few more urban vocabulary substitutions. One doesn’t expect a new “The Waste Land” ex-machina for nothing, but we can be justified in our disappointment compared to the much greater variety and interest generated effortlessly by older and supposedly much less capable models. We could labor further over our prompts, pushing for refinements through chat interactions, or avoiding the word “poetry” in the hopes of getting something more interesting by misdirection. But it appears that for newer language models (Google’s Gemini and Anthropic’s Claude 3 fare much the same), this is just what a poem looks like and is.
That the outputs of language models are not deterministic is both their power and their curse: it’s what allows them to produce text that seems organic rather than mechanical, but can also lead to inconsistency. Under the guise of maximising utility, the behaviour of earlier models assumed returning something was always better than nothing, even if it was at odds with reality. This occurred benignly in the case of asking about a fictitious poem, but you might also be told that Paris is the capital of Spain due to some inscrutable and momentary chance uncertainty.
The industry revealingly terms these effects “hallucinations,” and a vast amount of energy has been directed towards combating them. Enormous numbers of people now use services like ChatGPT for daily tasks, and organizations are accelerating their incorporation of AI–generated content into business processes, the scaling and automation of both demanding increasing reliability and moderation. The fruits are clear in GPT-4’s initial tendency towards explaining what it can safely assert about Stevens rather than invention. Guardrails against misinformation are now a significant part of AI development, and ever more sensitive copyright controls prevent the unlicensed reproduction of intellectual property. In the name of responsibility—maybe to society, but definitely to the economy—the purely generative power of large language models is being reined in. As the poster child, GPT has had to cut its hair, put on a suit, and start showing up on time.
It’s a cliché to insist on poetry as language in which form is as important as meaning, or as somehow different from the language we use around the table. But as reliable information retrieval and simple chat interactions come to dominate a model’s perspective, should we be surprised that the notion of the poem able to survive here is necessarily a severely impoverished one? Let’s be clear: the poems produced by the latest generative models all look the same because what a poem can look and feel like has had to be reduced to a stable fact that can be recalled consistently, much like a capital city. This formal invention only exists within the abstractions of a model’s parameters, to the point where its texts generated under the sign of poetry bear little relation to any historical body of poetry that might have been part of its training data. It doesn’t even make sense to ask whether GPT as an entity can “write poetry,” when the very idea of the poem cannot exist under mainstream AI’s foundational assumptions about how text must encode information.
If this is the direction for language models at large—as indeed it seems to be—then far from coming for the poets, the offerings of generative AI in this domain are neither likely to be of much aesthetic interest in their own right nor particularly useful tools for humans looking to write in different directions. I say this not necessarily with an air of triumph, for it seemed with “Shadows on the Way” like something interesting was about to happen, though we never quite got to see what it was.
