AI-Generated Fiction Is More Than Fluent Text: How Can We Tell If an AI Story Actually Works?
Artificial intelligence has changed creative writing.
With generative AI, producing a paragraph, short story, or even a lengthy fictional narrative can take only a few prompts. The resulting text can be grammatically correct, descriptive, imaginative, and stylistically convincing.
But there is a major difference between writing sentences and building a coherent story.
A successful long-form narrative needs characters who remain recognizable, events that connect with one another, symbols that have meaning, and decisions that make sense within the fictional world.
This raises an important question:
How can we evaluate AI-generated fiction systematically?
A recent study published in the Open Access Journal of Data Science & Artificial Intelligence explores this question through an Arabic fiction case study, The Temple. The research proposes a reproducible matrix for evaluating narrative coherence, character continuity, and ethical narrative development.
What Is AI-Generated Fiction?
AI-generated fiction is creative writing produced with the assistance of generative artificial intelligence systems.
These systems can generate:
- characters,
- dialogue,
- descriptions,
- plots,
- scenes,
- fictional documents,
- narrative settings,
- and extended storylines.
However, generating individual components is not the same as maintaining them throughout an entire novel.
For a long-form AI-generated story, consistency becomes increasingly important.
A character introduced at the beginning should not suddenly become a completely different person without narrative justification. A mysterious object introduced early should have some relationship to later developments. A major decision should ideally have understandable causes and consequences.
These are examples of narrative coherence.
What Is Narrative Coherence?
Narrative coherence is essentially about whether the parts of a story work together.
Imagine a story introducing a mysterious stone in its opening chapters. If the stone later becomes central to an investigation, the earlier reference gains additional significance.
The study of The Temple tracks recurring anchors including:
110 Hz → The Stone → The Temple → Geneva → Investigation
These recurring elements help connect different sections of the fictional world.
Researchers can therefore ask whether an element is:
- introduced,
- repeated,
- reactivated,
- connected to later events,
- and involved in the final narrative resolution.
This is more informative than simply asking whether each individual paragraph sounds good.
How Do Researchers Measure a Story?
The study combines qualitative and quantitative approaches.
Qualitative analysis asks:
What is happening in the story?
Researchers examine:
- symbols,
- character development,
- point of view,
- narrative transitions,
- documents,
- system logs,
- pivotal events,
- decisions,
- and ethical consequences.
Quantitative analysis asks:
What measurable patterns can we identify?
Researchers examine:
- entity mentions,
- page coverage,
- consecutive presence,
- anchor frequency,
- and textual similarity.
The methodology keeps these two forms of analysis connected without treating numerical measurements as a replacement for literary interpretation.
Character Continuity: Does the Character Stay the Same?
One of the biggest challenges for AI-generated storytelling is maintaining character continuity.
A character's name may appear throughout a novel, but that alone does not demonstrate meaningful development.
Researchers therefore need to consider questions such as:
Does the character maintain recognizable motivations?
Do relationships evolve logically?
Are important turning points connected to previous events?
Does the character's value system change gradually or suddenly?
The study combines entity tracking with qualitative interpretation to address these questions.
Its quantitative results show measurable continuity across major characters and narrative entities. Maha, for example, appears across 92 pages with a longest consecutive run of 24 pages, while Ibrahim appears across 102 pages with a longest run of 13 pages.
But the researchers are careful about what these numbers mean.
A high number of mentions does not automatically prove strong characterization.
That distinction is crucial.
What Is TF-IDF?
One of the technical terms in the study is TF-IDF cosine similarity.
It may sound complicated, but the basic idea is relatively straightforward.
TF-IDF can represent the importance of words within a text segment. Cosine similarity can then compare two segments based on their vocabulary patterns.
In this study, adjacent pages are compared to provide a proxy for local lexical and topical cohesion.
The reported average similarity between adjacent pages was approximately 0.111.
But researchers do not treat this as a simple “good” or “bad” score.
Why?
Because stories naturally change.
A transition from an investigation scene to a character conversation may produce very different vocabulary. A low similarity value could therefore represent an intentional narrative transition rather than a problem.
This is why computational measurements need context.
Why the 110 Hz Motif Matters
One of the recurring elements in The Temple is the 110 Hz motif.
The motif is treated as one of the story's global anchors alongside the stone, temple, Geneva, and investigation frame.
Recurring anchors can provide a kind of narrative memory.
They allow readers to recognize connections between different portions of a long story.
For AI-generated fiction, this is especially important because maintaining narrative memory across a long text is a significant challenge.
A reproducible evaluation framework can therefore ask:
Where does the anchor appear?
How frequently does it return?
What events are associated with it?
Does its meaning change or develop?
Does it contribute to the story's eventual resolution?
These questions transform an impression of “the story feels connected” into something that can be documented and examined.
What Is the Ethical Narrative Brake?
The study also introduces the Ethical Narrative Brake (ENB).
The concept describes fictional-world constraints that limit irresponsible disclosure, reduce binary moral judgments, and redirect narrative action toward accountable preservation of knowledge.
In simpler terms, imagine a fictional character discovering sensitive evidence.
A simplistic story might immediately move toward:
discover → expose → punish
An ethically more complex narrative might instead consider:
discover → assess risk → examine evidence → consider responsibility → control access → preserve information
The second approach introduces additional narrative conditions.
The study evaluates these conditions through parameters such as:
- risk intensity,
- evidence certainty,
- potential harm,
- responsibility,
- right-to-know,
- preservation need,
- narrative stability.
Ethical Storytelling Can Also Improve Narrative Stability
Ethics and storytelling structure are not necessarily separate.
The study found repeated ENB patterns in scenes involving disclosure, investigation, blame, and evidence.
A notable pattern was movement from exposure toward preservation. Information that might otherwise trigger public exposure or revenge-oriented action is instead redirected toward controlled storage, system logging, limited access, delayed exposure, or accountable responsibility.
This can also help maintain character continuity.
Instead of a character suddenly changing their behavior, the decision can emerge from identifiable factors such as risk, evidence certainty, potential harm, and responsibility.
The result is a more traceable chain:
event → assessment → decision → consequence
That is valuable when evaluating AI-generated narratives.
Can AI Fiction Be Given a Score?
The study does provide exploratory ratings.
The reported averages were:
Narrative coherence: 4.25
Character continuity and development: 4.00
Originality and Arabic narrative heritage: 4.333
But these numbers should not be interpreted as a universal AI-fiction score.
The study itself notes that the rating matrix is small and based on a single case study.
This means the results demonstrate how the framework operates rather than establishing a universal standard for every AI-generated novel.
Why Human Readers Still Matter
One of the most important lessons from this research is that AI evaluation cannot be reduced entirely to automated metrics.
A computer can count:
- how often a character appears,
- how many pages contain a term,
- how similar adjacent pages are,
- and how frequently a particular anchor occurs.
But a computer metric alone cannot establish:
- emotional depth,
- literary significance,
- psychological complexity,
- symbolism,
- cultural meaning,
- or aesthetic value.
The researchers explicitly state that quantitative indicators should support, rather than replace, qualitative interpretation.
This creates a useful model for future AI-literature research:
Computational evidence + human interpretation = richer evaluation
What Could AI Fiction Evaluation Look Like in the Future?
The proposed framework could be expanded significantly.
Future researchers could use:
Semantic similarity
Contextual embeddings and BERTScore could provide deeper comparisons than simple word-frequency methods.
Topic modeling
Researchers could track how themes emerge, disappear, and return.
Narrative-role tracking
Characters could be evaluated according to their changing functions within scenes.
Readability and cohesion analysis
Tools such as Coh-Metrix could provide additional linguistic indicators.
Multi-rater evaluation
Multiple human evaluators could assess the same scenes and calculate inter-rater agreement.
These possibilities are identified as potential future extensions of the framework.
The Bigger Question: What Makes AI Writing Good?
The discussion surrounding generative AI often asks:
“Can AI write?”
But perhaps that is no longer the most interesting question.
The more important questions are:
Can AI maintain a fictional world?
Can it preserve character identity across a long narrative?
Can it connect symbols and events meaningfully?
Can it develop ethical conflicts without reducing them to simple good-versus-bad choices?
Can researchers measure these qualities consistently?
The The Temple study provides a framework for beginning to answer those questions.
Final Thoughts
AI-generated fiction is moving beyond short demonstrations and isolated passages.
As AI systems increasingly produce longer narratives, researchers need methods capable of evaluating what happens across the entire story.
The reproducible matrix proposed in this study offers one approach by combining structural-semiotic analysis with measurable indicators such as entity mentions, page coverage, consecutive presence, anchor tracking, and TF-IDF similarity.
Its introduction of the Ethical Narrative Brake adds another dimension by examining whether ethical constraints are actually embedded within the fictional world's decisions and consequences rather than appearing only as surface-level moral language.
The larger lesson is simple:
Evaluating AI-generated fiction requires more than asking whether the writing sounds human.
It requires asking whether the story holds together, whether its characters remain traceable, whether its symbols and events connect, and whether its ethical decisions meaningfully influence the fictional world.
That may become one of the defining challenges—and opportunities—of AI and literature research.
