Stripping Markdown Barely Affects Jev's LLM Style Scores: An Empirical Test
Contents
In our previous post on TypeSafe AI’s Jev, we covered the architectural concept of a decision-only model that outputs probabilistic classifications rather than generating natural language text. When passed text and a predefined schema, Jev rapidly returns probabilities for boolean, scored, or categorical choices.
This time, we sent actual blog posts to Jev to test whether stripping newlines or Markdown formatting alters its assessment of LLM-like writing style.
Evaluation Schema
Jev requires a schema defining the primitive type, instructions, and evaluation criteria for each field.
We took the primitive type templates and state placeholder JSON from the Jev Playground and passed them to Codex, which generated the English instructions and criteria.
We used that schema directly, testing one field each from the core types: Noul, Score, and Choice.
llm_style uses Noul (a boolean-equivalent primitive unique to Jev returning true/false probabilities) to evaluate whether formulaic, LLM-typical patterns stand out.
Values are returned as the probability of true.
rhythm uses Score (discrete grading) to rate the natural flow and variation of the prose on a 4-level scale from 0 to 3.
| Level | Criteria |
|---|---|
| 0 | Highly repetitive or mechanically structured |
| 1 | Somewhat repetitive |
| 2 | Mostly natural and varied |
| 3 | Highly natural and varied |
dominant_style uses Choice (single categorical selection) to classify the prose into one of four descriptions.
| Option | Criteria |
|---|---|
| formulaic | Formulaic and predictably structured |
| fragmented | Fragmented, with frequent abrupt breaks |
| verbose | Verbose or unnecessarily explanatory |
| natural | Natural prose without a dominant structural issue |
The instructions and criteria were passed directly in English as generated by Codex.
The complete schema looks as follows, and the article body is passed as {"text": "prose text..."}.
{
"llm_style": {
"type": "noul",
"instructions": "Does this text exhibit stylistic patterns commonly associated with LLM-generated writing?",
"criteria": {
"true": "The writing prominently exhibits formulaic patterns commonly associated with LLM-generated prose",
"false": "The writing does not prominently exhibit such patterns"
}
},
"rhythm": {
"type": "score",
"instructions": "How natural and varied is the rhythm of the prose?",
"criteria": [
"Highly repetitive or mechanically structured",
"Somewhat repetitive",
"Mostly natural and varied",
"Highly natural and varied"
]
},
"dominant_style": {
"type": "choice",
"instructions": "Which description best characterizes the prose?",
"criteria": {
"formulaic": "Formulaic and predictably structured",
"fragmented": "Fragmented, with frequent abrupt breaks",
"verbose": "Verbose or unnecessarily explanatory",
"natural": "Natural prose without a dominant structural issue"
}
}
}Input Articles
As a baseline for human-written prose, we selected the post on installing Ubuntu on a Fujitsu ESPRIMO G5010/E. Frontmatter metadata was excluded.
While Claude drafted an initial outline, the author rewrote most of the body text directly.
Three Input Formats
From this identical text, we prepared three input formats with varying degrees of newlines and Markdown formatting.
Format A is the untouched original Markdown. Format B strips all newlines. Format C strips both newlines and Markdown syntax into plain text.
| Format | A (Original) | B (No newlines) | C (Plain text) |
|---|---|---|---|
| Character count | 3,075 | 2,943 | 2,485 |
In Format B, trailing double spaces (Markdown line breaks) were removed, and single spaces were inserted only where alphanumeric tokens would otherwise merge across line breaks.
In Format C, heading markers (##), link targets, backticks, table delimiters and divider rows, and images were stripped.
Link anchor text was retained, and code block fences were removed while preserving code content.
Numbered list indicators were kept because the ESPRIMO article explicitly refers to items by number (e.g., “all 1 to 4 failed”).
Results for the ESPRIMO Article
The llm_style value is the probability of true.
rhythm equals the weighted expectation across levels 0 to 3.
Rows under dominant show the probability distribution across dominant_style options.
Confidence is the UI value shown for rhythm and dominant_style.
| Field | A | B | C |
|---|---|---|---|
| llm_style | 29% | 25% | 26% |
| rhythm | 2.42 | 2.33 | 2.21 |
| Level 0 prob | 0% | 0% | 0% |
| Level 1 prob | 1% | 2% | 8% |
| Level 2 prob | 56% | 63% | 63% |
| Level 3 prob | 43% | 35% | 29% |
| Confidence | 56% | 62% | 63% |
| dominant | |||
| natural | 81% | 72% | 51% |
| verbose | 17% | 26% | 42% |
| fragmented | 0% | 1% | 6% |
| formulaic | 1% | 1% | 1% |
| Confidence | 75% | 64% | 35% |
The UI reported latencies of 118 + 215ms for A, 161 + 287ms for B, and 182 + 260ms for C.
Stripping newlines and Markdown syntax (Format C) barely moved the llm_style score, which stayed between 25% and 29%.
In contrast, rhythm dropped from 2.42 to 2.21, while natural fell from 81% to 51% and verbose increased to 42% under dominant_style.
Execution result for Format A (Original Markdown):
Execution result for Format B (Stripped newlines):
Execution result for Format C (Plain text):
Run-to-Run Variance on Identical Input
Format A was evaluated initially, and then run a second time after testing B and C.
The table and screenshots above reflect this second run.
| Field | Run 1 | Run 2 |
|---|---|---|
| llm_style | 29% | 29% |
| rhythm | 2.46 | 2.42 |
| Level 0 prob | — | 0% |
| Level 1 prob | — | 1% |
| Level 2 prob | — | 56% |
| Level 3 prob | — | 43% |
| Confidence | 52% | 56% |
| dominant | ||
| natural | 81% | 81% |
| verbose | 17% | 17% |
| fragmented | 1% | 0% |
| formulaic | — | 1% |
| Confidence | 75% | 75% |
In Run 1, detailed distribution breakdowns were collapsed when captured, so individual probabilities for rhythm and formulaic were not displayed on screen.
Reported UI latencies were 148 + 264ms for Run 1 and 118 + 215ms for Run 2.
Results for the Qwen-Generated Article
As a counterpoint, we evaluated the post on building a fictional record label site with pi.dev and Qwen across the same three formats.
That post was generated and styled almost entirely with Qwen3.7 / 3.8, with virtually no manual line editing.
We omitted the production notes and closing commentary after the final --- divider and passed only the main body.
| Format | A (Original) | B (No newlines) | C (Plain text) |
|---|---|---|---|
| Character count | 9,875 | 9,654 | 8,252 |
| Field | A | B | C |
|---|---|---|---|
| llm_style | 61% | 60% | 58% |
| rhythm | 1.38 | 1.21 | 0.89 |
| Level 0 prob | 15% | 22% | 36% |
| Level 1 prob | 35% | 38% | 40% |
| Level 2 prob | 46% | 38% | 22% |
| Level 3 prob | 4% | 2% | 2% |
| Confidence | 29% | 36% | 38% |
| dominant | |||
| natural | 27% | 20% | 6% |
| verbose | 67% | 72% | 48% |
| fragmented | 3% | 4% | 44% |
| formulaic | 3% | 4% | 2% |
| Confidence | 56% | 63% | 30% |
Reported UI latencies were 125 + 211ms for A, 135 + 238ms for B, and 111 + 168ms for C.
The Qwen article started with a higher llm_style score near 60%. Across formats, llm_style held steady between 58% and 61%. In contrast, rhythm dropped from 1.38 to 0.89, and fragmented climbed to 44% in dominant_style.
The Qwen post contained six tables and six code blocks. In Format C, their contents merged directly into running text without delimiter lines or formatting cues.
Execution result for Format A (Original Markdown):
Execution result for Format B (Stripped newlines):
Execution result for Format C (Plain text):
Pre-ChatGPT Human Writing from 2011
To test older authentic writing, we fed in two posts written in 2011 on a previous personal blog, well before ChatGPT existed.
Because prose-only posts from that era were typically around 1,000 characters, we selected one photo diary (953 characters) and one tech study group report (825 characters).
Both were original plain text without Markdown syntax, evaluated in their native formatting with newlines intact.
| Field | Diary | Study Group Report |
|---|---|---|
| llm_style | 16% | 14% |
| rhythm | 2.45 | 2.69 |
| Level 0 prob | 0% | 0% |
| Level 1 prob | 6% | 1% |
| Level 2 prob | 43% | 29% |
| Level 3 prob | 51% | 70% |
| Confidence | 45% | 69% |
| dominant | ||
| natural | 5% | 55% |
| verbose | 0% | 8% |
| fragmented | 95% | 37% |
| formulaic | 0% | 0% |
| Confidence | 93% | 39% |
Reported UI latencies were 136 + 226ms for the diary and 74 + 172ms for the study group report.
Writing from 2011 scored low on llm_style (14% to 16%) while maintaining high rhythm scores (2.45 to 2.69). However, the diary returned an extreme 95% score for fragmented.
This diary was originally a photo log. When converted to text, former image placements became empty lines, taking up 71 of its 120 total lines.
Qwen (A) returned only 3% fragmented despite a 61% llm_style score. In contrast, the 2011 diary reached 95% fragmented while staying at 16% llm_style. The two metrics did not correlate in these runs.
Execution result for the 2011 Diary:
Execution result for the 2011 Study Group Report:
Varying the Rubric-Authoring LLMs
The initial schema used above was generated by Codex from Playground templates.
Because altering input formats caused almost no shift in llm_style, we next kept the input articles fixed and varied the LLM that authored the evaluation schema.
The judging model itself remained fixed to Jev; only the prompt instructions and criteria passed to Jev were rewritten by different models.
We provided identical Playground templates to Claude Opus 5, Qwen3.8-Max, and Gemini 3.8 Flash High (via AGY) to generate evaluation schemas.
Without constraints, each model used different field counts, scale ranges, and languages.
To keep the comparison direct, we fixed the field keys, primitive types, the 4-level scale for rhythm, and choice keys for dominant_style to match Codex.
The models authored only the English instructions and criteria.
For Codex (GPT), we retained the schema from the earlier tests.
Inputs were fixed to Format A (original text) for both the ESPRIMO and Qwen articles.
Under the Codex schema, llm_style separated clearly at 29% for ESPRIMO and 61% for Qwen.
We tested whether this gap persisted across rubrics authored by the other models.
| Rubric Author | Target Article | llm_style | rhythm | dominant_style (Top) |
|---|---|---|---|---|
| Codex (GPT) | ESPRIMO (A) | 29% | 2.42 | natural 81%, verbose 17% |
| Codex (GPT) | Qwen (A) | 61% | 1.38 | verbose 67%, natural 27% |
| Claude Opus 5 | ESPRIMO (A) | 9% | 2.86 | natural 100% |
| Claude Opus 5 | Qwen (A) | 21% | 1.72 | natural 52%, formulaic 43% |
| Gemini 3.8 Flash High | ESPRIMO (A) | 17% | 2.80 | natural 99% |
| Gemini 3.8 Flash High | Qwen (A) | 47% | 2.02 | natural 51%, formulaic 34% |
| Qwen3.8-Max | ESPRIMO (A) | 30% | 2.75 | natural 98% |
| Qwen3.8-Max | Qwen (A) | 55% | 1.71 | natural 67%, verbose 26% |
Reported UI latencies were as follows:
- Claude Opus 5: ESPRIMO was 179 + 235ms, Qwen was 177 + 169ms
- Gemini 3.8 Flash High: ESPRIMO was 84 + 169ms, Qwen was 161 + 173ms
- Qwen3.8-Max: ESPRIMO was 119 + 174ms, Qwen was 127 + 208ms
Under every rubric, the Qwen post scored 12 to 32 percentage points higher on llm_style than the ESPRIMO post.
For rhythm, ESPRIMO scored consistently in the high 2.7s to 2.8s, while Qwen ranged from 1.38 to 2.02.
Claude Opus 5 returned lower llm_style probabilities for both articles (9% and 21%).
Claude’s criteria explicitly specified Japanese stock phrases such as 「〜と言えるでしょう」, 「〜することが重要です」, 「いかがでしたか」, and transitions like 「まず／次に／さらに／最後に」. Stricter pattern definitions likely kept true probabilities lower overall.
The Gemini rubric (17% vs 47%) and Qwen rubric (30% vs 55%) produced 25-to-30 point gaps similar to Codex (29% vs 61%).
Under Qwen’s rubric, dominant_style for the Qwen post allocated 26% to verbose, matching the tendency toward over-explanation seen in Codex’s 67%.
While absolute scores shifted depending on which model wrote the rubric, the gap between human-written and LLM-generated prose remained across all schema variants.
Execution result for Claude Opus 5 schema on ESPRIMO (Format A):
Execution result for Claude Opus 5 schema on Qwen (Format A):
Execution result for Gemini 3.8 Flash High schema on ESPRIMO (Format A):
Execution result for Gemini 3.8 Flash High schema on Qwen (Format A):
Execution result for Qwen3.8-Max schema on ESPRIMO (Format A):
Execution result for Qwen3.8-Max schema on Qwen (Format A):
