Chapter 3 - Parser
The story so far. I have an-almost functioning parser for my prose-in-code format, but it keeps on collapsing under its own weight. I’d like to ignore this and get on with publishing the first chapter, but I can’t see how to add the features that I need to make this a reality without being left with an irredeemable mess. At this point I’m offered some help to show me how to use a grown-up parser, and I leap at the chance.
Alan and I go looking for parser libraries for Java. ANTLR (ANother Tool for Language Recognition) is the standard, but it is a parser generator - meaning that it generates Java source files that you have to compile. For a large project that might be acceptable, but here, its a build step too far.
Google’s fifth result is Parboiled. Parboiled allows us to specify our format in Java code, and make use of it without code generation, so it looks like a good bet. It also has a functioning Markdown parser as one of its examples, which feels like a good portent. We tell Gradle about it’s JAR file and convert a simple calculator example to Kotlin to see if it runs at all.
1 class ParserTests {
2
3 @org.junit.Rule @JvmField val approver = approvalsRule()
4
5 @Test fun test() {
6 val parser = Parboiled.createParser(CalculatorParser::class.java)
7 val result = ReportingParseRunner<Any>(parser.Expression()).run("1+2");
8 approver.assertApproved(ParseTreeUtils.printNodeTree(result));
9 }
10 }
11
12 @BuildParseTree
13 open class CalculatorParser : BaseParser<Any>() {
14
15 open fun Expression(): Rule {
16 return Sequence(
17 Term(),
18 ZeroOrMore(AnyOf("+-"), Term())
19 )
20 }
21
22 open fun Term(): Rule {
23 return Sequence(
24 Factor(),
25 ZeroOrMore(AnyOf("*/"), Factor())
26 )
27 }
28
29 open fun Factor(): Rule {
30 return FirstOf(
31 Number(),
32 Sequence('(', Expression(), ')')
33 )
34 }
35
36 open fun Number(): Rule {
37 return OneOrMore(CharRange('0', '9'))
38 }
39 }
The only change we had to make from IntelliJ’s conversion of the Java to Kotlin was to add the open keyword, as it turns out that Parboiled makes use of bytecode generation to enhance our parser class.
We used another Approvals Test here so that we could record the output from parseNodeTree for your benefit - it looks like this
1 [Expression] '1+2'
2 [Term] '1'
3 [Factor] '1'
4 [Number] '1'
5 [0..9] '1'
6 [ZeroOrMore]
7 [ZeroOrMore] '+2'
8 [Sequence] '+2'
9 [[+-]] '+'
10 [Term] '2'
11 [Factor] '2'
12 [Number] '2'
13 [0..9] '2'
14 [ZeroOrMore]
which is a representation of the Rules that matched the parsed text 1+2.
So far so good.
Now to try a subset of the book format.
1 class ParserTests {
2
3 val parser = Parboiled.createParser(Context3.BookParser::class.java) // TODO\
4 - remove Context3
5
6 @Test fun test() {
7 check("""
8 |hidden
9 |/*-
10 |first shown
11 |last shown
12 |-*/
13 |hidden
14 """);
15 }
16
17 private fun check(input: String) {
18 val parsingResult = ReportingParseRunner<Any>(parser.Root()).run(input.t\
19 rimMargin())
20 println(ParseTreeUtils.printNodeTree(parsingResult))
21 if (!parsingResult.matched) {
22 fail()
23 }
24 }
25 }
Full disclosure - this test was the end result of at least a couple of hours of iterating in on a solution as we gained understanding of how Parboiled works, and tried to find a way of having the tests tell us how well we were doing. The println is a sure sign that we’re really only using JUnit as way of exploring an API rather than writing TDD or regression tests.
What does the parser look like?
1 @BuildParseTree
2 open class BookParser : BaseParser<Any>() {
3
4 open fun NewLine() = Ch('\n')
5 open fun OptionalNewLine() = Optional(NewLine())
6 open fun Line() = Sequence(ZeroOrMore(NoneOf("\n")), NewLine())
7
8 open fun TextBlock(): Rule {
9 return Sequence(
10 Sequence(String("/*-"), NewLine()),
11 ZeroOrMore(
12 Sequence(
13 TestNot(String("-*/")),
14 ZeroOrMore(NoneOf("\n")),
15 NewLine())),
16 Sequence(String("-*/"), OptionalNewLine())
17 )
18 }
19
20 open fun Root() = ZeroOrMore(FirstOf(TextBlock(), Line()))
21 }
I won’t take this opportunity to explain PEG parsers, largely because you already know that at the time of writing I have no more than 4 hours experience. It’s a selling point of Parboiled that you can probably read the code and have a good idea how it works. That isn’t to say that we just rattled this out - it took a good 3 hours of experiment and blind alleys, even pairing with a trained computer scientist - and I’m still not convinced that it will work in all real circumstances. In order to find out we’re going to need a better way of seeing what matches.
Reading the documentation shows that we can add Actions in Sequences to capture results, which are added to a stack that is made available in the parsingResult. Let’s define a TextBlock class to hold the lines that we’re interested in and ask Parboiled to push one on the stack when when the result is available.
1 data class TextBlock(val lines: MutableList<String>) {
2 constructor(vararg lines: String) : this(mutableListOf(*lines))
3
4 fun addLine(line: String) = lines.add(line)
5 }
1 open fun TextBlockRule(): Rule {
2 val block = Var<TextBlock>(TextBlock())
3 return Sequence(
4 Sequence(String("/*-"), NewLine()),
5 ZeroOrMore(
6 Sequence(
7 TestNot(String("-*/")),
8 ZeroOrMore(NoneOf("\n")),
9 NewLine()),
10 block.get().addLine(match())),
11 Sequence(String("-*/"), OptionalNewLine()),
12 push(block.get())
13 )
14 }
Now our test can access the result stack and actually make assertions. As TextBlock is a data class we get a working definition of equals for free.
1 class ParserTests {
2
3 val parser = Parboiled.createParser(BookParser::class.java)
4
5 @Test fun `single text block`() {
6 check("""
7 |hidden
8 |/*-
9 |first shown
10 |last shown
11 |-*/
12 |hidden""",
13 expected = listOf(
14 TextBlock("first shown\n", "last shown\n"))
15 )
16 }
17
18 @Test fun `two text blocks`() {
19 check("""
20 |/*-
21 |first shown
22 |-*/
23 |hidden
24 |/*-
25 |last shown
26 |-*/""",
27 expected = listOf(
28 TextBlock("first shown\n"),
29 TextBlock("last shown\n"))
30 )
31 }
32
33 private fun check(input: String, expected: List<TextBlock>) {
34 val parsingResult = ReportingParseRunner<Any>(parser.Root()).run(input.t\
35 rimMargin())
36 if (!parsingResult.matched) {
37 fail()
38 }
39 assertEquals(expected, parsingResult.valueStack.toList().asReversed())
40 }
41 }
This obviously isn’t perfect, but it is a promising start. Parboiled is evidently working very hard behind the scenes to make this Rule-as-function which is then interpreted in context work in Java - this worries me a bit, but I’m prepared to suspend disbelief for now.
One interesting effect of the use of Parboiled is that we have introduced an intermediate form in our parsing. Previously we parsed the input and generated the output lines in one step (if you discount filtering nulls). In the new version we have built a list of TextBlock objects that we will later have to render into a Markdown file. This separation of concerns introduces complexity, but it also buys us flexibility and allows us to focus on one task at a time. Whilst there is a risk that we will have some difficulty with the rendering, I think it’s small, so I’m going to concentrate on extending the parsing for now - adding the code block markers.
First let’s add a test
1 @Test fun `single code block`() {
2 check("""
3 |hidden
4 | //`
5 | first shown
6 | last shown
7 | //`
8 |hidden""",
9 expected = listOf(
10 CodeBlock("first shown\n", "last shown\n"))
11 )
12 }
and now duplicate-and-edit an implementation
1 interface Block {
2 val lines: MutableList<String>
3 fun addLine(line: String) = lines.add(line)
4 }
5
6 data class TextBlock(override val lines: MutableList<String>) : Block {
7 constructor(vararg lines: String) : this(mutableListOf(*lines))
8 }
9
10 data class CodeBlock(override val lines: MutableList<String>) : Block {
11 constructor(vararg lines: String) : this(mutableListOf(*lines))
12 }
13
14 open fun TextBlockRule(): Rule {
15 val block = Var<TextBlock>(TextBlock())
16 return Sequence(
17 Sequence(String("/*-"), NewLine()),
18 ZeroOrMore(
19 Sequence(
20 TestNot(String("-*/")),
21 ZeroOrMore(NoneOf("\n")),
22 NewLine()),
23 block.get().addLine(match())),
24 Sequence(String("-*/"), OptionalNewLine()),
25 push(block.get())
26 )
27 }
28
29 open fun CodeBlockRule(): Rule {
30 val block = Var<CodeBlock>(CodeBlock())
31 return Sequence(
32 Sequence(String("//`"), NewLine()),
33 ZeroOrMore(
34 Sequence(
35 TestNot(String("//`")),
36 ZeroOrMore(NoneOf("\n")),
37 NewLine()),
38 block.get().addLine(match())),
39 Sequence(String("//`"), OptionalNewLine()),
40 push(block.get())
41 )
42 }
43
44 open fun Root() = ZeroOrMore(
45 FirstOf(
46 TextBlockRule(),
47 CodeBlockRule(),
48 Line()))
Unfortunately this fails, and a little thought reveals that it’s because our block-matching code only works for block markers at the start of a line. The previous behaviour was that blocks were defined by markers as the first non-space characters. My instinct here is to back up and extract some reusable Rule from TextBlockRule, and then use that to implement CodeBlockRule. Otherwise I’m going to have diverging Rules that I have to try to find the commonality in.
So, reverting to our two tests for the text block, we refactor the parser and define CodeBlockRule in much the same way as text TextBlockRule.
1 open fun TextBlockRule(): Rule {
2 val result = Var(TextBlock())
3 return BlockRule(String("/*-"), String("-*/"), result)
4 }
5
6 open fun CodeBlockRule(): Rule {
7 val result = Var(CodeBlock())
8 return BlockRule(String("//`"), String("//`"), result)
9 }
10
11 open fun <T: Block> BlockRule(startRule: Rule, endRule: Rule, accumulator: Var<T\
12 >) = Sequence(
13 BlockEnter(startRule),
14 ZeroOrMore(
15 LinesNotStartingWith(endRule),
16 accumulator.get().addLine(match())),
17 BlockExit(endRule),
18 push(accumulator.get())
19 )
20
21 open fun LinesNotStartingWith(rule: Rule) = Sequence(
22 TestNot(rule),
23 ZeroOrMore(NoneOf("\n")),
24 NewLine())
25
26 open fun BlockEnter(rule: Rule) = Sequence(rule, NewLine())
27 open fun BlockExit(rule: Rule) = Sequence(rule, OptionalNewLine())
I test this with an unindented code block, and with mixed text and code blocks, before turning to indented blocks.
1 class ParserTests {
2
3 val parser = Parboiled.createParser(BookParser::class.java)
4
5 @Test fun `single text block`() {
6 check("""
7 |hidden
8 |/*-
9 |first shown
10 |last shown
11 |-*/
12 |hidden""",
13 expected = listOf(
14 TextBlock("first shown\n", "last shown\n"))
15 )
16 }
17
18 @Test fun `two text blocks`() {
19 check("""
20 |/*-
21 |first shown
22 |-*/
23 |hidden
24 |/*-
25 |last shown
26 |-*/""",
27 expected = listOf(
28 TextBlock("first shown\n"),
29 TextBlock("last shown\n"))
30 )
31 }
32
33 @Test fun `indented code block`() {
34 check("""
35 |hidden
36 | //`
37 | first shown
38 | last shown
39 | //`
40 |hidden""",
41 expected = listOf(
42 CodeBlock(" first shown\n", " last shown\n"))
43 )
44 }
45
46 @Test fun `mix and match`() {
47 check("""
48 |hidden
49 |/*-
50 |first text
51 |last text
52 |-*/
53 | //`
54 | first code
55 | last code
56 | //`
57 |hidden""",
58 expected = listOf(
59 TextBlock("first text\n", "last text\n"),
60 CodeBlock(" first code\n", " last code\n")
61 )
62 )
63 }
64
65 private fun check(input: String, expected: List<Any>) {
66 assertEquals(expected, parse(input.trimMargin()))
67 }
68
69 private fun parse(input: String) = ReportingParseRunner<Any>(parser.Root()).\
70 run(input).valueStack.toList().asReversed()
71 }
72
73 interface Block {
74 val lines: MutableList<String>
75 fun addLine(line: String) = lines.add(line)
76 }
77
78 data class TextBlock(override val lines: MutableList<String>) : Block {
79 constructor(vararg lines: String) : this(mutableListOf(*lines))
80 }
81
82 data class CodeBlock(override val lines: MutableList<String>) : Block {
83 constructor(vararg lines: String) : this(mutableListOf(*lines))
84 }
85
86 @BuildParseTree
87 open class BookParser : BaseParser<Any>() {
88
89 open fun NewLine() = Ch('\n')
90 open fun OptionalNewLine() = Optional(NewLine())
91 open fun WhiteSpace() = AnyOf(" \t")
92 open fun OptionalWhiteSpace() = ZeroOrMore(WhiteSpace())
93 open fun Line() = Sequence(ZeroOrMore(NoneOf("\n")), NewLine())
94
95 open fun TextBlockRule(): Rule {
96 val result = Var(TextBlock())
97 return BlockRule(String("/*-"), String("-*/"), result)
98 }
99
100 open fun CodeBlockRule(): Rule {
101 val result = Var(CodeBlock())
102 return BlockRule(String("//`"), String("//`"), result)
103 }
104
105 open fun <T: Block> BlockRule(startRule: Rule, endRule: Rule, accumulator: V\
106 ar<T>) = Sequence(
107 BlockEnter(startRule),
108 ZeroOrMore(
109 LinesNotStartingWith(endRule),
110 accumulator.get().addLine(match())),
111 BlockExit(endRule),
112 push(accumulator.get())
113 )
114
115 open fun LinesNotStartingWith(rule: Rule) = Sequence(
116 OptionalWhiteSpace(),
117 TestNot(rule),
118 ZeroOrMore(NoneOf("\n")),
119 NewLine())
120
121 open fun BlockEnter(rule: Rule) = Sequence(OptionalWhiteSpace(), rule, NewLi\
122 ne())
123 open fun BlockExit(rule: Rule) = Sequence(OptionalWhiteSpace(), rule, Option\
124 alNewLine())
125
126 open fun Root() = ZeroOrMore(
127 FirstOf(
128 TextBlockRule(),
129 CodeBlockRule(),
130 Line()))
131
132 }
Well that went surprisingly well, in that it passes the tests and is pretty understandable as a parser. The main wrinkle is that I don’t seem to be able to inline the
1 val result = Var(CodeBlock())
expressions - this must be something to do with the aggressive bytecode manipulation that Parboiled is performing. I expect that I will either understand this or be bitten by it again before the end of this project.
I’m a little dissatisfied by the fact that the lines in the Block objects contain the line separators and indents, but it’s easier to throw away information than try to reconstruct it, so reason that it should stay at least for now. At least the tests document the behaviour.
Now that we have a parser that converts our source into a list of blocks, let’s see how close we can get to having our original parser tests work against the new code.
Let’s remind ourselves what our test looks like. Previously this was our only test, now we have a separate parser it looks like a low-level integration test.
1 class CodeExtractorTests {
2
3 @Test fun writes_a_markdown_file_from_Kotlin_file() {
4 checkApprovedTranslation("""
5 |package should.not.be.shown
6 |/*-
7 |Title
8 |=====
9 |
10 |This is Markdown paragraph
11 |-*/
12 |
13 |object HiddenContext {
14 | //`
15 | /* This is a code comment
16 | */
17 | fun aFunction() {
18 | return 42
19 | }
20 | //`
21 |}
22 |
23 |/*-
24 |More book text.
25 |-*/
26 """,
27 expected = """
28 |Title
29 |=====
30 |
31 |This is Markdown paragraph
32 |
33 |```kotlin
34 | /* This is a code comment
35 | */
36 | fun aFunction() {
37 | return 42
38 | }
39 |```
40 |
41 |More book text.
42 |""")
43 }
44
45 private fun checkApprovedTranslation(source: String, expected: String) {
46 assertEquals(expected.trimMargin(), translate(source.trimMargin()).joinT\
47 oString(""))
48 }
49
50 fun translate(source: String): List<String> {
51 TODO()
52 }
53 }
In order to get this to pass, I’m going to parse the source into blocks, and then render each one.
1 fun translate(source: String): List<String> {
2 val parser = Parboiled.createParser(ContextB4.BookParser::class.java)
3 val blocks: List<Block> = ReportingParseRunner<Block>(parser.Root())
4 .run(source)
5 .valueStack
6 .toList()
7 .asReversed()
8 return blocks.flatMap { render(it) }
9 }
10
11 fun render(block: Block): List<String> = when (block) {
12 is TextBlock -> block.lines
13 is CodeBlock -> listOf("\n```kotlin\n") + block.lines + "```\n\n"
14 else -> error("Unexpected block type")
15 }
The render function took a bit of tweaking with newlines in code blocks, but it works. And I don’t mind the newlines, as they allow us to express the realities of the requirements of the output Markdown. Obviously the render function should be moved to a method on Block, but I can’t resist the temptation to try the new code on the whole of the book so far. We just have to plug the latest translate into the old file-reading code and
…
it works!
Flushed with success I reorganise things and tidy up before bed.
Firstly, the top level functions are
1 fun main(args: Array<String>) {
2 val srcDir = File(args[0])
3 val outFile = File(args[1]).apply {
4 absoluteFile.parentFile.mkdirs()
5 }
6
7 val translatedLines: Sequence<String> = sourceFilesIn(srcDir)
8 .flatMap { translate(it).plus("\n") }
9
10 outFile.bufferedWriter(Charsets.UTF_8).use { writer ->
11 translatedLines.forEach {
12 writer.appendln(it)
13 }
14 }
15 }
16
17 private fun translate(file: File): Sequence<String> = Translator.translate(file.\
18 readText())
19
20 private fun sourceFilesIn(dir: File) = dir
21 .listFiles { file -> file.isSourceFile() }
22 .toList()
23 .sortedBy(File::getName)
24 .asSequence()
25
26 private fun File.isSourceFile() = isFile && !isHidden && name.endsWith(".kt")
I then pull the parser running into a Translator object - this allows us to lazily create the parser and runner and reuse them. Kotlin gives us other ways of achieving this aim, but I’m trying this object scoping out to see how it feels.
1 object Translator {
2 private val parser = Parboiled.createParser(BookParser::class.java)
3 private val runner = ReportingParseRunner<Block>(parser.Root())
4
5 fun translate(source: String): Sequence<String> = parse(source).flatMap { it\
6 .render() }.asSequence()
7
8 fun parse(source: String): List<Block> = runner.run(source).valueStack.toLis\
9 t().asReversed()
10 }
As promised I move the render function to a method on Block, with a default implementation in the interface
1 interface Block {
2 val lines: MutableList<String>
3 fun addLine(line: String) = lines.add(line)
4 fun render(): List<String> = lines
5 }
6
7 data class TextBlock(override val lines: MutableList<String>) : Block {
8 constructor(vararg lines: String) : this(mutableListOf(*lines))
9 }
10
11 data class CodeBlock(override val lines: MutableList<String>) : Block {
12 constructor(vararg lines: String) : this(mutableListOf(*lines))
13
14 override fun render() = listOf("\n```kotlin\n") + lines + "```\n\n"
15 }
and finally use install the Kotlin All-Open compiler plugin and configure it to make @BuildParseTree annotated classes open, so that we can remove the open modifiers from every method in the BookParser
1 @BuildParseTree
2 class BookParser : BaseParser<Any>() {
3
4 fun NewLine() = Ch('\n')
5 fun OptionalNewLine() = Optional(NewLine())
6 fun WhiteSpace() = AnyOf(" \t")
7 fun OptionalWhiteSpace() = ZeroOrMore(WhiteSpace())
8 fun Line() = Sequence(ZeroOrMore(NoneOf("\n")), NewLine())
9
10 fun TextBlockRule(): Rule {
11 val result = Var(TextBlock())
12 return BlockRule(String("/*-"), String("-*/"), result)
13 }
14
15 fun CodeBlockRule(): Rule {
16 val result = Var(CodeBlock())
17 return BlockRule(String("//`"), String("//`"), result)
18 }
19
20 fun <T : Block> BlockRule(startRule: Rule, endRule: Rule, accumulator: Var<T\
21 >) = Sequence(
22 BlockEnter(startRule),
23 ZeroOrMore(
24 LinesNotStartingWith(endRule),
25 accumulator.get().addLine(match())),
26 BlockExit(endRule),
27 push(accumulator.get())
28 )
29
30 fun LinesNotStartingWith(rule: Rule) = Sequence(
31 OptionalWhiteSpace(),
32 TestNot(rule),
33 ZeroOrMore(NoneOf("\n")),
34 NewLine())
35
36 fun BlockEnter(rule: Rule) = Sequence(OptionalWhiteSpace(), rule, NewLine())
37 fun BlockExit(rule: Rule) = Sequence(OptionalWhiteSpace(), rule, OptionalNew\
38 Line())
39
40 fun Root() = ZeroOrMore(
41 FirstOf(
42 TextBlockRule(),
43 CodeBlockRule(),
44 Line()))
45
46 }
Now, a day’s work from where we decided to try a proper parser, I am in a position to add the file include directive.
Firstly, a test for the parsing.
1 @Test fun `include directive`() {
2 check("""
3 |hidden
4 |//#include "filename1.txt"
5 |//#include "filename2.txt"
6 |hidden""",
7 expected = listOf(
8 IncludeDirective("filename1.txt"),
9 IncludeDirective("filename2.txt")
10 )
11 )
12 }
We can punt on how IncludeDirective should behave for now
1 data class IncludeDirective(var path: String) : Block {
2 override val lines: MutableList<String>
3 get() = TODO("not implemented")
4 }
and just implement the parsing
1 fun IncludeRule(): Rule {
2 return Sequence(
3 OptionalWhiteSpace(),
4 String("//#include "),
5 OptionalWhiteSpace(),
6 Ch('"'),
7 ZeroOrMore(NoneOf("\"")),
8 pushIncludeDirective,
9 Ch('"'),
10 OptionalWhiteSpace(),
11 NewLine()
12 )
13 }
14
15 private val pushIncludeDirective = Action<Any> { context ->
16 context.valueStack.push(IncludeDirective(context.match))
17 true
18 }
19
20 fun Root() = ZeroOrMore(
21 FirstOf(
22 IncludeRule(),
23 TextBlockRule(),
24 CodeBlockRule(),
25 Line()))
Of course it took me another hour to get IncludeRule working, even after discovering Parboiled’s useful TracingParseRunner. I am beginning to internalise, if not actually understand, the interaction between Rules and Actions - leading to the creation of pushIncludeDirective, so maybe I’m becoming a real programmer after all these years. Not so fast that I trust my intuition though, which is why the test has two include directives to check that I don’t end up sharing state like last time.
Now we know where includes are in our source, we need to process them. In order to do this we need to know how to resolve a path to its contents.
For now, I’m just going to pass a contentResolver function to the render method of Block, which, now it is the parent of IncludeDirective, had better be restructured a bit
1 interface Block {
2 fun render(contentResolver: (String) -> List<String>): List<String>
3 }
4
5 abstract class SourceBlock() : Block {
6 abstract val lines: MutableList<String>
7 fun addLine(line: String) = lines.add(line)
8 override fun render(contentResolver: (String) -> List<String>): List<String>\
9 = lines
10 }
11
12 data class TextBlock(override val lines: MutableList<String>) : SourceBlock() {
13 constructor(vararg lines: String) : this(mutableListOf(*lines))
14 }
15
16 data class CodeBlock(override val lines: MutableList<String>) : SourceBlock() {
17 constructor(vararg lines: String) : this(mutableListOf(*lines))
18
19 override fun render(contentResolver: (String) -> List<String>) = listOf("\n`\
20 ``kotlin\n") + lines + "```\n\n"
21 }
22
23 data class IncludeDirective(var path: String) : Block {
24 override fun render(contentResolver: (String) -> List<String>) = contentReso\
25 lver(path)
26 }
Now our Translator object can pass the contentResolver to each block’s render
1 object Translator {
2 private val parser = Parboiled.createParser(BookParser::class.java)
3 private val runner = ReportingParseRunner<Block>(parser.Root())
4
5 fun translate(source: String, contentResolver: (String) -> List<String>): Se\
6 quence<String> = parse(source)
7 .flatMap { it.render(contentResolver) }
8 .asSequence()
9
10 fun parse(source: String): List<Block> = runner.run(source).valueStack.toLis\
11 t().asReversed()
12 }
leaving the top-level translate(File) function to supply an implementation of the contentResolver
1 private fun translate(file: File): Sequence<String> =
2 Translator.translate(file.readText(),
3 contentResolver = { path ->
4 file.parentFile.resolve(path).readLines().map { it + "\n" }
5 }
6 )
and yes it did take me a while to realise that I needed to re-add the newlines stripped by File.readLines.