E1

""

Empty string (parsed as whitespace)

E2

"\n"

{}

Empty line

E3

"''\n"

{""}

Empty single-quoted string (parsed as empty string)

E4

"\"\"\n"

{""}

Empty double-quoted string (parsed as empty string)

W1

"foo bar baz\n"

{"foo", "bar", "baz"}

Space-delimited words

W2

"foo\tbar\tbaz\n"

{"foo", "bar", "baz"}

Tab-delimited words

W3

"foo\rbar\rbaz\n"

{"foo", "bar", "baz"}

Words delimited by odd, but valid, whitespace

W4

"silly windows\r\n"

{"silly", "windows"}

CRLF line ending tolerance

W5

" abc def\n"

{"abc", "def"}

Leading whitespace

W6

"ghi jkl \n"

{"ghi", "jkl"}

Trailing whitespace

W7

" mno pqr \n"

{"mno", "pqr"}

Surrounding whitespace

Q1

"abc\\\ndef\n"

{"abc\ndef"}

Backslash escaping

Q2

"\"abc\ndef\"\n"

{"abc\ndef"}

Double-quoting

Q3

"\"abc\\\ndef\"\n"

{"abc\ndef"}

Double-quoting with backslash escape

Q4

"'abc\ndef'\n"

{"abc\ndef"}

Single-quoting

Q5

"'abc\\\ndef'\n"

{"abc\\\ndef"}

Single-quoting with (failed) backslash escape

Q6

"Scare\\\" quotes\\\"\n"

{"Scare\"", "quotes\""}

Backslash-escaped double quote

Q7

"I\\'m free\n"

{"I’m", "free"}

Backslash-escaped single quote

Q8

"'hello, I'\\''m an escaped single quote'\n"

{"hello, I’m an escaped single quote"}

Single-quoted single quote

Q9

"\"hello, this is an \\\" escaped double quote\"\n"

{"hello, this is an \" escaped double quote"}

Double-quoted double quote

M1

"first line\nsecond line\n"

{"first", "line"}, {"second", "line"}

Multiple lines in one tokenisation

U1

"北野 武\n"

{"北野", "武"}

Valid UTF-8 (provided the string literals are interpreted as UTF-8, for example in Go)

U2

"f\xfcr\n"

{"f\xef\xbf\xbdr"}

Invalid UTF-8 (ISO-8859-1, using Go syntax)

X1

"enqueue file \"C:\\\\Users\\\\Test\\\\Artist - Title.mp3\" 1\n"

{"enqueue", "file", "C:\\Users\\Test\\Artist - Title.mp3", "1"}

Representative command with Windows path