Tokeniser Class Reference

A string tokeniser. More...

#include <tokeniser.hpp>

Collaboration diagram for Tokeniser:

Public Member Functions
	Tokeniser ()
	Constructs a new Tokeniser.
std::vector< std::vector< std::string > >	Feed (const std::string &raw)
	Feeds a string into a Tokeniser. More...

Private Types
enum	QuoteType : std::uint8_t { QuoteType::NONE, QuoteType::SINGLE, QuoteType::DOUBLE }
	Enumeration of quotation types. More...

Private Member Functions
void	Emit ()
	Finishes the current word and sends the line to the CommandHandler.
void	EndWord ()
	Finishes the current word, adding it to the tokenised line.
void	Push (char c)
	Pushes a raw character onto the end of the current word. More...

Private Attributes
std::vector< std::vector< std::string > >	ready_lines
	The current vector of completed, tokenised lines. More...
std::vector< std::string >	words
	The current vector of completed, tokenised words.
std::string	current_word
	The current, incomplete word to which new characters should be added. More...
bool	escape_next
	Whether the next character is to be interpreted as an escape code. More...
bool	in_word
	Whether the tokeniser is currently in a word.
QuoteType	quote_type
	The type of quotation currently being used in this Tokeniser.

Detailed Description

A string tokeniser.

A Tokeniser is fed chunks of incoming data from the IO system, and emits any fully-formed command lines it encounters to the command handler.

See also

CommandHandler

IoCore

Definition at line 24 of file tokeniser.hpp.

Member Enumeration Documentation

QuoteType

enum Tokeniser::QuoteType : std::uint8_t

strongprivate

Enumeration of quotation types.

Enumerator
NONE	Not currently in a quote pair.
SINGLE	In single quotes ('').
DOUBLE	In double quotes ("").

Definition at line 42 of file tokeniser.hpp.

                         : std::uint8_t {
        NONE,   
        SINGLE, 
        DOUBLE  
    };

Member Function Documentation

Feed()

std::vector< std::vector< std::string > > Tokeniser::Feed

(

const std::string &

raw

)

Feeds a string into a Tokeniser.

Parameters

raw	Const reference to the raw string to feed. The string need not contain complete lines.

Returns

The vector of lines that have been successfully tokenised in this tokenising pass. This vector may be empty.

Note

Escaping a multi-byte UTF-8 character is undefined behaviour.

Definition at line 24 of file tokeniser.cpp.

References DOUBLE, Emit(), EndWord(), escape_next, in_word, NONE, Push(), quote_type, ready_lines, and SINGLE.

Referenced by Connection::Read().

{
    // The list of ready lines should be cleared by any previous Feed.
    assert(this->ready_lines.empty());

    for (char c : raw) {
        if (this->escape_next) {
            this->Push(c);
            continue;
        }

        switch (this->quote_type) {
            case QuoteType::SINGLE:
                if (c == '\'') {
                    this->quote_type = QuoteType::NONE;
                } else {
                    this->Push(c);
                }
                break;

            case QuoteType::DOUBLE:
                switch (c) {
                    case '\"':
                        this->quote_type =
                                QuoteType::NONE;
                        break;

                    case '\\':
                        this->escape_next = true;
                        break;

                    default:
                        this->Push(c);
                        break;
                }
                break;

            case QuoteType::NONE:
                switch (c) {
                    case '\n':
                        this->Emit();
                        break;

                    case '\'':
                        this->in_word = true;
                        this->quote_type =
                                QuoteType::SINGLE;
                        break;

                    case '\"':
                        this->in_word = true;
                        this->quote_type =
                                QuoteType::DOUBLE;
                        break;

                    case '\\':
                        this->escape_next = true;
                        break;

                    default:
                        isspace(c) ? this->EndWord()
                                   : this->Push(c);
                        break;
                }
                break;
        }
    }

    auto lines = this->ready_lines;
    this->ready_lines.clear();

    return lines;
}

Push()

void Tokeniser::Push ( char  c )

private

Pushes a raw character onto the end of the current word.

This also clears the escape_next flag.

Parameters

c	The character to push onto the current word.

Definition at line 98 of file tokeniser.cpp.

References current_word, escape_next, in_word, NONE, and quote_type.

Referenced by Feed().

{
    assert(this->escape_next ||
           !(this->quote_type == QuoteType::NONE && isspace(c)));
    this->in_word = true;
    this->current_word.push_back(c);
    this->escape_next = false;
    assert(!this->current_word.empty());
}

Member Data Documentation

current_word

std::string Tokeniser::current_word

private

The current, incomplete word to which new characters should be added.

Definition at line 57 of file tokeniser.hpp.

Referenced by Emit(), EndWord(), and Push().

escape_next

bool Tokeniser::escape_next

private

Whether the next character is to be interpreted as an escape code.

This usually gets set to true when a backslash is detected.

Definition at line 61 of file tokeniser.hpp.

Referenced by Emit(), Feed(), and Push().

ready_lines

std::vector<std::vector<std::string> > Tokeniser::ready_lines

private

The current vector of completed, tokenised lines.

This is cleared at the end of every Tokeniser::Feed.

Definition at line 50 of file tokeniser.hpp.

Referenced by Emit(), and Feed().

---

The documentation for this class was generated from the following files:

• src/tokeniser.hpp
• src/tokeniser.cpp