Constructor
new Tokenizer(proto)
Constructs a new Tokenizer.
Parameters:
| Name | Type | Description |
|---|---|---|
proto |
string | Proto to tokenize |
- Source:
Members
index :number
Current index.
Type:
- number
- Source:
line :number
Current line.
Type:
- number
- Source:
source :string
Source to parse.
Type:
- string
- Source:
(non-null) stack :Array.<string>
Token stack.
Type:
- Array.<string>
- Source:
Methods
next() → (nullable) {string}
Gets the next token and advances by one.
- Source:
Returns:
Token or null on EOF
- Type
- string
omit(expected) → {boolean}
Omits an optional token.
Parameters:
| Name | Type | Description |
|---|---|---|
expected |
string | Expected optional token |
- Source:
Returns:
true if the token exists
- Type
- boolean
peek() → (nullable) {string}
Peeks for the next token.
- Source:
Returns:
Token or null on EOF
- Type
- string
skip(expected)
Skips a specific token and throws if it differs.
Parameters:
| Name | Type | Description |
|---|---|---|
expected |
string | Expected token |
- Source:
Throws:
-
If the actual token differs
- Type
- Error
toString() → {string}
Returns a string representation of this object.
- Source:
Returns:
String representation as of "Tokenizer(index/length)"
- Type
- string