# Class: Tokenizer

## ProtoBuf .DotProto. Tokenizer

prototype tokenizer

## Constructor

#### new Tokenizer(proto)

Constructs a new Tokenizer.

##### Parameters:

Name | Type | Description
--- | --- | ---
`proto` | string | Proto to tokenize

Source:
  - protobuf.js, line 495

### Members

#### index :number

Current index.

##### Type:

- number

Source:
  - protobuf.js, line 509

#### line :number

Current line.

##### Type:

- number

Source:
  - protobuf.js, line 516

#### source :string

Source to parse.

##### Type:

- string

Source:
  - protobuf.js, line 502

#### (non-null) stack :Array.<string>

Token stack.

##### Type:

- Array.<string>

Source:
  - protobuf.js, line 523

### Methods

#### next() → (nullable) {string}

Gets the next token and advances by one.

Source:
  - protobuf.js, line 563

##### Returns:

Token or `null` on EOF

Type
  string

#### omit(expected) → {boolean}

Omits an optional token.

##### Parameters:

Name | Type | Description
--- | --- | ---
`expected` | string | Expected optional token

Source:
  - protobuf.js, line 658

##### Returns:

`true` if the token exists

Type
  boolean

#### peek() → (nullable) {string}

Peeks for the next token.

Source:
  - protobuf.js, line 632

##### Returns:

Token or `null` on EOF

Type
  string

#### skip(expected)

Skips a specific token and throws if it differs.

##### Parameters:

Name | Type | Description
--- | --- | ---
`expected` | string | Expected token

Source:
  - protobuf.js, line 647

##### Throws:

If the actual token differs

Type
  Error

#### toString() → {string}

Returns a string representation of this object.

Source:
  - protobuf.js, line 671

##### Returns:

String representation as of "Tokenizer(index/length)"

Type
  string