provided.token.tokenizer

Interface ITokenizer

All Known Implementing Classes:

ABCTokenizer

public interface ITokenizer

Extract and return an appropriate Token from some given source.

Method Summary

Modifier and Type	Method and Description
Token	getNextToken() Return the next token.
void	putBack(Token t) Put the previously consumed token back into the token stream.

Method Detail

getNextToken

Token getNextToken()

Return the next token.

Returns:

the next token in the tokenizer

putBack

void putBack(Token t)

Put the previously consumed token back into the token stream.

Parameters:

t - the token to put back into the tokenizer