Package provided.token.tokenizer

Interface Summary

IStringTokenizer	Tokenizer that only returns string tokens.
ITokenizer	Extract and return an appropriate Token from some given source.

Class Summary

ABCTokenizer	Tokenizer for the abc music language.
StringTokenizer	Special tokenizer that converts a stream of various types of tokens from StreamTokenizer and converts them into a stream of strings.