provided.token.tokenizer
Class StringTokenizer

java.lang.Object
  provided.token.tokenizer.StringTokenizer

All Implemented Interfaces:

IStringTokenizer

Direct Known Subclasses:

ABCTokenizer

public class StringTokenizer
extends java.lang.Object
implements IStringTokenizer

Special tokenizer that converts a stream of various types of tokens from StreamTokenizer and converts them into a stream of strings.

Author:

swong

Field Summary

protected java.io.Reader	_fileReader Reader to use.
protected java.io.StreamTokenizer	_st StreamTokenizer to use.
private java.util.Stack<java.lang.String>	stack Stack used for token push-back

Constructor Summary

StringTokenizer(java.lang.String inputFileName)
Initialize _st to read from a input Reader file with the given input file name.

Method Summary

java.lang.String	getNextStringToken() Get the next token in the stream
private java.lang.String	makeNextStringToken() Returns the next string in the stream.
void	putStringBack(java.lang.String str) Puts the given string back into the stream.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

stack

private java.util.Stack<java.lang.String> stack

Stack used for token push-back

_st

protected java.io.StreamTokenizer _st

StreamTokenizer to use.

_fileReader

protected java.io.Reader _fileReader

Reader to use.

Constructor Detail

StringTokenizer

public StringTokenizer(java.lang.String inputFileName)
                throws java.io.FileNotFoundException

Initialize _st to read from a input Reader file with the given input file name.

Parameters:

inputFileName - the name of the input text file

Throws:

java.io.FileNotFoundException

Method Detail

getNextStringToken

public java.lang.String getNextStringToken()

Get the next token in the stream

Specified by:

getNextStringToken in interface IStringTokenizer

Returns:

the next token.

putStringBack

public void putStringBack(java.lang.String str)

Puts the given string back into the stream. If nothing else is done, this would be the next token returned by getNextToken

Specified by:

putStringBack in interface IStringTokenizer

makeNextStringToken

private java.lang.String makeNextStringToken()

Returns the next string in the stream. Returns LF ("\n") at end of line. Returns null for EOF or error

Returns:

A string token or null if EOF