org.sandev.generator
Class RemainderTokenizer

java.lang.Object
  org.sandev.generator.RemainderTokenizer

public class RemainderTokenizer

extends java.lang.Object

Similar functionality to StringTokenizer, but with the advantage of being able to retrieve what's left of the input verbatim at any point. Since a lot of the tags we work with involve N parameters followed by a comment, we tokenize our way through the params and then retrieve the comment verbatim.

This class leverages the StringCharacterIterator combined with Character.isWhite to do its work. It does not actually make use of StringTokenizer or StreamTokenizer (not that those share anything in common either).

Field Summary

protected java.text.StringCharacterIterator	charIter Our character iterator.
protected java.lang.String	input The text we were initialized with
protected boolean	quotedTokens By default we understand anything enclosed in double quotes to be a single token.

Constructor Summary

RemainderTokenizer(java.lang.String text)
Initialize a new instance of this tokenizer

Method Summary

java.text.StringCharacterIterator	getCharIter() accessor for charIter
java.lang.String	getInput() accessor for input
java.lang.String	getRemainder() Return the remaining text verbatim.
boolean	hasMoreTokens() Return true if there are more tokens in the input.
void	init(java.lang.String text) Initialize this tokenizer to be working off the given String input.
java.lang.String	nextToken() Return the next token from the input.
boolean	readQuotedTokens() accessor for quotedTokens
void	setCharIter(java.text.StringCharacterIterator iter) mutator for charIter
void	setInput(java.lang.String input) mutator for input
void	setQuotedTokens(boolean val) mutator for quotedTokens

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

input

protected java.lang.String input

The text we were initialized with

charIter

protected java.text.StringCharacterIterator charIter

Our character iterator. This is initially created, or reset by the init method.

quotedTokens

protected boolean quotedTokens

By default we understand anything enclosed in double quotes to be a single token. This behavior can be toggled with this field at any point during processing.

Constructor Detail

RemainderTokenizer

public RemainderTokenizer(java.lang.String text)

Initialize a new instance of this tokenizer

Method Detail

getInput

public java.lang.String getInput()

accessor for input

setInput

public void setInput(java.lang.String input)

mutator for input

getCharIter

public java.text.StringCharacterIterator getCharIter()

accessor for charIter

setCharIter

public void setCharIter(java.text.StringCharacterIterator iter)

mutator for charIter

readQuotedTokens

public boolean readQuotedTokens()

accessor for quotedTokens

setQuotedTokens

public void setQuotedTokens(boolean val)

mutator for quotedTokens

init

public void init(java.lang.String text)

Initialize this tokenizer to be working off the given String input. This can also be used to reset an existing tokenizer with different input, which saves on allocation overhead.

getRemainder

public java.lang.String getRemainder()

Return the remaining text verbatim. Note that the returned result includes any whitespace around the remaining text.

hasMoreTokens

public boolean hasMoreTokens()

Return true if there are more tokens in the input. Note that it is safe to call nextToken() regardless.

nextToken

public java.lang.String nextToken()

Return the next token from the input. If there are no more tokens, then return null. Note that CharacterIterator uses a preincrement approach to next(). In other words it increments the index by one and then returns that character. That means you better make sure that your first call is to current() or you are going to eat the first character.