Class CommonTokenStream
- All Implemented Interfaces:
IntStream
,TokenStream
BufferedTokenStream
with functionality to filter
token streams to tokens on a particular channel (tokens where
Token.getChannel()
returns a particular value).
This token stream provides access to all tokens by index or when calling
methods like BufferedTokenStream.getText()
. The channel filtering is only used for code
accessing tokens via the lookahead methods BufferedTokenStream.LA(int)
, LT(int)
, and
LB(int)
.
By default, tokens are placed on the default channel
(Token.DEFAULT_CHANNEL
), but may be reassigned by using the
->channel(HIDDEN)
lexer command, or by using an embedded action to
call Lexer.setChannel(int)
.
Note: lexer rules which use the ->skip
lexer command or call
Lexer.skip()
do not produce tokens at all, so input text matched by
such a rule will not be available as part of the token stream, regardless of
channel.
-
Field Summary
FieldsModifier and TypeFieldDescriptionprotected int
Specifies the channel to use for filtering tokens.Fields inherited from class org.antlr.v4.runtime.BufferedTokenStream
fetchedEOF, p, tokens, tokenSource
Fields inherited from interface org.antlr.v4.runtime.IntStream
EOF, UNKNOWN_SOURCE_NAME
-
Constructor Summary
ConstructorsConstructorDescriptionCommonTokenStream
(TokenSource tokenSource) Constructs a newCommonTokenStream
using the specified token source and the default token channel (Token.DEFAULT_CHANNEL
).CommonTokenStream
(TokenSource tokenSource, int channel) Constructs a newCommonTokenStream
using the specified token source and filtering tokens to the specified channel. -
Method Summary
Modifier and TypeMethodDescriptionprotected int
adjustSeekIndex
(int i) Allowed derived classes to modify the behavior of operations which change the current stream position by adjusting the target token index of a seek operation.int
Count EOF just once.protected Token
LB
(int k) LT
(int k) Methods inherited from class org.antlr.v4.runtime.BufferedTokenStream
consume, fetch, fill, filterForChannel, get, get, getHiddenTokensToLeft, getHiddenTokensToLeft, getHiddenTokensToRight, getHiddenTokensToRight, getSourceName, getText, getText, getText, getText, getTokens, getTokens, getTokens, getTokens, getTokenSource, index, LA, lazyInit, mark, nextTokenOnChannel, previousTokenOnChannel, release, reset, seek, setTokenSource, setup, size, sync
-
Field Details
-
channel
protected int channelSpecifies the channel to use for filtering tokens.The default value is
Token.DEFAULT_CHANNEL
, which matches the default channel assigned to tokens created by the lexer.
-
-
Constructor Details
-
CommonTokenStream
Constructs a newCommonTokenStream
using the specified token source and the default token channel (Token.DEFAULT_CHANNEL
).- Parameters:
tokenSource
- The token source.
-
CommonTokenStream
Constructs a newCommonTokenStream
using the specified token source and filtering tokens to the specified channel. Only tokens whoseToken.getChannel()
matcheschannel
or have theToken.getType()
equal toToken.EOF
will be returned by the token stream lookahead methods.- Parameters:
tokenSource
- The token source.channel
- The channel to use for filtering tokens.
-
-
Method Details
-
adjustSeekIndex
protected int adjustSeekIndex(int i) Description copied from class:BufferedTokenStream
Allowed derived classes to modify the behavior of operations which change the current stream position by adjusting the target token index of a seek operation. The default implementation simply returnsi
. If an exception is thrown in this method, the current stream index should not be changed.For example,
CommonTokenStream
overrides this method to ensure that the seek target is always an on-channel token.- Overrides:
adjustSeekIndex
in classBufferedTokenStream
- Parameters:
i
- The target token index.- Returns:
- The adjusted target token index.
-
LB
- Overrides:
LB
in classBufferedTokenStream
-
LT
Description copied from interface:TokenStream
Get theToken
instance associated with the value returned byLA(k)
. This method has the same pre- and post-conditions asIntStream.LA(int)
. In addition, when the preconditions of this method are met, the return value is non-null and the value ofLT(k).getType()==LA(k)
.- Specified by:
LT
in interfaceTokenStream
- Overrides:
LT
in classBufferedTokenStream
- See Also:
-
getNumberOfOnChannelTokens
public int getNumberOfOnChannelTokens()Count EOF just once.
-