Class UnbufferedCharStream

java.lang.Object
org.antlr.v4.runtime.UnbufferedCharStream
All Implemented Interfaces:
CharStream, IntStream

public class UnbufferedCharStream extends Object implements CharStream
Do not buffer up the entire char stream. It does keep a small buffer for efficiency and also buffers while a mark exists (set by the lookahead prediction in parser). "Unbuffered" here refers to fact that it doesn't buffer all data, not that's it's on demand loading of char. Before 4.7, this class used the default environment encoding to convert bytes to UTF-16, and held the UTF-16 bytes in the buffer as chars. As of 4.7, the class uses UTF-8 by default, and the buffer holds Unicode code points in the buffer as ints.
  • Field Details

    • data

      protected int[] data
      A moving window buffer of the data being scanned. While there's a marker, we keep adding to buffer. Otherwise, consume() resets so we start filling at index 0 again.
    • n

      protected int n
      The number of characters currently in data.

      This is not the buffer capacity, that's data.length.

    • p

      protected int p
      0..n-1 index into data of next character.

      The LA(1) character is data[p]. If p == n, we are out of buffered characters.

    • numMarkers

      protected int numMarkers
      Count up with mark() and down with release(). When we release() the last mark, numMarkers reaches 0 and we reset the buffer. Copy data[p]..data[n-1] to data[0]..data[(n-1)-p].
    • lastChar

      protected int lastChar
      This is the LA(-1) character for the current position.
    • lastCharBufferStart

      protected int lastCharBufferStart
      When numMarkers > 0, this is the LA(-1) character for the first character in data. Otherwise, this is unspecified.
    • currentCharIndex

      protected int currentCharIndex
      Absolute character index. It's the index of the character about to be read via LA(1). Goes from 0 to the number of characters in the entire stream, although the stream size is unknown before the end is reached.
    • input

      protected Reader input
    • name

      public String name
      The name or source of this char stream.
  • Constructor Details

    • UnbufferedCharStream

      public UnbufferedCharStream()
      Useful for subclasses that pull char from other than this.input.
    • UnbufferedCharStream

      public UnbufferedCharStream(int bufferSize)
      Useful for subclasses that pull char from other than this.input.
    • UnbufferedCharStream

      public UnbufferedCharStream(InputStream input)
    • UnbufferedCharStream

      public UnbufferedCharStream(Reader input)
    • UnbufferedCharStream

      public UnbufferedCharStream(InputStream input, int bufferSize)
    • UnbufferedCharStream

      public UnbufferedCharStream(InputStream input, int bufferSize, Charset charset)
    • UnbufferedCharStream

      public UnbufferedCharStream(Reader input, int bufferSize)
  • Method Details

    • consume

      public void consume()
      Description copied from interface: IntStream
      Consumes the current symbol in the stream. This method has the following effects:
      • Forward movement: The value of index() before calling this method is less than the value of index() after calling this method.
      • Ordered lookahead: The value of LA(1) before calling this method becomes the value of LA(-1) after calling this method.
      Note that calling this method does not guarantee that index() is incremented by exactly 1, as that would preclude the ability to implement filtering streams (e.g. CommonTokenStream which distinguishes between "on-channel" and "off-channel" tokens).
      Specified by:
      consume in interface IntStream
    • sync

      protected void sync(int want)
      Make sure we have 'need' elements from current position p. Last valid p index is data.length-1. p+need-1 is the char index 'need' elements ahead. If we need 1 element, (p+1-1)==p must be less than data.length.
    • fill

      protected int fill(int n)
      Add n characters to the buffer. Returns the number of characters actually added to the buffer. If the return value is less than n, then EOF was reached before n characters could be added.
    • nextChar

      protected int nextChar() throws IOException
      Override to provide different source of characters than input.
      Throws:
      IOException
    • add

      protected void add(int c)
    • LA

      public int LA(int i)
      Description copied from interface: IntStream
      Gets the value of the symbol at offset i from the current position. When i==1, this method returns the value of the current symbol in the stream (which is the next symbol to be consumed). When i==-1, this method returns the value of the previously read symbol in the stream. It is not valid to call this method with i==0, but the specific behavior is unspecified because this method is frequently called from performance-critical code.

      This method is guaranteed to succeed if any of the following are true:

      • i>0
      • i==-1 and index() returns a value greater than the value of index() after the stream was constructed and LA(1) was called in that order. Specifying the current index() relative to the index after the stream was created allows for filtering implementations that do not return every symbol from the underlying source. Specifying the call to LA(1) allows for lazily initialized streams.
      • LA(i) refers to a symbol consumed within a marked region that has not yet been released.

      If i represents a position at or beyond the end of the stream, this method returns IntStream.EOF.

      The return value is unspecified if i<0 and fewer than -i calls to consume() have occurred from the beginning of the stream before calling this method.

      Specified by:
      LA in interface IntStream
    • mark

      public int mark()
      Return a marker that we can release later.

      The specific marker value used for this class allows for some level of protection against misuse where seek() is called on a mark or release() is called in the wrong order.

      Specified by:
      mark in interface IntStream
      Returns:
      An opaque marker which should be passed to release() when the marked range is no longer required.
    • release

      public void release(int marker)
      Decrement number of markers, resetting buffer if we hit 0.
      Specified by:
      release in interface IntStream
      Parameters:
      marker -
      See Also:
    • index

      public int index()
      Description copied from interface: IntStream
      Return the index into the stream of the input symbol referred to by LA(1).

      The behavior of this method is unspecified if no call to an initializing method has occurred after this stream was constructed.

      Specified by:
      index in interface IntStream
    • seek

      public void seek(int index)
      Seek to absolute character index, which might not be in the current sliding window. Move p to index-bufferStartIndex.
      Specified by:
      seek in interface IntStream
      Parameters:
      index - The absolute index to seek to.
    • size

      public int size()
      Description copied from interface: IntStream
      Returns the total number of symbols in the stream, including a single EOF symbol.
      Specified by:
      size in interface IntStream
    • getSourceName

      public String getSourceName()
      Description copied from interface: IntStream
      Gets the name of the underlying symbol source. This method returns a non-null, non-empty string. If such a name is not known, this method returns IntStream.UNKNOWN_SOURCE_NAME.
      Specified by:
      getSourceName in interface IntStream
    • getText

      public String getText(Interval interval)
      Description copied from interface: CharStream
      This method returns the text for a range of characters within this input stream. This method is guaranteed to not throw an exception if the specified interval lies entirely within a marked range. For more information about marked ranges, see IntStream.mark().
      Specified by:
      getText in interface CharStream
      Parameters:
      interval - an interval within the stream
      Returns:
      the text of the specified interval
    • getBufferStartIndex

      protected final int getBufferStartIndex()