Overview

Package token defines constants representing the lexical tokens of the Go
programming language and basic operations on tokens (printing, predicates).

Index

position.go token.go

A set of constants for precedence-based expression parsing. Non-operators have
lowest precedence, followed by operators starting with precedence 1 up to unary
operators. The highest precedence serves as “catch-all” precedence for selector,
indexing, and other operator and delimiter tokens.

type 

  1. type File struct {
  2.     // contains filtered or unexported fields
  3. }

A File is a handle for a file belonging to a FileSet. A File has a name, size,
and line offset table.

func (*File) 

  1. func (f *) AddLine(offset int)

AddLine adds the line offset for a new line. The line offset must be larger than
the offset for the previous line and smaller than the file size; otherwise the
line offset is ignored.

func (*File) 

  1. func (f *) AddLineInfo(offset int, filename , line int)

AddLineInfo adds alternative file and line number information for a given file
offset. The offset must be larger than the offset for the previously added
alternative line info and smaller than the file size; otherwise the information
is ignored.

AddLineInfo is typically used to register alternative position information for
//line filename:line comments in source files.

func (*File) 

  1. func (f *) Base() int

Base returns the base offset of file f as registered with AddFile.

func (*File) 

  1. func (f *) Line(p Pos)

Line returns the line number for the given file position p; p must be a Pos
value in that file or NoPos.

func (*File) LineCount

  1. func (f *File) LineCount()

LineCount returns the number of lines in file f.

func (*File) MergeLine

  1. func (f *File) MergeLine(line )

MergeLine merges a line with the following line. It is akin to replacing the
newline character at the end of the line with a space (to not change the
remaining offsets). To obtain the line number, consult e.g. Position.Line.
MergeLine will panic if given an invalid line number.

func (*File) Name

  1. func (f *File) Name()

Name returns the file name of file f as registered with AddFile.

func (*File) Offset

  1. func (f *File) Offset(p ) int

Offset returns the offset for the given file position p; p must be a valid Pos
value in that file. f.Offset(f.Pos(offset)) == offset.

func (*File) 

  1. func (f *) Pos(offset int)

Pos returns the Pos value for the given file offset; the offset must be <=
f.Size(). f.Pos(f.Offset(p)) == p.

func (*File) Position

  1. func (f *File) Position(p ) (pos Position)

Position returns the Position value for the given file position p. Calling
f.Position(p) is equivalent to calling f.PositionFor(p, true).

  1. func (f *) PositionFor(p Pos, adjusted ) (pos Position)

func (*File) 

  1. func (f *) SetLines(lines []int)

SetLines sets the line offsets for a file and reports whether it succeeded. The
line offsets are the offsets of the first character of each line; for instance
for the content “ab\nc\n” the line offsets are {0, 3}. An empty file has an
empty line offset table. Each line offset must be larger than the offset for the
previous line and smaller than the file size; otherwise SetLines fails and
returns false. Callers must not mutate the provided slice after SetLines
returns.

func (*File) SetLinesForContent

SetLinesForContent sets the line offsets for the given file content. It ignores
position-altering //line comments.

func (*File) Size

  1. func (f *File) Size()

Size returns the size of file f as registered with AddFile.

type FileSet

  1. type FileSet struct {
  2.     // contains filtered or unexported fields
  3. }

A FileSet represents a set of source files. Methods of file sets are
synchronized; multiple goroutines may invoke them concurrently.

func NewFileSet

  1. func NewFileSet() *FileSet

NewFileSet creates a new file set.

func (*FileSet) 

  1. func (s *) AddFile(filename string, base, size ) *File

AddFile adds a new file with a given filename, base offset, and file size to the
file set s and returns the file. Multiple files may have the same name. The base
offset must not be smaller than the FileSet’s Base(), and size must not be
negative. As a special case, if a negative base is provided, the current value
of the FileSet’s Base() is used instead.

Adding the file will set the file set’s Base() value to base + size + 1 as the
minimum base value for the next file. The following relationship exists between
a Pos value p for a given file offset offs:

  1. int(p) = base + offs

with offs in the range [0, size] and thus p in the range [base, base+size]. For
convenience, File.Pos may be used to create file-specific position values from a
file offset.

func (*FileSet) 

  1. func (s *) Base() int

Base returns the minimum base offset that must be provided to AddFile when
adding the next file.

func (*FileSet)

    File returns the file that contains the position p. If no such file is found
    (for instance for p == NoPos), the result is nil.

    func (*FileSet) 

    1. func (s *) Iterate(f func(*File) )

    Iterate calls f for the files in the file set in the order they were added until
    f returns false.

    func (*FileSet) Position

    1. func (s *FileSet) Position(p ) (pos Position)

    Position converts a Pos p in the fileset into a Position value. Calling
    s.Position(p) is equivalent to calling s.PositionFor(p, true).

    func (*FileSet) 

    1. func (s *) PositionFor(p Pos, adjusted ) (pos Position)

    PositionFor converts a Pos p in the fileset into a Position value. If adjusted
    is set, the position may be adjusted by position-altering //line comments;
    otherwise those comments are ignored. p must be a Pos value in s or NoPos.

    1. func (s *) Read(decode func(interface{}) error)

    Read calls decode to deserialize a file set into s; s must not be nil.

    func (*FileSet) Write

    1. func (s *FileSet) Write(encode func(interface{}) ) error

    1. type Pos

    Pos is a compact encoding of a source position within a file set. It can be
    converted into a Position for a more convenient, but much larger,
    representation.

    The Pos value for a given file is a number in the range [base, base+size], where
    base and size are specified when adding the file to the file set via AddFile.

    To create the Pos value for a specific source offset (measured in bytes), first
    add the respective file to the current file set using FileSet.AddFile and then
    call File.Pos(offset) for that file. Given a Pos value p for a specific file set
    fset, the corresponding Position value is obtained by calling fset.Position(p).

    Pos values can be compared directly with the usual comparison operators: If two
    Pos values p and q are in the same file, comparing p and q is equivalent to
    comparing the respective source file offsets. If p and q are in different files,
    p < q is true if the file implied by p was added to the respective file set
    before the file implied by q.

    The zero value for Pos is NoPos; there is no file and line information
    associated with it, and NoPos.IsValid() is false. NoPos is always smaller than
    any other Pos value. The corresponding Position value for NoPos is the zero
    value for Position.

    func (Pos) IsValid

    1. func (p Pos) IsValid()

    IsValid reports whether the position is valid.

    type Position

    1. type Position struct {
    2.     Filename string // filename, if any
    3.     Offset       // offset, starting at 0
    4.     Line     int    // line number, starting at 1
    5.     Column       // column number, starting at 1 (byte count)
    6. }

    Position describes an arbitrary source position including the file, line, and
    column location. A Position is valid if the line number is > 0.

    func (*Position) IsValid

    1. func (pos *Position) IsValid()

    IsValid reports whether the position is valid.

    func (Position) String

    1. func (pos Position) String()

    String returns a string in one of several forms:

    2. line:column         valid position without file name
    4. -                   invalid position without file name

    type Token

    1. type Token int

    Token is the set of lexical tokens of the Go programming language.

    1. const (
    2.     // Special tokens
    3.     ILLEGAL  = iota
    4.     EOF
    5.     COMMENT
    6.  
    7.     // Identifiers and basic type literals
    8.     // (these tokens stand for classes of literals)
    9.     IDENT  // main
    10.     INT    // 12345
    11.     FLOAT  // 123.45
    12.     IMAG   // 123.45i
    13.     CHAR   // 'a'
    14.     STRING // "abc"
    15.  
    16.     // Operators and delimiters
    17.     ADD // +
    18.     SUB // -
    19.     MUL // *
    20.     QUO // /
    21.     REM // %
    22.  
    23.     AND     // &
    24.     OR      // |
    25.     XOR     // ^
    26.     SHL     // <<
    27.     SHR     // >>
    28.     AND_NOT // &^
    29.  
    30.     ADD_ASSIGN // +=
    31.     SUB_ASSIGN // -=
    32.     MUL_ASSIGN // *=
    33.     QUO_ASSIGN // /=
    34.     REM_ASSIGN // %=
    35.  
    36.     AND_ASSIGN     // &=
    37.     OR_ASSIGN      // |=
    38.     XOR_ASSIGN     // ^=
    39.     SHL_ASSIGN     // <<=
    40.     SHR_ASSIGN     // >>=
    41.     AND_NOT_ASSIGN // &^=
    42.  
    43.     LAND  // &&
    44.     LOR   // ||
    45.     INC   // ++
    46.     DEC   // --
    47.  
    48.     EQL    // ==
    49.     LSS    // <
    50.     GTR    // >
    51.     ASSIGN // =
    52.     NOT    // !
    53.  
    54.     NEQ      // !=
    55.     LEQ      // <=
    56.     GEQ      // >=
    57.     DEFINE   // :=
    58.     ELLIPSIS // ...
    59.  
    60.     LPAREN // (
    61.     LBRACK // [
    62.     LBRACE // {
    63.     COMMA  // ,
    64.     PERIOD // .
    65.  
    66.     RPAREN    // )
    67.     RBRACK    // ]
    68.     RBRACE    // }
    69.     SEMICOLON // ;
    70.     COLON     // :
    71.  
    72.     // Keywords
    73.     BREAK
    74.     CASE
    75.     CHAN
    76.     CONST
    77.     CONTINUE
    78.  
    79.     DEFAULT
    80.     DEFER
    81.     ELSE
    82.     FALLTHROUGH
    83.     FOR
    84.  
    85.     FUNC
    86.     GO
    87.     GOTO
    88.     IF
    89.     IMPORT
    90.  
    91.     INTERFACE
    92.     MAP
    93.     PACKAGE
    94.     RANGE
    95.     RETURN
    96.  
    97.     SELECT
    98.     STRUCT
    99.     SWITCH
    100.     TYPE
    101.     VAR
    102. )

    The list of tokens.

    func 

    1. func Lookup(ident ) Token

    Lookup maps an identifier to its keyword token or IDENT (if not a keyword).

    func (Token) 

    1. func (tok ) IsKeyword() bool

    IsKeyword returns true for tokens corresponding to keywords; it returns false
    otherwise.

    func (Token) 

    1. func (tok ) IsLiteral() bool

    IsLiteral returns true for tokens corresponding to identifiers and basic type
    literals; it returns false otherwise.

    func (Token) 

    1. func (tok ) IsOperator() bool

    IsOperator returns true for tokens corresponding to operators and delimiters; it
    returns false otherwise.

    func (Token) 

    1. func (op ) Precedence() int

    Precedence returns the operator precedence of the binary operator op. If op is
    not a binary operator, the result is LowestPrecedence.

    func (Token) 

    1. func (tok ) String() string