scanner.h File Reference

Definitions used by the token scanner. More...

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Data Structures

struct  KrkToken
 A token from the scanner. More...
 
struct  KrkScanner
 Token scanner state. More...
 

Enumerations

enum  KrkTokenType {
  TOKEN_LEFT_PAREN , TOKEN_RIGHT_PAREN , TOKEN_LEFT_BRACE , TOKEN_RIGHT_BRACE ,
  TOKEN_LEFT_SQUARE , TOKEN_RIGHT_SQUARE , TOKEN_COLON , TOKEN_COMMA ,
  TOKEN_DOT , TOKEN_MINUS , TOKEN_PLUS , TOKEN_SEMICOLON ,
  TOKEN_SOLIDUS , TOKEN_DOUBLE_SOLIDUS , TOKEN_ASTERISK , TOKEN_POW ,
  TOKEN_MODULO , TOKEN_AT , TOKEN_CARET , TOKEN_AMPERSAND ,
  TOKEN_PIPE , TOKEN_TILDE , TOKEN_LEFT_SHIFT , TOKEN_RIGHT_SHIFT ,
  TOKEN_BANG , TOKEN_GREATER , TOKEN_LESS , TOKEN_ARROW ,
  TOKEN_WALRUS , TOKEN_GREATER_EQUAL , TOKEN_LESS_EQUAL , TOKEN_BANG_EQUAL ,
  TOKEN_EQUAL_EQUAL , TOKEN_EQUAL , TOKEN_LSHIFT_EQUAL , TOKEN_RSHIFT_EQUAL ,
  TOKEN_PLUS_EQUAL , TOKEN_MINUS_EQUAL , TOKEN_PLUS_PLUS , TOKEN_MINUS_MINUS ,
  TOKEN_CARET_EQUAL , TOKEN_PIPE_EQUAL , TOKEN_AMP_EQUAL , TOKEN_SOLIDUS_EQUAL ,
  TOKEN_ASTERISK_EQUAL , TOKEN_POW_EQUAL , TOKEN_DSOLIDUS_EQUAL , TOKEN_AT_EQUAL ,
  TOKEN_MODULO_EQUAL , TOKEN_STRING , TOKEN_BIG_STRING , TOKEN_NUMBER ,
  TOKEN_IDENTIFIER , TOKEN_AND , TOKEN_CLASS , TOKEN_DEF ,
  TOKEN_DEL , TOKEN_ELSE , TOKEN_FALSE , TOKEN_FINALLY ,
  TOKEN_FOR , TOKEN_IF , TOKEN_IMPORT , TOKEN_IN ,
  TOKEN_IS , TOKEN_LET , TOKEN_NONE , TOKEN_NOT ,
  TOKEN_OR , TOKEN_ELIF , TOKEN_PASS , TOKEN_RETURN ,
  TOKEN_SUPER , TOKEN_TRUE , TOKEN_WHILE , TOKEN_TRY ,
  TOKEN_EXCEPT , TOKEN_RAISE , TOKEN_BREAK , TOKEN_CONTINUE ,
  TOKEN_AS , TOKEN_FROM , TOKEN_LAMBDA , TOKEN_ASSERT ,
  TOKEN_YIELD , TOKEN_ASYNC , TOKEN_AWAIT , TOKEN_WITH ,
  TOKEN_PREFIX_B , TOKEN_PREFIX_F , TOKEN_PREFIX_R , TOKEN_INDENTATION ,
  TOKEN_EOL , TOKEN_RETRY , TOKEN_ERROR , TOKEN_EOF ,
  TOKEN_ELLIPSIS
}
 

Functions

KrkScanner krk_initScanner (const char *src)
 Initialize the compiler to scan tokens from 'src'.
 
KrkToken krk_scanToken (KrkScanner *)
 Read the next token from the scanner.
 
void krk_ungetToken (KrkScanner *, KrkToken token)
 Push a token back to the scanner to be reprocessed. More...
 
void krk_rewindScanner (KrkScanner *, KrkScanner to)
 Rewind the scanner to a previous state. More...
 
KrkScanner krk_tellScanner (KrkScanner *)
 Retreive a copy of the current scanner state. More...
 

Detailed Description

Definitions used by the token scanner.

Definition in file scanner.h.

Function Documentation

◆ krk_rewindScanner()

void krk_rewindScanner ( KrkScanner scanner,
KrkScanner  to 
)

Rewind the scanner to a previous state.

Resets the current scanner to the state in 'to'. Used by the compiler to implement comprehensions, which would otherwise not be possible in a single-pass compiler.

Definition at line 291 of file scanner.c.

◆ krk_tellScanner()

KrkScanner krk_tellScanner ( KrkScanner scanner)

Retreive a copy of the current scanner state.

Used with krk_rewindScanner() to implement rescanning for comprehensions.

Definition at line 287 of file scanner.c.

◆ krk_ungetToken()

void krk_ungetToken ( KrkScanner scanner,
KrkToken  token 
)

Push a token back to the scanner to be reprocessed.

Pushes a previously-scanned token back to the scanner. Used to implement small backtracking operations at the end of block constructs like 'if' and 'try'.

Definition at line 279 of file scanner.c.