scanner.h File Reference

Definitions used by the token scanner. More...

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Data Structures
struct	KrkToken
	A token from the scanner. More...
struct	KrkScanner
	Token scanner state. More...

Enumerations
enum	KrkTokenType {   TOKEN_LEFT_PAREN , TOKEN_RIGHT_PAREN , TOKEN_LEFT_BRACE , TOKEN_RIGHT_BRACE ,   TOKEN_LEFT_SQUARE , TOKEN_RIGHT_SQUARE , TOKEN_COLON , TOKEN_COMMA ,   TOKEN_DOT , TOKEN_MINUS , TOKEN_PLUS , TOKEN_SEMICOLON ,   TOKEN_SOLIDUS , TOKEN_DOUBLE_SOLIDUS , TOKEN_ASTERISK , TOKEN_POW ,   TOKEN_MODULO , TOKEN_AT , TOKEN_CARET , TOKEN_AMPERSAND ,   TOKEN_PIPE , TOKEN_TILDE , TOKEN_LEFT_SHIFT , TOKEN_RIGHT_SHIFT ,   TOKEN_BANG , TOKEN_GREATER , TOKEN_LESS , TOKEN_ARROW ,   TOKEN_WALRUS , TOKEN_GREATER_EQUAL , TOKEN_LESS_EQUAL , TOKEN_BANG_EQUAL ,   TOKEN_EQUAL_EQUAL , TOKEN_EQUAL , TOKEN_LSHIFT_EQUAL , TOKEN_RSHIFT_EQUAL ,   TOKEN_PLUS_EQUAL , TOKEN_MINUS_EQUAL , TOKEN_PLUS_PLUS , TOKEN_MINUS_MINUS ,   TOKEN_CARET_EQUAL , TOKEN_PIPE_EQUAL , TOKEN_AMP_EQUAL , TOKEN_SOLIDUS_EQUAL ,   TOKEN_ASTERISK_EQUAL , TOKEN_POW_EQUAL , TOKEN_DSOLIDUS_EQUAL , TOKEN_AT_EQUAL ,   TOKEN_MODULO_EQUAL , TOKEN_STRING , TOKEN_BIG_STRING , TOKEN_NUMBER ,   TOKEN_IDENTIFIER , TOKEN_AND , TOKEN_CLASS , TOKEN_DEF ,   TOKEN_DEL , TOKEN_ELSE , TOKEN_FALSE , TOKEN_FINALLY ,   TOKEN_FOR , TOKEN_IF , TOKEN_IMPORT , TOKEN_IN ,   TOKEN_IS , TOKEN_LET , TOKEN_NONE , TOKEN_NOT ,   TOKEN_OR , TOKEN_ELIF , TOKEN_PASS , TOKEN_RETURN ,   TOKEN_SUPER , TOKEN_TRUE , TOKEN_WHILE , TOKEN_TRY ,   TOKEN_EXCEPT , TOKEN_RAISE , TOKEN_BREAK , TOKEN_CONTINUE ,   TOKEN_AS , TOKEN_FROM , TOKEN_LAMBDA , TOKEN_ASSERT ,   TOKEN_YIELD , TOKEN_ASYNC , TOKEN_AWAIT , TOKEN_WITH ,   TOKEN_PREFIX_B , TOKEN_PREFIX_F , TOKEN_PREFIX_R , TOKEN_INDENTATION ,   TOKEN_EOL , TOKEN_RETRY , TOKEN_ERROR , TOKEN_EOF ,   TOKEN_ELLIPSIS }

Functions
KrkScanner	krk_initScanner (const char *src)
	Initialize the compiler to scan tokens from 'src'.
KrkToken	krk_scanToken (KrkScanner *)
	Read the next token from the scanner.
void	krk_ungetToken (KrkScanner *, KrkToken token)
	Push a token back to the scanner to be reprocessed. More...
void	krk_rewindScanner (KrkScanner *, KrkScanner to)
	Rewind the scanner to a previous state. More...
KrkScanner	krk_tellScanner (KrkScanner *)
	Retreive a copy of the current scanner state. More...

Detailed Description

Definitions used by the token scanner.

Definition in file scanner.h.

Function Documentation

krk_rewindScanner()

void krk_rewindScanner	(	KrkScanner *	scanner,
		KrkScanner	to
	)

Rewind the scanner to a previous state.

Resets the current scanner to the state in 'to'. Used by the compiler to implement comprehensions, which would otherwise not be possible in a single-pass compiler.

Definition at line 297 of file scanner.c.

krk_tellScanner()

KrkScanner krk_tellScanner

(

KrkScanner *

scanner

)

Retreive a copy of the current scanner state.

Used with krk_rewindScanner() to implement rescanning for comprehensions.

Definition at line 293 of file scanner.c.

krk_ungetToken()

void krk_ungetToken	(	KrkScanner *	scanner,
		KrkToken	token
	)

Push a token back to the scanner to be reprocessed.

Pushes a previously-scanned token back to the scanner. Used to implement small backtracking operations at the end of block constructs like 'if' and 'try'.

Definition at line 285 of file scanner.c.