Lexical Analysis (Scanner)

Lexical Analysis (Scanner)

1. Role

Lexical analysis is Phase 1: reads source characters and groups them into tokens — the words of the language. Also strips whitespace/comments, counts line numbers, and enters identifiers into the symbol table. Output goes to the parser.

2. Token, Lexeme, Pattern (most confused trio)

Term	Meaning	Example for float pi = 3.14;
Lexeme	actual character sequence in source	pi, 3.14, float
Token	class: <name, attribute>	<id, "pi">, <num, 3.14>, <keyword, float>
Pattern	rule describing lexemes of a token	letter(letter丨digit)* for id

Other tokens: keywords (if), operators (+, <=), separators (;, {), literals.

3. How Scanner Works

source chars → [scanner] → token stream
   ↑               ↓
ch buffer (2 halves, N chars each) + symbol table insert

• Buffering: two N-char halves with sentinel so lookahead rarely needs refill.
• Recognition: longest-match (maximal munch) — <= is one token, not < + =.
• Panic on error: illegal char @ → skip + report illegal character at line n.

4. Worked Example

input:  total = sum + 10;  // add
output: <id,total> <=> <id,sum> <+> <num,10> <;>
comment + spaces dropped; total, sum inserted/looked up in symbol table

5. Exam Points

• Define token vs lexeme vs pattern with example — 5 marks, very frequent.
• Tasks of lexical analyzer + buffering / sentinel — 5 marks, frequent.
• Convert a statement into token stream — 3-5 marks, frequent.