Unit 2, Part 2 of 2 · CEUC302 Theory of Computation

The theory that builds a real compiler stage

Chapter 2 built machines and proved one non-regularity result the Myhill–Nerode way. This chapter finishes the toolkit — how to combine regular languages without starting from scratch (closure properties), and the official, examinable way to prove a language isn't regular (the Pumping Lemma) — then spends it immediately on the reason this unit exists at all: turning raw characters into the token stream a parser expects. Leven and Lends-ab make one last appearance each; a small toy scanner carries the second half of the chapter.

Leven Lends-ab aab

Closure properties

No new machine built from nothing

If you already have a machine for each piece, ∩, ∪, and complement are mechanical.

Section 3.1 builds the actual product machine for Leven ∩ Lends-ab — "aab" sits in the shaded overlap because it has an even a-count and ends in ab.

Colour key · same meaning as Chapters 1–2 teal — accepting / a completed, valid token rose — rejecting / a broken pumped string violet — the second language or machine in a combination amber — whatever is in focus right now (the piece being pumped, the current match)
Before you start

This chapter assumes Chapter 2 throughout: Meven's states and transitions, the Lends-ab NFA and its subset-construction DFA, and the general idea that a DFA's δ is always total. Section 3.2 also leans on Chapter 1's pigeonhole principle and Chapter 2's Myhill–Nerode proof for Leq — not to repeat them, but to contrast a third method against the same target.

3.1 · Closure Properties

Combining machines you already have

∩, ∪, and complement never need a new machine built from scratch — they're mechanical operations on the machines you already built.

Chapter 1 previewed closure as a definition, with finite languages and union as the toy example. Now that real machines exist — Meven and the Lends-ab DFA — the question gets teeth: given two regular languages, each with its own machine, can you build a machine for their intersection without redesigning anything?

The intuition, no symbols yet

Run both machines at once, side by side, feeding each the same input. At every moment, the pair of current states — one from each machine — tells you everything both machines currently know. A pair is like a single combined state; there are only (states in M1) × (states in M2) possible pairs, still a finite number. Whether that pair counts as "accepting" is just a choice you make afterward: both halves accepting means the pair is in for intersection; either half accepting means the pair is in for union.

The formal treatment

Regular languages are closed under union, intersection, complement, concatenation, and star — meaning each operation, applied to regular language(s), always produces another regular language. Each closure comes with an explicit construction:

closure constructions at a glance
operationconstruction
complement (L̄)Take a DFA for L. Swap F to Q−F. Nothing else changes.
intersection (L1∩L2)Product construction: states Q1×Q2, δ((p,q),a)=(δ1(p,a),δ2(q,a)), accept iff both p∈F1 and q∈F2.
union (L1∪L2)Identical product construction; accept iff either p∈F1 or q∈F2.
concatenation (L1L2)NFA gluing: every accepting state of M1 gets an ε-edge into M2's start (Section 2.4's construction, one operator at a time).
star (L1*)NFA gluing: ε-edges loop each accepting state back to the start, plus a bypass so zero repetitions is allowed.
Don't confuse these
  • Complement needs a total DFA, not an NFA. Flipping accept/non-accept on an NFA does not give the complement — a string with an empty active set is already "not accepted," and flipping wouldn't magically accept it (there's no state left to flip into acceptance). Complement only works cleanly on a DFA, and only because δ is guaranteed total (Section 2.1): every string leads somewhere, so swapping F is a genuine, complete flip of the verdict.
  • Product construction can look bigger than it needs to be. |Q1|×|Q2| states are built, but many might be unreachable or later merge under minimization (Section 2.2). Build first, minimize after, if a smaller machine matters.

Worked example · Leven ∩ Lends-ab

Worked example
product construction, 2 × 3 = 6 states
Build the product machine for Leven ∩ Lends-ab and identify its accepting states.
Step 1 · statesQ = {q0,q1} × {D0,D1,D2} — six pairs: (q0,D0), (q0,D1), (q0,D2), (q1,D0), (q1,D1), (q1,D2).
Step 2 · start(q0,D0) — both machines' start states, paired.
Step 3 · transitionsMove both halves at once: δ((qi,Dj),a) = (δeven(qi,a), δendsab(Dj,a)). E.g. from (q0,D0) on a: q0→q1 and D0→D1, giving (q1,D1).
Step 4 · accept, intersection ruleOnly (q0,D2) qualifies — q0 is Leven's only accept state, D2 is Lends-ab's only accept state, and intersection needs both.
Answer — six reachable states, one accepting: (q0,D2). Verified against direct definition for every string up to length 11: zero mismatches. Shortest accepted strings: "aab", "abab", "baab."

Same six states, three different languages

flip the accepting rule, get ∩, ∪, or the complement of ∩
fig 3.1 · interactive
q​\D D0 D1 D2 q0 q1 q0,D0 q0,D1 q0,D2 q1,D0 q1,D1 q1,D2
Teal cells are accepting under Leven ∩ Lends-ab: only (q0,D2) — even a-count AND ends in ab.

Practice · 3.1

Seven problems
direct → variation → interpretation → synthesis

3.1.1 · From (q1,D2), what state does the product machine move to on symbol b?

Show solution

δeven(q1,b)=q1; δendsab(D2,b)=D0. New state: (q1,D0).

3.1.2 · Complement Meven directly (state which states become accepting). What language results?

Show solution

Swap F from {q0} to {q1}. The resulting language is exactly Lodd — the complement of "even a-count" is "odd a-count" (over the full alphabet {a,b}, with no other strings excluded).

3.1.3 · Variation: build (on paper) the accept-state rule for Leven ∪ Lends-ab, using the same six product states.

Show solution

Accept iff q-part is q0 OR D-part is D2: (q0,D0), (q0,D1), (q0,D2), (q1,D2) — four of the six states, matching fig 3.1's "∪" toggle.

3.1.4 · Variation: is Leven − Lends-ab (strings in Leven but not Lends-ab) regular? If so, describe its accept rule on the same six states without building anything new.

Show solution

Yes — set difference is intersection with a complement, Leven ∩ (Lends-ab)̄, and both operations preserve regularity, so the combination does too. Accept rule: q-part is q0 AND D-part is not D2: (q0,D0), (q0,D1).

3.1.5 · Interpretation: a classmate complements the Lends-ab NFA (Chapter 2) by flipping which of p0,p1,p2 count as accepting, and claims the result recognises "doesn't end in ab." Check a string that breaks this.

Show solution

Try "ba": the NFA's active set ends at {p0,p1} wait — check directly: read b: {p0}; read a: {p0,p1}. Flipping accepting states to {p0,p1} (everything except the old p2) makes this flipped-NFA accept "ba" via p0 or p1 being active — correctly, since "ba" doesn't end in ab. Now try "ab" itself: active set ends at {p0,p2}; under the flipped rule only p0 or p1 count, and p0 IS active, so the flipped machine also accepts "ab" — incorrectly, since "ab" does end in ab and should be rejected by a true complement. The flip breaks precisely because "some thread is in a flipped-accepting state" isn't the same condition as "every thread failed the original acceptance" — confirming the trap box: complement needs a DFA.

3.1.6 · Interpretation: why doesn't building the 6-state product machine ever risk landing outside the regular languages, no matter which two DFAs you started from?

Show solution

The construction only ever produces a finite state set (bounded by |Q1|×|Q2|) with a fully total δ built from two already-total functions — which is, by definition, exactly another DFA. Since every DFA's language is regular, the result is automatically regular; there's no step in the recipe that could produce anything else.

3.1.7 · Synthesis (challenge) · Using closure under complement and intersection, express Leven ∪ Lends-ab using only ∩ and complement (De Morgan's law), and say why this proves union doesn't need its own separate closure proof.

Show solution

Leven ∪ Lends-ab = complement of ( (Leven)̄ ∩ (Lends-ab)̄ ) — De Morgan's law translated directly into languages. Since complement and intersection are each already proven closed, chaining them (complement, then intersection, then complement again) never leaves the regular languages at any step — union closure is technically a free corollary of the other two, though this chapter still gave it its own direct product-construction proof since that's more useful for actually building the machine.

3.2 · The Pumping Lemma

The proof examiners actually want to see

Chapter 1 sketched why Leq can't be regular. Chapter 2 proved it, formally, via Myhill–Nerode. This section proves it a third way — the way most syllabi mean when they just say "use the Pumping Lemma."

Every regular language has a machine with some fixed number of states, say p. Feed that machine a string longer than p symbols, and by the pigeonhole principle (Chapter 1, Section 1.4) it must revisit some state while reading it. Whatever the machine read between the two visits to that repeated state forms a loop — and loops can be gone around any number of times, including zero, without changing where the machine ends up. That's the entire engine behind the Pumping Lemma; everything below just makes it precise enough to write a proof with.

The intuition, no symbols yet

If a long-enough string is accepted, some middle chunk of it must be "loop-shaped" — cut it out, or paste in a second copy, and the machine, which only ever remembers its current state, cannot tell the difference. It will make the exact same decision either way. If you can find even one way of looping that breaks membership, the assumption that a machine existed at all must have been wrong.

The formal treatment

the Pumping Lemma for regular languagesIf L is regular, there exists an integer p ≥ 1 (the pumping length) such that every string w ∈ L with |w| ≥ p can be written w = xyz, satisfying all of: (1) |y| > 0 the pumped piece is never empty (2) |xy| ≤ p the split happens within the first p symbols (3) for every i ≥ 0, xyⁱz ∈ L pump it any number of times, including zero — still in L

Used directly, this only ever confirms something about a language already known to be regular. Used as a contrapositive, it becomes a disproof machine: to show L is not regular, show that for every candidate p, some string w∈L with |w|≥p has the property that every valid split (satisfying (1) and (2)) fails condition (3) for at least one i. One surviving split anywhere ruins the proof; the argument needs to close off every possible split at once.

Worked example · Leq is not regular, formally

Worked example
the canonical proof, for general p
Prove Leq = {anbn : n≥0} is not regular using the Pumping Lemma.
Step 1 · suppose notAssume Leq is regular. Then some pumping length p exists.
Step 2 · choose wPick w = apbp — it's in Leq and has length 2p ≥ p, so the lemma must apply to it.
Step 3 · every valid split looks the sameAny split w=xyz with |xy|≤p falls entirely within the first p symbols — all a's. So y consists only of a's: y = ak for some k≥1 (condition (1) forces k≥1).
Step 4 · pump downTake i=0: xy0z = xz = ap−kbp. Since k≥1, p−k ≠ p — unequal a's and b's.
Step 5 · contradictionap−kbp ∉ Leq, contradicting condition (3), which required xyiz ∈ Leq for every i ≥ 0, including i=0.
Answer — no valid split can survive pumping, for any p. The assumption breaks. Leq is not regular. (Numerically checked at p=3: all 6 valid splits fail under pumping — matching this general argument exactly.)
the split, made visible
fig 3.2
w  =  a…a a…a a…a b…b
x        y                z
↓ pump i=0, delete y ↓
a…a a…a b…b  — fewer a's than b's now
y is trapped inside the first p symbols by condition (2), and w's first p symbols are all a's — so y can only ever be more a's, never a mix that could keep the count balanced after pumping.
Don't confuse these
  • One split surviving ruins the whole proof. The disproof needs every valid (x,y,z) to fail — checking a single convenient split and finding a contradiction proves nothing, since a different split might not break.
  • Forgetting |xy|≤p loses the entire argument. Without this constraint, y could be chosen anywhere in w, including straddling the a's and b's boundary — and such a y might pump safely. The constraint is what forces y into "all a's" territory for Leq's proof to work at all.
  • Surviving the Pumping Lemma never proves regularity. The lemma is a necessary condition of being regular, not sufficient — some non-regular languages can still admit a pumping length that happens not to produce a contradiction through this particular test. A clean pumping-lemma failure proves non-regularity; a clean pumping-lemma "pass" proves nothing on its own.

Practice · 3.2

Seven problems
direct → variation → interpretation → synthesis

3.2.1 · State, from memory, the three conditions a split w=xyz must satisfy for the Pumping Lemma to apply.

Show solution

(1) |y| > 0 (never empty), (2) |xy| ≤ p (split happens early), (3) xyiz ∈ L for every i ≥ 0 (pump any number of times, including zero, and stay in the language).

3.2.2 · For p=4, w=a4b4, pick one valid split (x,y,z) with y=aa and show pumping to i=2 breaks Leq membership.

Show solution

x=aa, y=aa, z=b4 (check: |xy|=4≤p=4 ✓, |y|=2>0 ✓). At i=2: xy2z = aa·aaaa·bbbb = a6b4 — 6≠4, not in Leq. Contradiction confirmed for this split (and, per the worked example, for every other valid split too).

3.2.3 · Variation: adapt the worked proof to show L3eq = {anbncn} is not regular via the Pumping Lemma (Chapter 2 did this via Myhill–Nerode; now do it this way).

Show solution

Pick w=apbpcp. Any split with |xy|≤p lands entirely in the leading a's, so y=ak, k≥1. Pumping to i=0 gives ap−kbpcp, with a-count ≠ b-count — not in L3eq. Every split fails identically. Not regular. (Numerically checked at p=3: all 6 splits fail.)

3.2.4 · Variation: use the Pumping Lemma to show L> = {aibj : i>j} is not regular.

Show solution

Pick w=ap+1bp (i=p+1>j=p, so w∈L>). Any split with |xy|≤p lands in the leading a's (there are p+1 of them), so y=ak, k≥1. Pumping to i=0: ap+1−kbp. Since k≥1, p+1−k≤p, so the a-count no longer exceeds the b-count — not in L>. Not regular. (Verified at p=4: all 10 splits fail.)

3.2.5 · Interpretation: a proof picks w=apbp, chooses the specific split x=ε, y=ap, z=bp, shows pumping breaks it, and declares Leq not regular. What's missing?

Show solution

The proof only checked one split (admittedly, per condition (2), it's the only shape a split can take here since the first p symbols are all a's — but the write-up needs to say that every valid split has y consisting only of a's, and argue that generally, rather than silently picking one convenient case and hoping the grader doesn't notice the gap).

3.2.6 · Interpretation: does finding some string and split that survives pumping (stays in L for every i) prove L is regular?

Show solution

No. The Pumping Lemma is a necessary condition of regularity (every regular language satisfies it), not a sufficient one. A non-regular language could still happen to have some strings admit a pumping length and a split that survives — passing this one check is weak evidence at best, never a proof. Proving regularity means exhibiting an actual DFA, NFA, regular grammar, or RE (Chapters 1–2), not surviving a pumping attempt.

3.2.7 · Synthesis (challenge) · Chapters 1–3 have now shown Leq is not regular three ways: an informal pigeonhole sketch, a formal Myhill–Nerode argument, and this section's Pumping Lemma proof. What's the one idea all three genuinely share, and what's actually different between the Myhill–Nerode and Pumping Lemma versions specifically?

Show solution

Shared idea: a machine with finitely many states can't keep an unboundedly large count distinct forever — something must eventually repeat or collide. The difference between the two formal versions is where the argument starts: Myhill–Nerode works directly on the language (comparing strings ai, aj for future-behaviour differences, no machine needed at all), while the Pumping Lemma works by first assuming a machine/pumping-length exists and deriving a contradiction from how it would have to behave. Same destination, opposite starting assumptions.

3.3 · Lexical Analysis

Where all of this was always heading

A compiler's first stage turns raw characters into a stream of meaningful chunks. Every idea from Chapters 1–3 was quietly built for exactly this.

A token is a category — IDENTIFIER, NUMBER, KEYWORD. A lexeme is the actual text matched — "counter", "42", "if". A pattern is the rule deciding which lexemes belong to which token, and by now patterns are old news: they're regular expressions, exactly as defined in Section 2.4. The whole first stage of a compiler, called the lexical analyser or scanner, is a machine built from those patterns exactly the way Chapters 1–3 already know how to build machines.

The intuition, no symbols yet

Line up every token's pattern side by side and run them all at once against the input, the same parallel-threads idea Section 2.3 used for a single NFA. Whichever pattern (or patterns) are still alive tells you what the text read so far could still turn out to be. The one rule that decides when to stop reading and commit to a token: always take the longest lexeme some pattern will still accept, never the first one that happens to match.

The formal treatment

A lexical specification is a list of (token name, pattern) pairs. Building the scanner: take the RE→FA construction (Section 2.4) for each pattern separately, union them all under one shared start state exactly as Kleene's theorem's union case describes, then run subset construction and minimize (Sections 2.3, 2.2) — producing one DFA whose accepting states are each tagged with which token they complete.

Character classes keep the machine small

Real alphabets are large (26 letters, 10 digits, punctuation) but most patterns treat whole groups of characters identically. Section 3.4's machine uses just two abstract symbols, L (any letter) and D (any digit), standing in for 36 real characters — exactly what a real lexer generator does internally, not a simplification made only for this book.

the running token spec, using L (letter) and D (digit)IDENTIFIER: L(L+D)* a letter, then any mix of letters and digits NUMBER: D+ one or more digits LT: < the single character < LE: <= the two characters <=

Worked example · why the scanner doesn't stop at "if"

Worked example
maximal munch, one decision point
Scanning input "if12<=x" from position 0, the machine reaches an accepting state after "if" (a valid IDENTIFIER already). Should it stop there?
Step 1After "if," the machine is in an accepting state for IDENTIFIER — a valid token could be emitted right now.
Step 2But the next character, '1', is a digit — and IDENTIFIER's pattern L(L+D)* explicitly allows digits after the first letter. The machine still has a live transition on '1'.
Step 3Longest match says: keep going whenever a live transition exists, regardless of having already passed an accepting state. Continue through '1', '2' — still IDENTIFIER, still accepting, still extendable.
Step 4At '<', there's no transition from the IDENTIFIER state at all. Now stop, and emit the longest lexeme that was ever accepting: "if12."
Answer — the token is IDENTIFIER "if12," a single token, not IDENTIFIER "if" followed by NUMBER "12." Stopping at the first accepting state (a common instinct) would have wrongly split one identifier into two tokens.
first-match instinct vs. the actual longest-match rule
ruleresult on "if12"correct?
stop at first accepting stateIDENTIFIER "if", then restart at "12" → NUMBER "12"wrong
longest match (maximal munch)IDENTIFIER "if12", one tokencorrect
Keywords: a table lookup, not a separate machine path

"if" matches IDENTIFIER's pattern just fine — i and f are both letters. Rather than building competing literal-string machinery inside the DFA, real scanners let the DFA match "if" as an ordinary IDENTIFIER lexeme, then check the result against a small fixed table of reserved words. Only "if," "then," "else," and friends get relabelled KEYWORD after the fact; every other identifier passes through unchanged. Simpler than it sounds, and exactly how production lexers do it.

Don't confuse these
  • An accepting state mid-scan is not a stop signal. It only means "a valid token ends here if nothing longer is available" — keep extending while any transition still exists.
  • Ties between different token types need a priority rule, separately from length. If two different patterns could both match the exact same longest lexeme, list order (or an explicit priority) breaks the tie — length alone doesn't always settle it.
  • Whitespace and comments are usually tokens too, just discarded ones. They get matched and recognised by the scanner like anything else; they're simply never handed to the parser afterward.

Practice · 3.3

Seven problems
direct → variation → interpretation → synthesis

3.3.1 · Using IDENTIFIER = L(L+D)*, is "9x" a valid IDENTIFIER lexeme? Why or why not?

Show solution

No. The pattern requires the first symbol to be L (a letter); "9x" starts with a digit. "9x" would instead need to be scanned as NUMBER "9" followed by a separate IDENTIFIER "x" (two tokens).

3.3.2 · What is the longest-match token for the input "abc123"?

Show solution

IDENTIFIER "abc123" — one token. The pattern L(L+D)* happily consumes letters and digits in any mix after the first letter, and there's no character afterward to force a stop.

3.3.3 · Variation: add a token BOOL with pattern (true+false). Does this create any longest-match conflict with IDENTIFIER for the lexeme "true"?

Show solution

Both patterns match "true" at exactly the same length (4 characters) — a genuine tie, not resolved by "longest wins" since they're equally long. This needs an explicit priority rule (list BOOL's pattern before IDENTIFIER's, so BOOL wins ties) — the same style of fix as the keyword-table approach, just applied inside the DFA's tagging instead of after it.

3.3.4 · Variation: scan "12if" against the running token spec (IDENTIFIER, NUMBER, LT, LE). What token stream results?

Show solution

NUMBER's pattern D+ matches "12" but has no transition on a letter, so it stops there: NUMBER "12", then restart at "if": IDENTIFIER "if" (later relabelled KEYWORD by the reserved-word table). Two tokens, in that order.

3.3.5 · Interpretation: a classmate builds a separate 2-state machine just for recognising the literal word "if," wired in parallel with the IDENTIFIER machine. Is this wrong?

Show solution

Not wrong exactly, but needlessly complicated — and it reintroduces the BOOL-style tie problem from 3.3.3 for every keyword, needing an explicit priority rule for each one. The table-lookup approach sidesteps this entirely: let IDENTIFIER's machine match "if" like any other identifier, and relabel afterward. Fewer machine states, one small table, same result.

3.3.6 · Interpretation: why does the longest-match rule specifically need the machine to keep track of "the last position where I was in an accepting state," rather than just the current state?

Show solution

Because the machine might walk past an accepting state into further non-accepting territory that still has live transitions (as "if" → "if1" does, briefly passing through non-obviously-different accepting states) before eventually dying. Only by remembering the last accepting position can the scanner correctly "back off" to the longest valid lexeme once it finally gets stuck, rather than being stuck with only whatever the current (possibly non-accepting, dead) state says.

3.3.7 · Synthesis (challenge) · Explain, using Section 2.4's Kleene's-theorem construction by name, why a lexer generator can accept a plain-text file of (name, regex) pairs and mechanically produce a working scanner with no further human design work.

Show solution

Each regex, by Kleene's theorem, has a mechanically constructible equivalent NFA (RE→FA, Section 2.4). Unioning them under one shared start — itself just the union construction from that same theorem — produces a single NFA for "any of these patterns." Subset construction (Section 2.3) turns it into a DFA, minimization (Section 2.2) shrinks it, and tagging each accepting state with its originating token name finishes the job. Every step is an algorithm already covered, chained together; nothing requires a human to think up new automaton structure by hand, which is exactly why tools like flex can generate a scanner in milliseconds from a spec file.

3.4 · DFA-based Scanners

Building the machine and running it

One combined DFA, one scanning algorithm, one full trace — everything from Chapters 1–3 cashed in at once.

Section 3.3 named the pieces. Here they become one concrete machine: a single DFA covering IDENTIFIER, NUMBER, LT, and LE at once, each accepting state tagged with which token it completes, plus the exact algorithm that runs it correctly.

The intuition, no symbols yet

Walk forward through the input, one character at a time, always taking a live transition when one exists. Every time you land on an accepting state, jot down "a valid token could end right here" without stopping. The moment no transition exists for the next character, stop walking, rewind to the last spot you jotted down, and cut the token there. Then start over from that cut point for the next token.

The formal treatment · the combined machine

one DFA, four token types (dead-state arrows omitted for clarity)
fig 3.4
start S I N LT LE L D < L, D D =
I, N, LT, and LE are all accepting (double-circled) — tagged IDENTIFIER, NUMBER, LT, and LE respectively. Every arrow not drawn (e.g. I on '<', LT on 'D') leads to an implicit dead state: no live transition, extension stops.
the maximal-munch scanning algorithmpos = 0 while pos < length(input): state = S; lastAcceptPos = none; start = pos while a transition exists from state on input[pos]: state = δ(state, input[pos]); pos = pos + 1 if state is accepting: lastAcceptPos = pos; lastAcceptState = state if lastAcceptPos is none: report LEXICAL ERROR at position start; stop emit token (tag of lastAcceptState, input[start .. lastAcceptPos]) pos = lastAcceptPos restart the NEXT token from here, not from where we got stuck

Worked example · the full trace

Worked example
scanning "if12<=x" end to end
Run the algorithm above on "if12<=x" and report the full token stream.
Token 1, start pos 0S→i(L)I→f(L)I→1(D)I→2(D)I. Next char '<': no transition from I. Back off to lastAcceptPos (after "if12", since I stayed accepting the whole way). Emit IDENTIFIER "if12". pos→4.
Token 2, start pos 4S→<LT (accepting) →=LE (accepting, longer). Next char 'x': no transition from LE. Back off to lastAcceptPos (after "<="). Emit LE "<=". pos→6.
Token 3, start pos 6S→x(L)I (accepting). No characters remain. Emit IDENTIFIER "x". pos→7 = length. Done.
Answer — IDENTIFIER "if12", LE "<=", IDENTIFIER "x". Matches Section 3.3's prediction that "if12" survives as one token, and cleanly resolves the LT-vs-LE ambiguity via longest match.

Run your own input

the scanner, one character at a time
fig 3.4b · interactive
Type letters, digits, < and = only, then press "load & reset" and step through it.
Don't confuse these
  • Getting stuck is not the same as an error. Getting stuck with a recorded lastAcceptPos just means "back off and emit" — perfectly normal, happens after every single token. A real lexical error only occurs when the machine gets stuck with no accepting state ever reached since the current token's start.
  • The position after backing off is not where the machine got stuck. It's the last place the machine was accepting, which is usually earlier. Reporting the stuck position as the token boundary is a common, specific mistake.

Practice · 3.4

Seven problems
direct → variation → interpretation → synthesis

3.4.1 · Trace the algorithm on "x<y" and report the token stream.

Show solution

S→xI (accept). No transition on '<' from I — emit IDENTIFIER "x", pos→1. S→<LT (accept). No transition on 'y' from LT — emit LT "<", pos→2. S→yI (accept), end of input — emit IDENTIFIER "y". Stream: IDENTIFIER "x", LT "<", IDENTIFIER "y".

3.4.2 · Trace the algorithm on "12ab" and report the token stream.

Show solution

S→1N→2N (accept both steps). No transition on 'a' from N — emit NUMBER "12", pos→2. S→aI→bI, end of input — emit IDENTIFIER "ab". Stream: NUMBER "12", IDENTIFIER "ab".

3.4.3 · Variation: what happens on input "12#34"? Report the exact point of failure.

Show solution

NUMBER "12" emits normally (pos→2). Starting fresh at '#': S has no transition on '#' at all (not L, D, <, or =), so lastAcceptPos is never set even once — LEXICAL ERROR at position 2, the '#'. (Verified by direct simulation.)

3.4.4 · Variation: add a token ASSIGN with pattern = (a single equals sign) to the spec. Which existing state's transitions need to change, and how?

Show solution

Only S needs a new transition: currently δ(S,'=')=DEAD; change it to a new accepting state ASSIGN with no further outgoing transitions of its own (mirroring how LT was built before LE existed). Nothing about I, N, LT, or LE changes — adding a token only ever adds machinery reachable from S, consistent with how the union construction (Section 2.4) works.

3.4.5 · Interpretation: a student's trace on "if12<=x" reports the first token as IDENTIFIER "if" instead of "if12." What mistake did they make?

Show solution

They stopped extending at the first accepting state instead of continuing while transitions remained live — the first-match mistake Section 3.3 warned about, not the correct longest-match/maximal-munch behaviour the algorithm actually specifies.

3.4.6 · Interpretation: why does the algorithm reset to state S (not wherever it got stuck) at the start of every new token?

Show solution

Each token is scanned independently — the machine has no memory of "IDENTIFIER" vs "NUMBER" context once a token boundary is cut; every new attempt starts from the same blank slate S, exactly as if scanning a brand-new, shorter input starting at that position. This is what makes the combined DFA reusable token after token without rebuilding anything.

3.4.7 · Synthesis (challenge) · The combined DFA has 5 visible states (S, I, N, LT, LE) plus an implicit dead state. Relate this state count to the closure-property product construction of Section 3.1 — is this DFA "the union of four machines" in the same product-construction sense?

Show solution

No — this is the union-via-NFA-gluing-then-subset-construction route (Section 2.4's Kleene construction), not the product construction. Product construction (Section 3.1) tracks all component machines' states simultaneously as a tuple, growing multiplicatively (|Q1|×|Q2|×…); this scanner instead merges the token patterns into one shared exploration from a single start, growing more like a sum than a product, precisely because at most one "kind" of token is ever being attempted at a time from S. Both routes are valid closure-preserving constructions for union; they just answer different design needs.

Closing

Cheat sheet

One line per idea. If a line doesn't ring a bell, that section needs a re-read before Chapter 4.

Closure properties · 3.1

complement — flip F on a total DFA only, never an NFA

product construction — Q1×Q2, move both halves at once

intersection = AND both accept; union = OR either accepts

concatenation, star — NFA gluing via ε-edges (Section 2.4)

Pumping Lemma · 3.2

w=xyz, |y|>0, |xy|≤p, xyiz∈L for all i≥0

disproof: some w, every valid split fails for some i

necessary, not sufficient — passing proves nothing about regularity

one surviving split kills the whole disproof attempt

Lexical analysis · 3.3

token = category, lexeme = matched text, pattern = regex

longest match / maximal munch — never stop at the first accepting state

keywords: match as IDENTIFIER, relabel via reserved-word table lookup

character classes (L, D) keep the machine small

DFA-based scanners · 3.4

remember lastAcceptPos while extending; back off to it when stuck

no lastAcceptPos ever set ⇒ LEXICAL ERROR, not a normal token boundary

restart from the backed-off position, fresh, at state S every time

whole pipeline = RE→NFA (2.4) → DFA (2.3) → minimize (2.2) → tag & scan

Closing · mixed self-test

Ten questions, no section labels

Exams don't tell you which lecture a question came from. Neither does this.

ST1 · A product construction combines a 3-state DFA and a 4-state DFA. How many product states result?

Show solution

12 (3×4) — possibly fewer reachable in practice, but the construction itself always builds the full |Q1|×|Q2| set before any pruning.

ST2 · True or false: flipping which states are accepting on an NFA produces the complement of its language.

Show solution

False. Complement requires a total DFA; an NFA's "some thread accepts" condition doesn't invert cleanly into "some thread accepts" for the complement (Section 3.1's worked trap on "ab").

ST3 · Why does the Pumping Lemma require |xy| ≤ p specifically?

Show solution

It forces the split to happen within the first p symbols of w — which is exactly what pins y down to a predictable shape (e.g., "all a's" for w=apbp) and makes the contradiction constructible. Without it, y could be chosen anywhere, including a mix that might survive pumping harmlessly.

ST4 · If a string and split survive every pumping attempt, is the language proven regular?

Show solution

No. The lemma is necessary, not sufficient. Surviving pumping on some strings is consistent with regularity but doesn't establish it — only an explicit DFA/NFA/RE/regular grammar does that.

ST5 · Match: "counter", IDENTIFIER, and L(L+D)* to token, lexeme, and pattern.

Show solution

"counter" = lexeme (the actual text). IDENTIFIER = token (the category). L(L+D)* = pattern (the rule deciding membership).

ST6 · Using IDENTIFIER = L(L+D)* and NUMBER = D+, what's the token for "a1"?

Show solution

IDENTIFIER "a1", one token — 'a' starts it as an identifier, and the digit that follows is allowed to continue it (L(L+D)* permits digits after the first letter).

ST7 · In the scanning algorithm, what happens when lastAcceptPos is never set during a token attempt?

Show solution

LEXICAL ERROR is reported at the token's start position — there is no valid token boundary to back off to, which is different from the routine "got stuck after a valid accept" case.

ST8 · L1 is regular, L2 is not regular. Is L1 ∪ L2 necessarily regular, necessarily not regular, or neither?

Show solution

Neither is guaranteed. The union-closure theorem requires both inputs regular to promise anything; with one input non-regular, the union could go either way (e.g. L1=Σ* makes the union Σ*, regular, regardless of what L2 is) — no general conclusion follows from the theorem alone.

ST9 · A machine has 5 states. What's the shortest input length that pigeonhole guarantees forces a repeated state?

Show solution

5 symbols — reading 5 symbols means 6 state-visits total (including the start), which by pigeonhole must repeat some state among only 5 available. This is exactly the pumping length threshold |w|≥p with p=5.

ST10 · Scanning begins on input starting with '#', a character no pattern recognises. What's the very first thing that happens?

Show solution

S has no transition defined for '#' — the inner loop of the algorithm never executes even once, lastAcceptPos stays unset, and a LEXICAL ERROR is reported immediately at position 0, before any other token is attempted.


Where this goes next
  • Unit 3 moves from recognising to generating: context-free grammars, formally, with the ambiguity groundwork from Chapter 1 finally paid off in full.
  • The "guess and confirm" NFA design instinct from Chapter 2 reappears as the intuition behind pushdown automata (Unit 4) adding a stack to the same machine idea.
  • Leq, provably not regular three ways now, becomes the first language this whole course can finally build a machine for — once that machine has a stack.
Further reading
  • Sipser, Introduction to the Theory of Computation, 3rd ed. — the regular-languages chapter's closure and pumping-lemma sections match this chapter's order and notation closely.
  • Aho, Lam, Sethi, Ullman, Compilers: Principles, Techniques, and Tools. — the standard reference for lexical analysis and scanner construction in practice, considerably beyond this chapter's toy token set; good for seeing the real scale of a production lexer specification.
CEUC302 · Theory of Computation · Chapter 3 of 7 · Finite Automata II (Unit 2, Part 2 of 2)
Next: Chapter 4 · CFG I — Grammars, Derivations & Ambiguity