Skip to content

Commit 00b51b2

Browse files
jgrahamjgraham
committed
Remove the states dictionary from the tokenizer to reduce lookups
1 parent e9279e8 commit 00b51b2

1 file changed

Lines changed: 126 additions & 169 deletions

File tree

0 commit comments

Comments
 (0)