class BasicLexer

vedang6575 · web-flow · commit 6112d9b6ff7a · 2020-10-02T22:15:54.000+05:30
Thus we will need some basic tokens such as NAME, NUMBER, STRING. In any programming language, there will be space between two characters. Thus we create an ignore literal. Then we also create the basic literals like ‘=’, ‘+’ etc., NAME tokens are basically names of variables, which can be defined by the regular expression [a-zA-Z_][a-zA-Z0-9_]*. STRING tokens are string values and are bounded by quotation marks(” “). This can be defined by the regular expression \”.*?\”
diff --git a/class BasicLexer with python programming b/class BasicLexer with python programming
@@ -0,0 +1,28 @@
+ tokens = { NAME, NUMBER, STRING } 
+    ignore = '\t '
+    literals = { '=', '+', '-', '/',  
+                '*', '(', ')', ',', ';'} 
+  
+  
+    # Define tokens as regular expressions 
+    # (stored as raw strings) 
+    NAME = r'[a-zA-Z_][a-zA-Z0-9_]*'
+    STRING = r'\".*?\"'
+  
+    # Number token 
+    @_(r'\d+') 
+    def NUMBER(self, t): 
+        
+        # convert it into a python integer 
+        t.value = int(t.value)    return t 
+  
+    # Comment token 
+    @_(r'//.*') 
+    def COMMENT(self, t): 
+        pass
+  
+    # Newline token(used only for showing 
+    # errors in new line) 
+    @_(r'\n+') 
+    def newline(self, t): 
+        self.lineno = t.value.count('\n')
