Don't make slice copies in TokenList._token_matching().

living180 · andialbrecht · commit a3e19f1cbdf9 · 2022-08-08T10:52:33.000+02:00
Since we are working with indexes anyway, don't bother calling
enumerate() with a slice from self.tokens (which requires copying
memory).  Instead, just generate the indexes using range() and use
normal indexing to access the desired tokens.

The old behavior resulted in quadratic runtime with respect to the
number of tokens, which significantly impacted performance for
statements with very large numbers of tokens.

With the new behavior, the runtime is now linear with respect to the
number of tokens.
diff --git a/sqlparse/sql.py b/sqlparse/sql.py
@@ -240,7 +240,10 @@ def _token_matching(self, funcs, start=0, end=None, reverse=False):
                     if func(token):
                         return idx, token
         else:
-            for idx, token in enumerate(self.tokens[start:end], start=start):
+            if end is None:
+                end = len(self.tokens)
+            for idx in range(start, end):
+                token = self.tokens[idx]
                 for func in funcs:
                     if func(token):
                         return idx, token
