tuple unpacking is slower than tuple(list comprehension)

Feature or enhancement

Proposal:

In []: %timeit (*(x**2 for x in range(1000)),)  # (A)
47.2 μs ± 1.18 μs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

In []: %timeit tuple(x**2 for x in range(1000))  # most idiomatic
45.4 μs ± 5.57 μs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

In []: %timeit (*[x**2 for x in range(1000)],)  # (B)
36.5 μs ± 77.8 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

In []: %timeit tuple([x**2 for x in range(1000)])  # fastest
33.8 μs ± 710 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

Currently (A) and (B) are slower than the last one, even though it doesn't need to be so. There doesn't seem to be any bottleneck such as global namespace lookup.

Also, we want people to write the most idiomatic code, so it would be preferable to make the most idiomatic version to be as fast as the fastest version. Possibly with something like

if tuple is builtins.tuple:
    MAGIC
else:
    code as usual...

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

Linked PRs

gh-138325: steal list items in INTRINSIC_LIST_TO_TUPLE #149960

Currently (A) and (B) are slower than the last one, even though it doesn't need to be so

There are different constructions.

(A) is generator unpacking
"most idiomatic" has a very high standard deviation so, (A) and this are somehow equivalent IMO. Both of them use generators.
(B) don't use a generator. You can see this with the disassembly.
The "fastest" is the fastest because you don't have unpacking and you don't have the INTRINSIC_LIST_TO_TUPLE call, but it's actually roughly close to (B) so it's fine.

The difference between what's in (A) and what's in (B) is essentially the use of a generator expression which makes it slower (but it's not always the case IIRC).

I don't think it's a bug, nor a feature we can actually enhance more so that we can have an overall improvement.

@Fidget-Spinner

I'll still tag some experts in this area to see if we could somehow optimize such constructions but this may be tricky. @Fidget-Spinner @markshannon

I don't think it's worth it to do such optimizations in the interpreter. It's possible to do them in the JIT in the far future.

But like Benedikt said: this difference is down to respecting Python semantics for both. Special casing tuples or lists to force eager evaluation requires something that can "see" more than one instruction at a time (ie a JIT)

@user202729

@user202729 which version of Python are you testing on?

We already do some special casing of the pattern tuple(expr for var in seq) but we still create the generator expression. It is possible that with a bit more work we could inline the whole thing, but care would be needed to not break backwards compatibility.
See #131738

the benchmark is in 3.13. The same behavior can be seen on 3.12, and 3.10 is much slower.

Can you try 3.14? #131738 wasn't in 3.13.

the reason I'm opening this pull request is that people are already recommending the usage of tuple([...]) instead of tuple(...). While tuple([…]) is faster (for now), its peak memory consumption is (up to) twice that of tuple(…), as mentioned in the comment in the linked StackOverflow answer above.

benchmark:

3.13:

>>> timeit.timeit('tuple(x**2 for x in range(1000))', number=10000)
0.9464342619758099
>>> timeit.timeit('tuple(x**2 for x in range(1000))', number=10000)
0.9527434469200671
>>> timeit.timeit('tuple([x**2 for x in range(1000)])', number=10000)
0.7785886479541659
>>> timeit.timeit('tuple([x**2 for x in range(1000)])', number=10000)
0.7813697159290314

3.15 (heads/main:2a54acf3):

>>> timeit.timeit('tuple(x**2 for x in range(1000))', number=10000)
0.8060883909929544
>>> timeit.timeit('tuple(x**2 for x in range(1000))', number=10000)
0.8000914799049497
>>> timeit.timeit('tuple([x**2 for x in range(1000)])', number=10000)
0.6702207860071212
>>> timeit.timeit('tuple([x**2 for x in range(1000)])', number=10000)
0.6622067128773779

there's some improvement (or maybe it's just because building on my machine is faster), but somehow list comprehension is still slower.

disassembly in 3.15:

>>> dis.dis('tuple(x**2 for x in range(1000))')
  0           RESUME                   0

  1           LOAD_NAME                0 (tuple)
              COPY                     1
              LOAD_COMMON_CONSTANT     2 (tuple)
              IS_OP                    0 (is)
              POP_JUMP_IF_FALSE       26 (to L3)
              NOT_TAKEN
              POP_TOP
              BUILD_LIST               0
              LOAD_CONST               0 (<code object <genexpr> at 0x7f04b88b2230, file "<dis>", li
ne 1>)
              MAKE_FUNCTION
              LOAD_NAME                1 (range)
              PUSH_NULL
              LOAD_CONST               1 (1000)
              CALL                     1
              CALL                     0
              PUSH_NULL
      L1:     FOR_ITER                 3 (to L2)
              LIST_APPEND              3
              JUMP_BACKWARD            5 (to L1)
      L2:     END_FOR
              POP_ITER
              CALL_INTRINSIC_1         6 (INTRINSIC_LIST_TO_TUPLE)
              RETURN_VALUE
      L3:     PUSH_NULL
              LOAD_CONST               0 (<code object <genexpr> at 0x7f04b88b2230, file "<dis>", li
ne 1>)
              MAKE_FUNCTION
              LOAD_NAME                1 (range)
              PUSH_NULL
              LOAD_CONST               1 (1000)
              CALL                     1
              CALL                     0
              CALL                     1
              RETURN_VALUE

Disassembly of <code object <genexpr> at 0x7f04b88b2230, file "<dis>", line 1>:
   1           RETURN_GENERATOR
               POP_TOP
       L1:     RESUME                   0
               LOAD_FAST                0 (.0)
               GET_ITER
       L2:     FOR_ITER                13 (to L3)
               STORE_FAST_LOAD_FAST    17 (x, x)
               LOAD_SMALL_INT           2
               BINARY_OP                8 (**)
               YIELD_VALUE              0
               RESUME                   5
               POP_TOP
               JUMP_BACKWARD           15 (to L2)
       L3:     END_FOR
               POP_ITER
               LOAD_CONST               1 (None)
               RETURN_VALUE

  --   L4:     CALL_INTRINSIC_1         3 (INTRINSIC_STOPITERATION_ERROR)
               RERAISE                  1
ExceptionTable:
  L1 to L4 -> L4 [0] lasti

there's a complaint in faster-cpython/ideas#545 (comment) too. Either the optimization isn't working somehow, or it's not doing what I expect.

Would not it be easy to implement optimization of the generator expression unpacking? First, convert it to a list comprehension, then remove an intermediate list and make LIST_APPEND appending directly to the outer list.

Would not it be easy to implement optimization of the generator expression unpacking? First, convert it to a list comprehension, then remove an intermediate list and make LIST_APPEND appending directly to the outer list.

You mean as an AST optimization right? I think that's possible

I think it can be done at the code generation stage. A part of it (conversion of generator expression to a list or set comprehension) can be in an AST optimization if this simplifies code generation.

I have been thinking about this a bit since I've spent a few days looking at micro-optimizations in the stdlib. I don't know the CPython implementation or semantics well enough to say what exactly would be safe etc., but I made a little benchmark and asked Claude Code to read this ticket and suggest improvement ideas, and go from there. It tried a few approaches which failed, but then AST case worked.

Claude patch is all in Python/ast_preprocess.c. Unit tests pass except for test.test_inspect.test_inspect, test_builtin, test_code, and test_dis. The benchmark results:

tuple_unpack: Mean +- std dev: [genexpr_baseline] 127 us +- 2 us -> [genexpr_optimized] 115 us +- 2 us: 1.10x faster
tuple: Mean +- std dev: [genexpr_baseline] 140 us +- 4 us -> [genexpr_optimized] 77.2 us +- 0.5 us: 1.82x faster
list: Mean +- std dev: [genexpr_baseline] 137 us +- 6 us -> [genexpr_optimized] 72.2 us +- 0.5 us: 1.90x faster
set: Mean +- std dev: [genexpr_baseline] 157 us +- 6 us -> [genexpr_optimized] 88.5 us +- 1.1 us: 1.78x faster
dict: Mean +- std dev: [genexpr_baseline] 163 us +- 3 us -> [genexpr_optimized] 87.3 us +- 0.8 us: 1.87x faster
listcomp: Mean +- std dev: [genexpr_baseline] 82.2 us +- 0.5 us -> [genexpr_optimized] 72.1 us +- 0.4 us: 1.14x faster
setcomp: Mean +- std dev: [genexpr_baseline] 99.0 us +- 0.9 us -> [genexpr_optimized] 88.9 us +- 1.2 us: 1.11x faster
dictcomp: Mean +- std dev: [genexpr_baseline] 97.2 us +- 1.0 us -> [genexpr_optimized] 87.3 us +- 1.0 us: 1.11x faster

I am not sure what the policy is regarding AI patches. If it is ok to post them I can show it here.

This is the benchmark I created:

import pyperf

def bench_tuple_unpack():
    a = (*(x**2 for x in range(1000)),)

def bench_tuple():
    a = tuple(x**2 for x in range(1000))

def bench_list():
    a = list(x**2 for x in range(1000))

def bench_set():
    a = set(x**2 for x in range(1000))

def bench_dict():
    a = dict((x, x**2) for x in range(1000))

def bench_listcomp():
    a = [x**2 for x in range(1000)]

def bench_setcomp():
    a = {x**2 for x in range(1000)}

def bench_dictcomp():
    a = {x: x**2 for x in range(1000)}

def add_benchmarks(runner):
    runner.bench_func(
        "tuple_unpack",
        bench_tuple_unpack
    )
    runner.bench_func(
        "tuple",
        bench_tuple
    )
    runner.bench_func(
        "list",
        bench_list
    )
    runner.bench_func(
        "set",
        bench_set
    )
    runner.bench_func(
        "dict",
        bench_dict
    )
    runner.bench_func(
        "listcomp",
        bench_listcomp
    )
    runner.bench_func(
        "setcomp",
        bench_setcomp
    )
    runner.bench_func(
        "dictcomp",
        bench_dictcomp
    )

if __name__ == "__main__":
    runner = pyperf.Runner()
    add_benchmarks(runner)

I consider posting LLM generated code fine, just make sure to add disclaimer that it's LLM generated code and understand every line of it yourself (rule of thumb: spend at least the time reviewer of ability equivalent to yours would have spent understanding the code)
- though in this case you said some unit tests fail, so it wouldn't be ready anyway without further inspection what these unit tests do.
- I consider it suspicious that listcomp is faster since that case is already looked at a lot already (are you sure it isn't doing any breaking change such as leaking the iteration variable?)

We have an explicit AI policy in the devguide: https://devguide.python.org/getting-started/generative-ai/.
It is not a good sign especially if builtin tests fail.

EDIT: my previous comment was about another issue...

@user202729

I am not claiming that the Claude patch would be good to go; it is more like a proof-of-concept that might give ideas what a real fix could be like. But I did find it interesting that all (?) of the non-introspection tests passed, and the performance changes were significant in the places I expected.

I don't know why the performance benchmark showed changes in tuple unpacking and *comp speed; it did not look to me like the patch should affect those.

@user202729 showed output of dis.dis('tuple(x**2 for x in range(1000))') as it is today. With the Claude patch that looks almost identical to dis.dis('[x**2 for x in range(1000)]'), which seemed promising.

This is the list comp dis:

>>> dis.dis('[x**2 for x in range(1000)]') # unpatched
   0           RESUME                   0

   1           LOAD_NAME                0 (range)
               PUSH_NULL
               LOAD_CONST               0 (1000)
               CALL                     1
               GET_ITER
               LOAD_FAST_AND_CLEAR      0 (x)
               SWAP                     2
       L1:     BUILD_LIST               0
               SWAP                     2
       L2:     FOR_ITER                11 (to L3)
               STORE_FAST_LOAD_FAST     0 (x, x)
               LOAD_SMALL_INT           2
               BINARY_OP                8 (**)
               LIST_APPEND              2
               JUMP_BACKWARD           13 (to L2)
       L3:     END_FOR
               POP_ITER
       L4:     SWAP                     2
               STORE_FAST               0 (x)
               RETURN_VALUE

  --   L5:     SWAP                     2
               POP_TOP

   1           SWAP                     2
               STORE_FAST               0 (x)
               RERAISE                  0
ExceptionTable:
  L1 to L4 -> L5 [2]

And here are the differences:

->>> dis.dis('[x**2 for x in range(1000)]') # unpatched
+>>> dis.dis('tuple(x**2 for x in range(1000))') patched
    0           RESUME                   0
 
-   1           LOAD_NAME                0 (range)
+   1           LOAD_NAME                0 (tuple)
                PUSH_NULL
+               LOAD_NAME                1 (range)
+               PUSH_NULL
                LOAD_CONST               0 (1000)
                CALL                     1
-               GET_ITER
                LOAD_FAST_AND_CLEAR      0 (x)
                SWAP                     2
        L1:     BUILD_LIST               0
                SWAP                     2
+               GET_ITER
        L2:     FOR_ITER                11 (to L3)
                STORE_FAST_LOAD_FAST     0 (x, x)
                LOAD_SMALL_INT           2
                BINARY_OP                8 (**)
-               LIST_APPEND              2
+               LIST_APPEND              3
                JUMP_BACKWARD           13 (to L2)
        L3:     END_FOR
                POP_ITER
        L4:     SWAP                     2
                STORE_FAST               0 (x)
+               CALL                     1
                RETURN_VALUE
 
   --   L5:     SWAP                     2
@@ -29,4 +32,4 @@
                STORE_FAST               0 (x)
                RERAISE                  0
 ExceptionTable:
-  L1 to L4 -> L5 [2]
+  L1 to L4 -> L5 [4]

This is the patch created by Claude Code:

diff --git a/Python/ast_preprocess.c b/Python/ast_preprocess.c
index d45435257cc..a1a32068a67 100644
--- a/Python/ast_preprocess.c
+++ b/Python/ast_preprocess.c
@@ -581,6 +581,81 @@ astfold_expr(expr_ty node_, PyArena *ctx_, _PyASTPreprocessState *state)
         CALL(astfold_expr, expr_ty, node_->v.Call.func);
         CALL_SEQ(astfold_expr, expr, node_->v.Call.args);
         CALL_SEQ(astfold_keyword, keyword, node_->v.Call.keywords);
+
+        // Optimize tuple(genexpr) -> tuple([listcomp])
+        // This eliminates generator object creation overhead
+        if (node_->v.Call.func->kind == Name_kind &&
+            asdl_seq_LEN(node_->v.Call.args) == 1 &&
+            asdl_seq_LEN(node_->v.Call.keywords) == 0) {
+
+            expr_ty arg = asdl_seq_GET(node_->v.Call.args, 0);
+            PyObject *name = node_->v.Call.func->v.Name.id;
+
+            // Transform tuple(genexpr) to tuple([listcomp])
+            if (_PyUnicode_EqualToASCIIString(name, "tuple") &&
+                arg->kind == GeneratorExp_kind) {
+                expr_ty listcomp = _PyAST_ListComp(
+                    arg->v.GeneratorExp.elt,
+                    arg->v.GeneratorExp.generators,
+                    arg->lineno, arg->col_offset,
+                    arg->end_lineno, arg->end_col_offset, ctx_);
+                if (listcomp == NULL) {
+                    return 0;
+                }
+                asdl_seq_SET(node_->v.Call.args, 0, listcomp);
+            }
+            // Transform list(genexpr) to [listcomp]
+            else if (_PyUnicode_EqualToASCIIString(name, "list") &&
+                     arg->kind == GeneratorExp_kind) {
+                expr_ty listcomp = _PyAST_ListComp(
+                    arg->v.GeneratorExp.elt,
+                    arg->v.GeneratorExp.generators,
+                    arg->lineno, arg->col_offset,
+                    arg->end_lineno, arg->end_col_offset, ctx_);
+                if (listcomp == NULL) {
+                    return 0;
+                }
+                // Replace the entire Call node with just the listcomp
+                *node_ = *listcomp;
+            }
+            // Transform set(genexpr) to {setcomp}
+            else if (_PyUnicode_EqualToASCIIString(name, "set") &&
+                     arg->kind == GeneratorExp_kind) {
+                expr_ty setcomp = _PyAST_SetComp(
+                    arg->v.GeneratorExp.elt,
+                    arg->v.GeneratorExp.generators,
+                    arg->lineno, arg->col_offset,
+                    arg->end_lineno, arg->end_col_offset, ctx_);
+                if (setcomp == NULL) {
+                    return 0;
+                }
+                // Replace the entire Call node with just the setcomp
+                *node_ = *setcomp;
+            }
+            // Transform dict((k, v) for ...) to {k: v for ...}
+            else if (_PyUnicode_EqualToASCIIString(name, "dict") &&
+                     arg->kind == GeneratorExp_kind) {
+                // Check if the genexpr yields tuples with exactly 2 elements
+                expr_ty elt = arg->v.GeneratorExp.elt;
+                if (elt->kind == Tuple_kind &&
+                    asdl_seq_LEN(elt->v.Tuple.elts) == 2) {
+                    // Extract key and value from the tuple
+                    expr_ty key = asdl_seq_GET(elt->v.Tuple.elts, 0);
+                    expr_ty value = asdl_seq_GET(elt->v.Tuple.elts, 1);
+
+                    expr_ty dictcomp = _PyAST_DictComp(
+                        key, value,
+                        arg->v.GeneratorExp.generators,
+                        arg->lineno, arg->col_offset,
+                        arg->end_lineno, arg->end_col_offset, ctx_);
+                    if (dictcomp == NULL) {
+                        return 0;
+                    }
+                    // Replace the entire Call node with just the dictcomp
+                    *node_ = *dictcomp;
+                }
+            }
+        }
         break;
     case FormattedValue_kind:
         CALL(astfold_expr, expr_ty, node_->v.FormattedValue.value);

I don't think it'll work. This is going to pass through all calls, but I think it will fail with:

list = tuple
s = list(1 for _ in range(3))

In addition, what about obj.list() calls? I think it would also be caught here.

Good point. I also tried this same exercise with Amp Code Free, and while it didn't get to anything that would work, it did point out cases that we need to watch out for, like subclassing.

I don't think an AST change is actually a good idea. But it could be done in the JIT perhaps. Or we could have an other instruction where we don't create the generator but create a real list as Serhiy suggested:

Would not it be easy to implement optimization of the generator expression unpacking? First, convert it to a list comprehension, then remove an intermediate list and make LIST_APPEND appending directly to the outer list.

pythongh-138325: steal list items in INTRINSIC_LIST_TO_TUPLE

gh-138325: steal list items in `INTRINSIC_LIST_TO_TUPLE` opcode (#149960

I don't think an AST change is actually a good idea. But it could be done in the JIT perhaps. Or we could have an other instruction where we don't create the generator but create a real list as Serhiy suggested:

No. For large inputs, the generator form can significantly reduce peak memory usage.

In short: tuple([x for x in list_a]) does temporarily require almost the memory of both a list and a tuple of the same size at once, but it does not duplicate the actual element objects, only the references to them.

Navigation Menu

Repository navigation

tuple unpacking is slower than tuple(list comprehension) #138325

Feature or enhancement

Proposal:

Has this already been discussed elsewhere?

Links to previous discussion of this feature:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

tuple unpacking is slower than tuple(list comprehension) #138325

Description

Feature or enhancement

Proposal:

Has this already been discussed elsewhere?

Links to previous discussion of this feature:

Linked PRs

Activity

picnixz commented on Sep 1, 2025

picnixz commented on Sep 1, 2025

Fidget-Spinner commented on Sep 1, 2025

markshannon commented on Sep 1, 2025

user202729 commented on Sep 1, 2025

markshannon commented on Sep 1, 2025

user202729 commented on Sep 1, 2025

serhiy-storchaka commented on Sep 2, 2025

Fidget-Spinner commented on Sep 2, 2025

serhiy-storchaka commented on Sep 2, 2025

heikkitoivonen commented on Jan 10, 2026

user202729 commented on Jan 10, 2026

picnixz commented on Jan 10, 2026

heikkitoivonen commented on Jan 10, 2026

picnixz commented on Jan 10, 2026

heikkitoivonen commented on Jan 10, 2026

picnixz commented on Jan 10, 2026

lbaumbauer commented on Aug 3, 2026

lbaumbauer commented on Aug 3, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions