Skip to content

GH-99554: marshal bytecode more efficiently #99555

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
brandtbucher wants to merge 9 commits into from
Closed

GH-99554: marshal bytecode more efficiently #99555

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
4034318
Compress marshalled bytecode
brandtbucher Nov 15, 2022
8b6b393
blurb add
brandtbucher Nov 16, 2022
5f81b82
Add opcode_for_build
brandtbucher Nov 16, 2022
e34c460
Catch up with main
brandtbucher Dec 15, 2022
23ba802
fixup
brandtbucher Dec 15, 2022
6ed83af
Catch up with main
brandtbucher Dec 17, 2022
5534112
Address review comments
brandtbucher Dec 17, 2022
ead8f26
Catch up with main
brandtbucher Dec 17, 2022
bcd7980
Terminate the compressed bytecode with two zeros
brandtbucher Dec 18, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add opcode_for_build
  • Loading branch information
@brandtbucher
brandtbucher committed Nov 16, 2022
commit 5f81b82cc2d00a567389c8e83ae1104906a80fc5
2 changes: 1 addition & 1 deletion Lib/opcode.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -205,7 +205,7 @@ def pseudo_op(name, op, real_ops):
hasfree.append(148)
def_op('COPY_FREE_VARS', 149)
def_op('YIELD_VALUE', 150)
def_op('RESUME', 151) # This must be kept in sync with deepfreeze.py
def_op('RESUME', 151)
def_op('MATCH_CLASS', 152)

def_op('FORMAT_VALUE', 155)
Expand Down
4 changes: 2 additions & 2 deletions Tools/build/deepfreeze.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,13 @@
from typing import Dict, FrozenSet, TextIO, Tuple

import umarshal
import opcode_for_build as opcode
Copy link
Copy Markdown
Member

@markshannon markshannon Nov 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't a normal import so it shouldn't look like one. Maybe something like opcode = opcode_finder.get_opcodes()

from generate_global_objects import get_identifiers_and_strings

verbose = False
identifiers, strings = get_identifiers_and_strings()

# This must be kept in sync with opcode.py
RESUME = 151
RESUME = opcode.opmap["RESUME"]

def isprintable(b: bytes) -> bool:
return all(0x20 <= c < 0x7f for c in b)
Expand Down
17 changes: 17 additions & 0 deletions Tools/build/opcode_for_build.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
"""
Parts of our build process (looking at you, deepfreeze) need the opcode module
for the Python *being built*, not the Python *doing the building*.

This basically just loads ../../Lib/opcode.py and re-exports everything:

>>> import opcode_for_build as opcode
"""

import os

_opcode_path = os.path.join(
os.path.dirname(__file__), os.pardir, os.pardir, "Lib", "opcode.py"
)
with open(_opcode_path, encoding="utf-8") as _opcode_file:
# Don't try this at home, kids:
exec(_opcode_file.read())
4 changes: 2 additions & 2 deletions Tools/build/umarshal.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Implementat marshal.loads() in pure Python

import ast
import opcode
import opcode_for_build as opcode

from typing import Any, Tuple

Expand Down Expand Up @@ -180,7 +180,7 @@ def r_object(self) -> Any:
return self._r_object()
finally:
self.level = old_level

def r_bytecode(self) -> bytes:
nbytes = self.r_long() * 2
bytecode = bytearray()
Expand Down