Skip to content

Commit 96e5c07

Browse files
partheavchudnov-g
parthea
and
vchudnov-g
authored
fix: Add compatibility with protobuf==5.x (#433)
* ci: add support for pre-release sessions * add compatibility with protobuf 5.x * fix RecursionError: maximum recursion depth exceeded while calling a Python object in tests * update required checks * update comments * update required checks * See protocolbuffers/protobuf#16596 breaking change * lint * Address review feedback * lint * add tests * Update tests/test_fields_mitigate_collision.py * add deprecation warning * Cater for protobuf 5.x+ * style * Filter deprecation warning * remove redundant code * revert * add comment * add comment * refactor code * style * update warning filter * address review comments * map_composite_types_str->map_composite_type_names * update comment * lint * add more test cases in test_json_default_values * add more test cases to tests/test_message.py * address review feedback * add comment * formatting * formatting * typo * Address review feedback * Update proto/marshal/marshal.py Co-authored-by: Victor Chudnovsky <vchudnov@google.com> * typo * address review feedback * address review feedback * fix test case * stye * add more test cases * add test case * Update tests/test_message.py Co-authored-by: Victor Chudnovsky <vchudnov@google.com> --------- Co-authored-by: Victor Chudnovsky <vchudnov@google.com>
1 parent b6f5db8 commit 96e5c07

File tree

12 files changed

+447
-58
lines changed

12 files changed

+447
-58
lines changed

‎packages/proto-plus/.github/sync-repo-settings.yaml‎

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -7,24 +7,27 @@ branchProtectionRules:
77
requiredStatusCheckContexts:
88
- 'style-check'
99
- 'docs'
10-
- 'unit (3.6)'
1110
- 'unit (3.6, cpp)'
12-
- 'unit (3.7)'
11+
- 'unit (3.6, python)'
12+
- 'unit (3.6, upb)'
1313
- 'unit (3.7, cpp)'
14+
- 'unit (3.7, python)'
1415
- 'unit (3.7, upb)'
15-
- 'unit (3.8)'
1616
- 'unit (3.8, cpp)'
17+
- 'unit (3.8, python)'
1718
- 'unit (3.8, upb)'
18-
- 'unit (3.9)'
1919
- 'unit (3.9, cpp)'
20+
- 'unit (3.9, python)'
2021
- 'unit (3.9, upb)'
21-
- 'unit (3.10)'
2222
- 'unit (3.10, cpp)'
23+
- 'unit (3.10, python)'
2324
- 'unit (3.10, upb)'
24-
- 'unit (3.11)'
25+
- 'unit (3.11, python)'
2526
- 'unit (3.11, upb)'
26-
- 'unit (3.12)'
27+
- 'unit (3.12, python)'
2728
- 'unit (3.12, upb)'
29+
- 'prerelease (3.12, python)'
30+
- 'prerelease (3.12, upb)'
2831
- cover
2932
- OwlBot Post Processor
3033
- 'cla/google'

‎packages/proto-plus/.github/workflows/tests.yml‎

Lines changed: 37 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -42,19 +42,27 @@ jobs:
4242
runs-on: ubuntu-20.04
4343
strategy:
4444
matrix:
45-
python: ['3.6', '3.7', '3.8', '3.9', '3.10', '3.11', '3.12']
46-
variant: ['', 'cpp', 'upb']
45+
python: ['3.6', '3.7', '3.8', '3.9', '3.10', '3.11', '3.12', '3.13']
46+
variant: ['cpp', 'python', 'upb']
47+
# TODO(https://github.com/googleapis/proto-plus-python/issues/389):
48+
# Remove the 'cpp' implementation once support for Protobuf 3.x is dropped.
49+
# The 'cpp' implementation requires Protobuf == 3.x however version 3.x
50+
# does not support Python 3.11 and newer. The 'cpp' implementation
51+
# must be excluded from the test matrix for these runtimes.
4752
exclude:
4853
- variant: "cpp"
4954
python: 3.11
5055
- variant: "cpp"
5156
python: 3.12
57+
- variant: "cpp"
58+
python: 3.13
5259
steps:
5360
- uses: actions/checkout@v4
5461
- name: Set up Python ${{ matrix.python }}
5562
uses: actions/setup-python@v5
5663
with:
5764
python-version: ${{ matrix.python }}
65+
allow-prereleases: true
5866
- name: Install nox
5967
run: |
6068
pip install nox
@@ -68,12 +76,38 @@ jobs:
6876
env:
6977
COVERAGE_FILE: .coverage-${{ matrix.variant }}-${{ env.PYTHON_VERSION_TRIMMED }}
7078
run: |
71-
nox -s unit${{ matrix.variant }}-${{ env.PYTHON_VERSION_TRIMMED }}
79+
nox -s "unit-${{ env.PYTHON_VERSION_TRIMMED }}(implementation='${{ matrix.variant }}')"
7280
- name: Upload coverage results
7381
uses: actions/upload-artifact@v4
7482
with:
7583
name: coverage-artifact-${{ matrix.variant }}-${{ env.PYTHON_VERSION_TRIMMED }}
7684
path: .coverage-${{ matrix.variant }}-${{ env.PYTHON_VERSION_TRIMMED }}
85+
prerelease:
86+
runs-on: ubuntu-20.04
87+
strategy:
88+
matrix:
89+
python: ['3.12']
90+
variant: ['python', 'upb']
91+
steps:
92+
- uses: actions/checkout@v4
93+
- name: Set up Python ${{ matrix.python }}
94+
uses: actions/setup-python@v5
95+
with:
96+
python-version: ${{ matrix.python }}
97+
allow-prereleases: true
98+
- name: Install nox
99+
run: |
100+
pip install nox
101+
- name: Run unit tests
102+
env:
103+
COVERAGE_FILE: .coverage-prerelease-${{ matrix.variant }}
104+
run: |
105+
nox -s "prerelease_deps(implementation='${{ matrix.variant }}')"
106+
- name: Upload coverage results
107+
uses: actions/upload-artifact@v4
108+
with:
109+
name: coverage-artifact-prerelease-${{ matrix.variant }}
110+
path: .coverage-prerelease-${{ matrix.variant }}
77111
cover:
78112
runs-on: ubuntu-latest
79113
needs:

‎packages/proto-plus/noxfile.py‎

Lines changed: 63 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -13,10 +13,9 @@
1313
# limitations under the License.
1414

1515
from __future__ import absolute_import
16-
import os
17-
import pathlib
1816

1917
import nox
18+
import pathlib
2019

2120

2221
CURRENT_DIRECTORY = pathlib.Path(__file__).parent.absolute()
@@ -30,33 +29,38 @@
3029
"3.10",
3130
"3.11",
3231
"3.12",
32+
"3.13",
3333
]
3434

3535
# Error if a python version is missing
3636
nox.options.error_on_missing_interpreters = True
3737

3838

3939
@nox.session(python=PYTHON_VERSIONS)
40-
def unit(session, proto="python"):
40+
@nox.parametrize("implementation", ["cpp", "upb", "python"])
41+
def unit(session, implementation):
4142
"""Run the unit test suite."""
4243

4344
constraints_path = str(
4445
CURRENT_DIRECTORY / "testing" / f"constraints-{session.python}.txt"
4546
)
4647

47-
session.env["PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION"] = proto
48+
session.env["PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION"] = implementation
4849
session.install("coverage", "pytest", "pytest-cov", "pytz")
4950
session.install("-e", ".[testing]", "-c", constraints_path)
50-
if proto == "cpp": # 4.20 does not have cpp.
51-
session.install("protobuf==3.19.0")
51+
# TODO(https://github.com/googleapis/proto-plus-python/issues/389):
52+
# Remove the 'cpp' implementation once support for Protobuf 3.x is dropped.
53+
# The 'cpp' implementation requires Protobuf<4.
54+
if implementation == "cpp":
55+
session.install("protobuf<4")
5256

5357
# TODO(https://github.com/googleapis/proto-plus-python/issues/403): re-enable `-W=error`
5458
# The warnings-as-errors flag `-W=error` was removed in
5559
# https://github.com/googleapis/proto-plus-python/pull/400.
5660
# It should be re-added once issue
57-
# https://github.com/protocolbuffers/protobuf/issues/12186 is fixed.
61+
# https://github.com/protocolbuffers/protobuf/issues/15077 is fixed.
5862
session.run(
59-
"py.test",
63+
"pytest",
6064
"--quiet",
6165
*(
6266
session.posargs # Coverage info when running individual tests is annoying.
@@ -71,17 +75,59 @@ def unit(session, proto="python"):
7175
)
7276

7377

74-
# Check if protobuf has released wheels for new python versions
75-
# https://pypi.org/project/protobuf/#files
76-
# This list will generally be shorter than 'unit'
77-
@nox.session(python=["3.6", "3.7", "3.8", "3.9", "3.10"])
78-
def unitcpp(session):
79-
return unit(session, proto="cpp")
78+
# Only test upb and python implementation backends.
79+
# As of protobuf 4.x, the "ccp" implementation is not available in the PyPI package as per
80+
# https://github.com/protocolbuffers/protobuf/tree/main/python#implementation-backends
81+
@nox.session(python=PYTHON_VERSIONS[-2])
82+
@nox.parametrize("implementation", ["python", "upb"])
83+
def prerelease_deps(session, implementation):
84+
"""Run the unit test suite against pre-release versions of dependencies."""
8085

86+
session.env["PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION"] = implementation
8187

82-
@nox.session(python=PYTHON_VERSIONS)
83-
def unitupb(session):
84-
return unit(session, proto="upb")
88+
# Install test environment dependencies
89+
session.install("coverage", "pytest", "pytest-cov", "pytz")
90+
91+
# Install the package without dependencies
92+
session.install("-e", ".", "--no-deps")
93+
94+
prerel_deps = [
95+
"google-api-core",
96+
# dependency of google-api-core
97+
"googleapis-common-protos",
98+
]
99+
100+
for dep in prerel_deps:
101+
session.install("--pre", "--no-deps", "--upgrade", dep)
102+
103+
session.install("--pre", "--upgrade", "protobuf")
104+
# Print out prerelease package versions
105+
session.run(
106+
"python", "-c", "import google.protobuf; print(google.protobuf.__version__)"
107+
)
108+
session.run(
109+
"python", "-c", "import google.api_core; print(google.api_core.__version__)"
110+
)
111+
112+
# TODO(https://github.com/googleapis/proto-plus-python/issues/403): re-enable `-W=error`
113+
# The warnings-as-errors flag `-W=error` was removed in
114+
# https://github.com/googleapis/proto-plus-python/pull/400.
115+
# It should be re-added once issue
116+
# https://github.com/protocolbuffers/protobuf/issues/15077 is fixed.
117+
session.run(
118+
"pytest",
119+
"--quiet",
120+
*(
121+
session.posargs # Coverage info when running individual tests is annoying.
122+
or [
123+
"--cov=proto",
124+
"--cov-config=.coveragerc",
125+
"--cov-report=term",
126+
"--cov-report=html",
127+
"tests",
128+
]
129+
),
130+
)
85131

86132

87133
@nox.session(python="3.9")

‎packages/proto-plus/proto/marshal/compat.py‎

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,9 @@
1919
# not be included.
2020

2121
from google.protobuf.internal import containers
22+
import google.protobuf
23+
24+
PROTOBUF_VERSION = google.protobuf.__version__
2225

2326
# Import protobuf 4.xx first and fallback to earlier version
2427
# if not present.
@@ -37,14 +40,28 @@
3740
repeated_scalar_types = (containers.RepeatedScalarFieldContainer,)
3841
map_composite_types = (containers.MessageMap,)
3942

43+
# In `proto/marshal.py`, for compatibility with protobuf 5.x,
44+
# we'll use `map_composite_type_names` to check whether
45+
# the name of the class of a protobuf type is
46+
# `MessageMapContainer`, and, if `True`, return a MapComposite.
47+
# See https://github.com/protocolbuffers/protobuf/issues/16596
48+
map_composite_type_names = ("MessageMapContainer",)
49+
4050
if _message:
4151
repeated_composite_types += (_message.RepeatedCompositeContainer,)
4252
repeated_scalar_types += (_message.RepeatedScalarContainer,)
43-
map_composite_types += (_message.MessageMapContainer,)
4453

54+
# In `proto/marshal.py`, for compatibility with protobuf 5.x,
55+
# we'll use `map_composite_type_names` to check whether
56+
# the name of the class of a protobuf type is
57+
# `MessageMapContainer`, and, if `True`, return a MapComposite.
58+
# See https://github.com/protocolbuffers/protobuf/issues/16596
59+
if PROTOBUF_VERSION[0:2] in ["3.", "4."]:
60+
map_composite_types += (_message.MessageMapContainer,)
4561

4662
__all__ = (
4763
"repeated_composite_types",
4864
"repeated_scalar_types",
4965
"map_composite_types",
66+
"map_composite_type_names",
5067
)

‎packages/proto-plus/proto/marshal/marshal.py‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -188,7 +188,14 @@ def to_python(self, proto_type, value, *, absent: bool = None):
188188
return Repeated(value, marshal=self)
189189

190190
# Same thing for maps of messages.
191-
if value_type in compat.map_composite_types:
191+
# See https://github.com/protocolbuffers/protobuf/issues/16596
192+
# We need to look up the name of the type in compat.map_composite_type_names
193+
# as class `MessageMapContainer` is no longer exposed
194+
# This is done to avoid taking a breaking change in proto-plus.
195+
if (
196+
value_type in compat.map_composite_types
197+
or value_type.__name__ in compat.map_composite_type_names
198+
):
192199
return MapComposite(value, marshal=self)
193200
return self.get_rule(proto_type=proto_type).to_python(value, absent=absent)
194201

0 commit comments

Comments
 (0)