Skip to content

Commit 4f91fc0

Browse files
coryancoryan
authored
feat(storage)!: move GCS+gRPC to v2/ protos (#7254)
GCS+gRPC has published a new set of protos to target. These protos rename and change the types for a number of fields, and they *only* describe the RPCs that are actually implemented. The major modifications in this change include: - The type and names of the crc32c and md5_hash fields changed. - Previously the md5 hash was sometimes represented as a hex-encoded binary blob and sometimes as just a binary blob. Now it is always a binary blob. - Several fields (like `if_generation_match` and `crc32c`) are now marked as `optional` in the protos, instead of using `*Value` message wrappers. - The proto version of `ObjectAccessControl` does not contain the `bucket`, `object` or `generation` fields. - Likewise, `BucketAccessControl` does not contain the bucket name. - We use https://github.com/googleapis/storage-testbench as it supports the v2/ protos.
1 parent cdbfdea commit 4f91fc0

57 files changed

Lines changed: 549 additions & 6504 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitignore‎

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,3 @@ cmake-build-*/
2323
# Ignore Visual Studio Code files
2424
.vsbuild/
2525
.vscode/
26-
27-
google/cloud/storage/emulator/**/__pycache__/
28-
google/cloud/storage/emulator/**/*.pyc

‎CHANGELOG.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,16 @@
5454

5555
## v1.32.0 - TBD
5656

57+
### [Storage](https://github.com/googleapis/google-cloud-cpp/blob/main/google/cloud/storage/README.md)
58+
59+
**BREAKING CHANGES:**
60+
61+
* We have removed the code generated from `storage/v1` protos. These protos were
62+
not GA themselves, and the underlying service will be disabled in the near
63+
future. Furthermore, the library that used them (the GCS+gRPC plugin) is
64+
clearly labeled `experimental`. Regardless, we apologize if this causes you
65+
inconvenience or additional work.
66+
5767
## v1.31.0 - 2021-09
5868

5969
### [BigQuery](https://github.com/googleapis/google-cloud-cpp/blob/main/google/cloud/bigquery/README.md)

‎ci/cloudbuild/builds/cmake-install.sh‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -84,7 +84,6 @@ expected_dirs=(
8484
./include/google/spanner/admin/database/v1
8585
./include/google/spanner/admin/instance/v1
8686
./include/google/spanner/v1
87-
./include/google/storage/v1
8887
./include/google/storage/v2
8988
./include/google/type
9089
./lib64/cmake/bigtable_client

‎ci/cloudbuild/builds/lib/integration.sh‎

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ source module ci/lib/io.sh
2828

2929
# To run the integration tests we need to install the dependencies for the storage emulator
3030
export PATH="${HOME}/.local/bin:${PATH}"
31-
python3 -m pip install --quiet --user -r "${PROJECT_ROOT}/google/cloud/storage/emulator/requirements.txt"
31+
python3 -m pip install --quiet --user "git+git://github.com/googleapis/storage-testbench@v0.4.0"
3232

3333
# Some of the tests will need a valid roots.pem file.
3434
rm -f /dev/shm/roots.pem
@@ -172,10 +172,6 @@ function integration::bazel_with_emulators() {
172172
"google/cloud/pubsub/ci/${EMULATOR_SCRIPT}" \
173173
bazel "${verb}" "${args[@]}"
174174

175-
io::log_h2 "Running Storage Emulator integration tests"
176-
"google/cloud/storage/emulator/ci/run_integration_tests_bazel.sh" \
177-
bazel "${args[@]}"
178-
179175
io::log_h2 "Running Storage integration tests (with emulator)"
180176
"google/cloud/storage/ci/${EMULATOR_SCRIPT}" \
181177
bazel "${verb}" "${args[@]}"

‎external/googleapis/CMakeLists.txt‎

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -134,8 +134,6 @@ set(EXTERNAL_GOOGLEAPIS_PROTO_FILES
134134
"google/spanner/v1/spanner.proto"
135135
"google/spanner/v1/transaction.proto"
136136
"google/spanner/v1/type.proto"
137-
"google/storage/v1/storage.proto"
138-
"google/storage/v1/storage_resources.proto"
139137
"google/storage/v2/storage.proto"
140138
"google/type/calendar_period.proto"
141139
"google/type/color.proto"
@@ -647,8 +645,6 @@ target_link_libraries(
647645

648646
google_cloud_cpp_grpcpp_library(
649647
google_cloud_cpp_storage_protos
650-
"${EXTERNAL_GOOGLEAPIS_SOURCE}/google/storage/v1/storage.proto"
651-
"${EXTERNAL_GOOGLEAPIS_SOURCE}/google/storage/v1/storage_resources.proto"
652648
"${EXTERNAL_GOOGLEAPIS_SOURCE}/google/storage/v2/storage.proto"
653649
PROTO_PATH_DIRECTORIES
654650
"${EXTERNAL_GOOGLEAPIS_SOURCE}"

‎google/cloud/storage/BUILD‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -59,8 +59,8 @@ cc_library(
5959
"@com_google_absl//absl/algorithm:container",
6060
"@com_google_absl//absl/strings",
6161
"@com_google_absl//absl/time",
62-
"@com_google_googleapis//google/storage/v1:storage_cc_grpc",
63-
"@com_google_googleapis//google/storage/v1:storage_cc_proto",
62+
"@com_google_googleapis//google/storage/v2:storage_cc_grpc",
63+
"@com_google_googleapis//google/storage/v2:storage_cc_proto",
6464
],
6565
)
6666

‎google/cloud/storage/benchmarks/BUILD‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -31,8 +31,8 @@ cc_library(
3131
"@boringssl//:crypto",
3232
"@boringssl//:ssl",
3333
"@com_github_curl_curl//:curl",
34-
"@com_google_googleapis//google/storage/v1:storage_cc_grpc",
35-
"@com_google_googleapis//google/storage/v1:storage_cc_proto",
34+
"@com_google_googleapis//google/storage/v2:storage_cc_grpc",
35+
"@com_google_googleapis//google/storage/v2:storage_cc_proto",
3636
],
3737
)
3838

‎google/cloud/storage/benchmarks/throughput_experiment.cc‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@
2424
#include "absl/algorithm/container.h"
2525
#include "absl/memory/memory.h"
2626
#include "absl/strings/str_split.h"
27-
#include <google/storage/v1/storage.grpc.pb.h>
27+
#include <google/storage/v2/storage.grpc.pb.h>
2828
#include <curl/curl.h>
2929
#include <vector>
3030

@@ -275,20 +275,20 @@ std::shared_ptr<grpc::ChannelInterface> CreateGcsChannel(int thread_id) {
275275
class DownloadObjectRawGrpc : public ThroughputExperiment {
276276
public:
277277
explicit DownloadObjectRawGrpc(int thread_id)
278-
: stub_(google::storage::v1::Storage::NewStub(
278+
: stub_(google::storage::v2::Storage::NewStub(
279279
CreateGcsChannel(thread_id))) {}
280280
~DownloadObjectRawGrpc() override = default;
281281

282282
ThroughputResult Run(std::string const& bucket_name,
283283
std::string const& object_name,
284284
ThroughputExperimentConfig const& config) override {
285285
auto timer = Timer::PerThread();
286-
google::storage::v1::GetObjectMediaRequest request;
286+
google::storage::v2::ReadObjectRequest request;
287287
request.set_bucket(bucket_name);
288288
request.set_object(object_name);
289289
grpc::ClientContext context;
290-
auto stream = stub_->GetObjectMedia(&context, request);
291-
google::storage::v1::GetObjectMediaResponse response;
290+
auto stream = stub_->ReadObject(&context, request);
291+
google::storage::v2::ReadObjectResponse response;
292292
std::int64_t bytes_received = 0;
293293
while (stream->Read(&response)) {
294294
if (response.has_checksummed_data()) {
@@ -312,7 +312,7 @@ class DownloadObjectRawGrpc : public ThroughputExperiment {
312312
}
313313

314314
private:
315-
std::unique_ptr<google::storage::v1::Storage::StubInterface> stub_;
315+
std::unique_ptr<google::storage::v2::Storage::StubInterface> stub_;
316316
};
317317

318318
} // namespace

‎google/cloud/storage/ci/run_integration_tests_emulator_bazel.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -85,12 +85,12 @@ printf '{"name": "%s"}' "${GOOGLE_CLOUD_CPP_STORAGE_TEST_DESTINATION_BUCKET_NAME
8585
curl -X POST -H "Content-Type: application/json" --data-binary @- \
8686
"${CLOUD_STORAGE_EMULATOR_ENDPOINT}/storage/v1/b?project=${GOOGLE_CLOUD_PROJECT}"
8787

88-
EMULATOR_SHA=$(git ls-files google/cloud/storage/emulator | sort | cat | sha256sum)
88+
TESTBENCH_SHA="$(pip show googleapis-storage-testbench | sha256sum)"
8989
emulator_args=(
90+
"--test_env=TESTBENCH_SHA=${TESTBENCH_SHA}"
9091
"--test_env=CLOUD_STORAGE_EMULATOR_ENDPOINT=${CLOUD_STORAGE_EMULATOR_ENDPOINT}"
9192
"--test_env=CLOUD_STORAGE_GRPC_ENDPOINT=${CLOUD_STORAGE_GRPC_ENDPOINT}"
9293
"--test_env=HTTPBIN_ENDPOINT=${HTTPBIN_ENDPOINT}"
93-
"--test_env=EMULATOR_SHA=${EMULATOR_SHA}"
9494
"--test_env=GOOGLE_CLOUD_CPP_STORAGE_TEST_HMAC_SERVICE_ACCOUNT=fake-service-account-sign@example.com"
9595
)
9696

‎google/cloud/storage/ci/run_integration_tests_emulator_cmake.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ start_emulator
4141
# `storage_bucket_samples` binary is missing the examples that use said bucket
4242
# are missing too.
4343
printf '{"name": "%s"}' "${GOOGLE_CLOUD_CPP_STORAGE_TEST_DESTINATION_BUCKET_NAME}" |
44-
curl -X POST -H "Content-Type: application/json" --data-binary @- \
44+
curl -s -X POST -H "Content-Type: application/json" --data-binary @- \
4545
"${CLOUD_STORAGE_EMULATOR_ENDPOINT}/storage/v1/b?project=${GOOGLE_CLOUD_PROJECT}"
4646

4747
ctest -R "^storage_" "${ctest_args[@]}"

0 commit comments

Comments
 (0)