Skip to content
This repository was archived by the owner on May 14, 2026. It is now read-only.

library_generation

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History
 
 

library_generation

README.md

Generate a repository containing GAPIC Client Libraries

The script, entry_point.py, allows you to generate a repository containing GAPIC client libraries with googleapis commit history (a monorepo, for example, google-cloud-java) from a configuration file.

Environment

  • OS: Linux
  • Java runtime environment (8 or above)
  • Apache Maven (used in formatting source code)
  • Python (3.11.6 or above)
  • Docker
  • Git

Prerequisite

In order to generate a version for each library, a versions.txt has to exist in repository_path. Please refer to Repository path for more information.

Parameters to generate a repository using entry_point.py

Baseline generation configuration yaml (baseline_generation_config)

An absolute or relative path to a generation_config.yaml. This config file is used for commit history generation, not library generation.

Current generation configuration yaml (current_generation_config)

An absolute or relative path to a configuration file containing parameters to generate the repository. Please refer Configuration to generate a repository for more information.

Repository path (repository_path), optional

The path to where the generated repository goes.

The default value is the current working directory when running the script. For example, cd google-cloud-java && python entry_point.py ... without specifying the --repository_path option will modify the google-cloud-java repository the user cd'd into.

Note that versions.txt has to exist in repository_path in order to generate right version for each library. Please refer here for more info of versions.txt.

Output of entry_point.py

GAPIC libraries

For each module (e.g. google-cloud-java/java-asset), the following files/folders will be created/modified:

Name Notes
google-*/ Source code generated by gapic-generator-java
google-*/pom.xml Only be generated if it does not exist
grpc-*/ Source code generated by grpc generator, one per each version
grpc-*/pom.xml Only be generated if it does not exist
proto-*/ Source code generated by Protobuf default compiler, one per each version
proto-*/pom.xml Only be generated if it does not exist
samples/snippets/generated/ Only be generated if include_samples is set to true
google-*-bom/pom.xml Library BOM, only be generated if it does not exist
pom.xml Library parent BOM, only be generated if it does not exist
.repo-metadata.json Always generated from inputs
.OwlBot-hermetic.yaml Only be generated from a template if it does not exist
owlbot.py Only be generated from a template if it does not exist
README.md Always generated from inputs
gapic-libraries-bom/pom.xml Always generated from inputs
pom.xml (repo root dir) Always generated from inputs
versions.txt New entries will be added if they don’t exist

googleapis commit history

If both baseline_generation_config and current_generation_config are specified, and they contain different googleapis commit, the commit history will be generated into pr_description.txt in the repository_path.

Configuration to generate a repository

There are three levels of parameters in the configuration: repository level, library level and GAPIC level.

Repository level parameters

The repository level parameters define the version of API definition (proto) and tools. They are shared by library level parameters.

Name Required Notes
gapic_generator_version Yes
protoc_version No inferred from the generator if not specified
grpc_version No inferred from the generator if not specified
googleapis-commitish Yes
libraries_bom_version Yes
template_excludes Yes

Library level parameters

The library level parameters define how to generate a (multi-versions) GAPIC library. They are shared by all GAPICs of a library.

Name Required Notes
api_shortname Yes
api_description Yes
name_pretty Yes
product_docs Yes
library_type No GAPIC_AUTO if not specified
release_level No preview if not specified
api_id No {api_shortname}.googleapis.com if not specified
api_reference No
codeowner_team No
client_documentation No
distribution_name No {group_id}:google-{cloud_prefix}{library_name} if not specified
excluded_poms No
excluded_dependencies No
googleapis_commitish No use repository level googleapis_commitish if not specified.
group_id No com.google.cloud if not specified
issue_tracker No
library_name No api_shortname is not specified. This value should be unique among all libraries.
rest_documentation No
rpc_documentation No
cloud_api No true if not specified
requires-billing No true if not specified

Note that cloud_prefix is cloud- if cloud_api is true; empty otherwise.

GAPIC level parameters

The GAPIC level parameters define how to generate a GAPIC library.

Name Required Notes
proto_path Yes versioned proto_path starts with google

An example of generation configuration

gapic_generator_version: 2.34.0
protoc_version: 25.2
googleapis_commitish: 1a45bf7393b52407188c82e63101db7dc9c72026
libraries_bom_version: 26.37.0
destination_path: google-cloud-java
template_excludes:
  - ".github/*"
  - ".kokoro/*"
  - "samples/*"
  - "CODE_OF_CONDUCT.md"
  - "CONTRIBUTING.md"
  - "LICENSE"
  - "SECURITY.md"
  - "java.header"
  - "license-checks.xml"
  - "renovate.json"
  - ".gitignore"
libraries:
  - api_shortname: apigeeconnect
    name_pretty: Apigee Connect
    product_documentation: "https://cloud.google.com/apigee/docs/hybrid/v1.3/apigee-connect/"
    api_description: "allows the Apigee hybrid management plane to connect securely to the MART service in the runtime plane without requiring you to expose the MART endpoint on the internet."
    release_level: "stable"
    library_name: "apigee-connect"
    GAPICs:
      - proto_path: google/cloud/apigeeconnect/v1

  - api_shortname: cloudasset
    name_pretty: Cloud Asset Inventory
    product_documentation: "https://cloud.google.com/resource-manager/docs/cloud-asset-inventory/overview"
    api_description: "provides inventory services based on a time series database. This database keeps a five week history of Google Cloud asset metadata. The Cloud Asset Inventory export service allows you to export all asset metadata at a certain timestamp or export event change history during a timeframe."
    library_name: "asset"
    client_documentation: "https://cloud.google.com/java/docs/reference/google-cloud-asset/latest/overview"
    distribution_name: "com.google.cloud:google-cloud-asset"
    release_level: "stable"
    issue_tracker: "https://issuetracker.google.com/issues/new?component=187210&template=0"
    api_reference: "https://cloud.google.com/resource-manager/docs/cloud-asset-inventory/overview"
    GAPICs:
      - proto_path: google/cloud/asset/v1
      - proto_path: google/cloud/asset/v1p1beta1
      - proto_path: google/cloud/asset/v1p2beta1
      - proto_path: google/cloud/asset/v1p5beta1
      - proto_path: google/cloud/asset/v1p7beta1

Local Environment Setup before running entry_point.py

  1. Assuming Python 3 is installed, follow official guide from Python.org to create a virtual environment. The virtual environment can be installed to any folder, usually it is recommended to be installed under the root folder of the project(sdk-platform-java in this case).
  2. Assuming the virtual environment is installed under sdk-platform-java. Run the following command under the root folder of sdk-platform-java to install the dependencies of library_generation
python -m pip install -r library_generation/requirements.txt
  1. Run the following command to install library_generation as a module, which allows the library_generation module to be imported from anywhere
python -m pip install library_generation/

An example to generate a repository using entry_point.py

python library_generation/entry_point.py generate \
--baseline-generation-config-path=/path/to/baseline_config_file \
--current-generation-config-path=/path/to/current_config_file \
--repository-path=path/to/repository

If you run entry_point.py with the example configuration shown above, the repository structure is:

$repository_path
|_gapic-libraries-bom
|    |_pom.xml
|_java-apigee-connect
|    |_google-cloud-apigee-connect
|    |    |_src
|    |    |_pom.xml
|    |_google-cloud-apigee-connect-bom
|    |    |_pom.xml
|    |_grpc-google-cloud-apigee-connect-v1
|    |    |_src
|    |    |_pom.xml
|    |_proto-google-cloud-apigee-connect-v1
|    |    |_src
|    |    |_pom.xml
|    |_samples
|    |    |_snippets
|    |    |    |_generated
|    |_.OwlBot-hermetic.yaml
|    |_.repo-metadata.json
|    |_owlbot.py
|    |_pom.xml
|    |_README.md
|_java-asset
|    |_google-cloud-asset
|    |    |_src
|    |    |_pom.xml
|    |_google-cloud-asset-bom
|    |    |_pom.xml
|    |_grpc-google-cloud-asset-v1
|    |    |_src
|    |    |_pom.xml
|    |_grpc-google-cloud-asset-v1p1beta1
|    |    |_src
|    |    |_pom.xml
|    |_grpc-google-cloud-asset-v1p2beta1
|    |    |_src
|    |    |_pom.xml
|    |_grpc-google-cloud-asset-v1p5beta1
|    |    |_src
|    |    |_pom.xml
|    |_grpc-google-cloud-asset-v1p7beta1
|    |    |_src
|    |    |_pom.xml
|    |_proto-google-cloud-asset-v1
|    |    |_src
|    |    |_pom.xml
|    |_proto-google-cloud-asset-v1p1beta1
|    |    |_src
|    |    |_pom.xml
|    |_proto-google-cloud-asset-v1p2beta1
|    |    |_src
|    |    |_pom.xml
|    |_proto-google-cloud-asset-v1p5beta1
|    |    |_src
|    |    |_pom.xml
|    |_proto-google-cloud-asset-v1p7beta1
|    |    |_src
|    |    |_pom.xml
|    |_samples
|    |    |_snippets
|    |    |    |_generated
|    |_.OwlBot-hermetic.yaml
|    |_.repo-metadata.json
|    |_owlbot.py
|    |_pom.xml
|    |_README.md
|_pom.xml
|_versions.txt

Owlbot Java Postprocessor

We have transferred the implementation of Java Owlbot Postprocessor into sdk-platform-java/library_generation. The implementation in synthtool is still valid and used by other services, so we have two versions during a transition period.

Reflecting changes in synthtool/docker/owlbot/java into this repository

The transfer was not a verbatim copy, it rather had modifications:

  • format-source.sh was replaced by a call to mvn fmt:format
  • entrypoint.sh was modified to have input arguments and slightly modified the way the helper scripts are called
  • Other helper scripts were modified to have input arguments.
  • fix_poms.py modified the way the monorepo is detected

All these modifications imply that whenever we want to reflect a change from the original owlbot in synthtool we may be better off modifying the affected source files one by one. The mapping is from synthtool/docker/owlbot/java to sdk-platform-java/library_generation/owlbot