-
Notifications
You must be signed in to change notification settings - Fork 1.1k
chore: Add a library generation skill #12689
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
lqiu96
wants to merge
9
commits into
main
Choose a base branch
from
library-generation-skill
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
4419e5d
chore: Add new library generation agent skill
lqiu96 90bfaa9
chore: Refine new-client-library-generator skill for best practices
lqiu96 7a01500
chore: Update the skill with new information (no Cloud Drop reference)
lqiu96 90fea12
chore: Add reference files
lqiu96 f2e6bb5
Merge branch 'main' of github.com:googleapis/google-cloud-java into l…
lqiu96 23819b3
chore: apply suggestions from code review
lqiu96 fef9c9b
Update .agents/skills/new-client-library-generator/references/generat…
lqiu96 bae703c
chore: Remove reference to github actions and use local generation
lqiu96 e50e79e
chore: Update SKILL.md
lqiu96 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,124 @@ | ||
| --- | ||
| name: new-client-library-generator | ||
| description: Generates new Google Cloud Java client libraries by processing service information and updating the hermetic build configuration. Use this skill when tasked with onboarding a new service or version to the google-cloud-java repository. | ||
| --- | ||
|
|
||
| # New Client Library Generator | ||
|
|
||
| This skill automates the process of adding a new client library to the `google-cloud-java` repository using the hermetic build system. It retrieves service information from a Buganizer ticket (which links to a Service YAML file) and runs the configuration script to update `generation_config.yaml`. | ||
|
|
||
| ## Workflow | ||
|
|
||
| ### 1. Retrieve Service Information from Buganizer | ||
|
|
||
| Use the available Buganizer MCP server to fetch the ticket content. The ticket will contain a link to a Service YAML file. Parse this YAML file to extract the fields below. | ||
|
|
||
| > [!IMPORTANT] | ||
| > Not all fields in the Service YAML file may exist. You must ensure that all **Required** fields are identified and populated before executing the generation script. | ||
|
|
||
| **Required:** | ||
| - `api_shortname`: Unique service identifier (e.g., `alloydb`). | ||
| - `name_pretty`: Human-friendly name (e.g., `AlloyDB API`). | ||
| - `proto_path`: Versioned path to protos (e.g., `google/cloud/alloydb/v1`). Must include the version component — root-level paths like `google/cloud/alloydb` are not supported. | ||
| - `product_docs`: Product documentation URL (must start with `https://`). | ||
| - `api_description`: First sentence of the service summary. | ||
|
|
||
| **Optional:** | ||
| - `rest_docs`: REST reference documentation URL. | ||
| - `rpc_docs`: RPC/proto reference documentation URL. | ||
| - `library_name`: Override the default `java-<api_shortname>` directory name. | ||
| - `distribution_name`: Override Maven coordinates (default: `com.google.cloud:google-cloud-<api_shortname>`). | ||
|
|
||
| For the field-to-flag mapping, see [references/service_yaml_mapping.md](references/service_yaml_mapping.md) (Service YAML fields → script flags). | ||
|
|
||
| ### 2. Check for Conflicts | ||
|
|
||
| Before running the script, verify `api_shortname` is not already in use: | ||
|
|
||
| ```bash | ||
| grep "api_shortname: <API_SHORTNAME>" generation_config.yaml | ||
| ``` | ||
|
|
||
| If a conflict exists, determine a unique name or use `--library-name` to set a distinct directory. See [references/generation_guide.md](references/generation_guide.md) for examples. | ||
|
|
||
| ### 3. Special Cases | ||
|
|
||
| Some APIs require non-default Maven coordinates and `api_shortname` values: | ||
|
|
||
| | Proto path prefix | `--api-shortname` | `--distribution-name` | | ||
| |---------------------|------------------------------|----------------------------------------------------------| | ||
| | `google/maps/*` | `maps-<api_short_name>` | `com.google.maps:google-maps-<api_short_name>` | | ||
| | `google/shopping/*` | `shopping-<api_short_name>` | `com.google.shopping:google-shopping-<api_short_name>` | | ||
|
|
||
| where `<api_short_name>` is the value from the Service YAML. | ||
|
|
||
| ### 4. Execution | ||
|
|
||
| #### Adding a new library | ||
|
|
||
| If the library does not exist yet, run the script with the gathered information: | ||
|
|
||
| ```bash | ||
| python3 generation/new_client_hermetic_build/add-new-client-config.py add-new-library \ | ||
| --api-shortname="[API_SHORTNAME]" \ | ||
| --name-pretty="[NAME_PRETTY]" \ | ||
| --proto-path="[PROTO_PATH]" \ | ||
| --product-docs="[PRODUCT_DOCS]" \ | ||
| --api-description="[API_DESCRIPTION]" \ | ||
| [OPTIONAL_FLAGS] | ||
| ``` | ||
|
|
||
| The script modifies `generation_config.yaml` and sorts the `libraries` list alphabetically. | ||
|
|
||
| To see all available flags: | ||
| ```bash | ||
| python3 generation/new_client_hermetic_build/add-new-client-config.py add-new-library --help | ||
| ``` | ||
|
|
||
| #### Adding a new version to an existing library | ||
|
|
||
| If the client library module already exists and the request is to add a new version, do NOT run the script. Instead, manually add the corresponding `proto_path` for the new version to the `GAPICs` list of the existing library entry in `generation_config.yaml`. | ||
|
|
||
| Example entry update: | ||
| ```yaml | ||
| - api_shortname: myapi | ||
| ... | ||
| GAPICs: | ||
| - proto_path: google/cloud/myapi/v1 | ||
| - proto_path: google/cloud/myapi/v2 # Manually added | ||
| ``` | ||
|
|
||
| ### 5. Verification | ||
|
|
||
| After execution or manual update: | ||
| 1. Confirm `generation_config.yaml` has a new or updated entry with the correct fields. See [references/generation_config_schema.md](references/generation_config_schema.md). | ||
| 2. Confirm `proto_path` under `GAPICs` includes a version component (e.g., `v1`, `v1beta`, `v1alpha`). | ||
| 3. Confirm `product_documentation` starts with `https://`. | ||
|
|
||
| ### 6. Create a Branch and PR | ||
|
|
||
| After verifying the changes: | ||
|
|
||
| ```bash | ||
| git checkout -b "new-library/[API_SHORTNAME]" | ||
| git add generation_config.yaml | ||
| git commit -m "feat: new module for [API_SHORTNAME]" | ||
| ``` | ||
|
|
||
| Then open a pull request. The Pull Request body must only contain the `add-new-library` command used. | ||
|
|
||
| **PR Body Template:** | ||
|
|
||
| ```text | ||
| Command used: | ||
|
|
||
| python3 generation/new_client_hermetic_build/add-new-client-config.py add-new-library \ | ||
| --api-shortname="[API_SHORTNAME]" \ | ||
| --name-pretty="[NAME_PRETTY]" \ | ||
| --proto-path="[PROTO_PATH]" \ | ||
| --product-docs="[PRODUCT_DOCS]" \ | ||
| --api-description="[API_DESCRIPTION]" \ | ||
| [OPTIONAL_FLAGS] | ||
| ``` | ||
|
|
||
| The hermetic library generation workflow will be triggered automatically upon changes to `generation_config.yaml`. |
57 changes: 57 additions & 0 deletions
57
.agents/skills/new-client-library-generator/references/generation_config_schema.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| # generation_config.yaml Schema | ||
|
|
||
| The `generation_config.yaml` file at the root of the repository controls the generation of all client libraries. | ||
|
|
||
| ## Key Fields | ||
|
|
||
| - `gapic_generator_version`: The version of the GAPIC generator used globally (unless overridden). | ||
| - `googleapis_commitish`: The commit of the `googleapis` repository used as the source for protos. | ||
| - `libraries`: A list of library configurations. | ||
|
|
||
| ### Library Configuration Fields | ||
|
|
||
| **Required (always present):** | ||
| - `api_shortname`: (String) Identifier (e.g., `alloydb`). | ||
| - `name_pretty`: (String) Display name (e.g., `AlloyDB API`). | ||
| - `product_documentation`: (URL) Product documentation URL. | ||
| - `api_description`: (String) Service description. | ||
| - `client_documentation`: (URL) Auto-generated link to the generated client docs. | ||
| - `release_level`: (String) Usually `preview` for new libraries. | ||
| - `distribution_name`: (String) Maven coordinates (e.g., `com.google.cloud:google-cloud-alloydb`). | ||
| - `api_id`: (String) API identifier (e.g., `alloydb.googleapis.com`). | ||
| - `library_type`: (String) Usually `GAPIC_AUTO`. | ||
| - `group_id`: (String) Maven group ID (e.g., `com.google.cloud`). | ||
| - `cloud_api`: (Boolean) `true` if distribution name starts with `google-cloud-`. | ||
| - `GAPICs`: (List) List of proto paths to generate from. | ||
| - `proto_path`: (String) Path to versioned protos (must include version, e.g., `google/cloud/alloydb/v1`). | ||
|
|
||
| **Optional:** | ||
| - `library_name`: (String) Override the output directory name (without `java-` prefix). | ||
| - `rest_documentation`: (URL) REST reference documentation URL. | ||
| - `rpc_documentation`: (URL) RPC/proto reference documentation URL. | ||
| - `requires_billing`: (Boolean) Whether billing is required (default: `true`). | ||
| - `api_reference`: (URL) API reference documentation link. | ||
| - `codeowner_team`: (String) GitHub team responsible for this library. | ||
| - `googleapis_commitish`: (String) Pin to a specific `googleapis/googleapis` commit (overrides repo-level setting). | ||
| - `issue_tracker`: (URL) Issue tracker for this library. | ||
| - `extra_versioned_modules`: (String) Extra modules managed via `versions.txt`. | ||
| - `excluded_dependencies`: (String) Comma-separated dependencies excluded from postprocessing. | ||
| - `excluded_poms`: (String) Comma-separated pom files excluded from postprocessing. | ||
|
|
||
| ## Example Library Entry | ||
|
|
||
| ```yaml | ||
| - api_shortname: alloydb | ||
| name_pretty: AlloyDB API | ||
| product_documentation: https://cloud.google.com/alloydb/docs | ||
| api_description: AlloyDB for PostgreSQL is an open source-compatible database service. | ||
| client_documentation: https://cloud.google.com/java/docs/reference/google-cloud-alloydb/latest/overview | ||
| release_level: preview | ||
| distribution_name: com.google.cloud:google-cloud-alloydb | ||
| api_id: alloydb.googleapis.com | ||
| library_type: GAPIC_AUTO | ||
| group_id: com.google.cloud | ||
| cloud_api: true | ||
| GAPICs: | ||
| - proto_path: google/cloud/alloydb/v1 | ||
| ``` |
75 changes: 75 additions & 0 deletions
75
.agents/skills/new-client-library-generator/references/generation_guide.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,75 @@ | ||
| # New Client Library Generation Guide | ||
|
|
||
| This guide describes how to generate a new client library by: | ||
| 1. Appending a new library configuration to `generation_config.yaml` using the provided scripts. | ||
| 2. Relying on the automated GitHub Actions to generate the code once the configuration is pushed in a Pull Request. | ||
|
|
||
| ## Components | ||
|
|
||
| ### `generation/new_client_hermetic_build/add-new-client-config.py` | ||
| This script updates `generation_config.yaml` with the necessary metadata for a new library. It takes several arguments that map to fields in the configuration file. | ||
|
|
||
| ## Local Execution | ||
|
|
||
| To add a new library configuration, you must run the `add-new-client-config.py` script. The parameters for this script are typically found in the **Service YAML** file linked in the Buganizer request. | ||
|
|
||
| ### Basic Usage | ||
|
|
||
| ```bash | ||
| python3 generation/new_client_hermetic_build/add-new-client-config.py add-new-library \ | ||
| --api-shortname="[API_SHORTNAME]" \ | ||
| --name-pretty="[NAME_PRETTY]" \ | ||
| --proto-path="[PROTO_PATH]" \ | ||
| --product-docs="[PRODUCT_DOCS]" \ | ||
| --api-description="[API_DESCRIPTION]" | ||
| ``` | ||
|
|
||
| ### Parameter Details | ||
|
|
||
| #### API Short Name (`--api-shortname`) | ||
| The unique identifier for the library (e.g., `alloydb`). This is used to determine the directory name and default artifact name. | ||
| - **Source:** `api_short_name` in the Service YAML. | ||
|
|
||
| #### Proto Path (`--proto-path`) | ||
| The path from the root of the `googleapis` repository to the directory containing the versioned proto definitions. | ||
| - **Example:** `google/cloud/alloydb/v1beta` | ||
| - **Note:** Root-level paths like `google/cloud/alloydb` are not supported. | ||
|
|
||
| #### Name Pretty (`--name-pretty`) | ||
| The human-friendly name of the API. | ||
| - **Source:** `title` in the Service YAML. | ||
| - **Example:** `AlloyDB API` | ||
|
|
||
| #### Product Docs (`--product-docs`) | ||
| The URL for the product documentation. Must start with `https://`. | ||
| - **Source:** `documentation_uri` in the Service YAML. | ||
|
|
||
| #### API Description (`--api-description`) | ||
| A concise summary of the API for the README. | ||
| - **Source:** `documentation.summary` or `documentation.overview` in the Service YAML. Use the first sentence. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| Ensure you have the following set up in your local environment: | ||
|
|
||
| 1. **Python 3.9+**: The scripts require Python 3.9 or higher. | ||
| 2. **Dependencies**: Install the required Python packages from the root of the repository: | ||
| ```bash | ||
| pip install -r generation/new_client_hermetic_build/requirements.txt | ||
| ``` | ||
|
|
||
| ## Advanced Options | ||
|
|
||
| For full control over the generation (e.g., overriding Maven coordinates or library names), run the script with the `--help` flag: | ||
|
|
||
| ```bash | ||
| python3 generation/new_client_hermetic_build/add-new-client-config.py add-new-library --help | ||
| ``` | ||
|
|
||
| ### Special Case: Non-Cloud APIs | ||
| Some libraries (like Google Maps or Shopping) require special handling for Maven coordinates. | ||
|
|
||
| | API Path Prefix | `--api-shortname` | `--distribution-name` | | ||
| |---|---|---| | ||
| | `google/maps/*` | `maps-<shortname>` | `com.google.maps:google-maps-<shortname>` | | ||
| | `google/shopping/*` | `shopping-<shortname>` | `com.google.shopping:google-shopping-<shortname>` | | ||
41 changes: 41 additions & 0 deletions
41
.agents/skills/new-client-library-generator/references/service_yaml_mapping.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| # Service YAML Mapping to Generation Script Parameters | ||
|
|
||
| This document explains how to extract information from a Service YAML file (linked in a Buganizer ticket) to populate the parameters for `generation/new_client_hermetic_build/add-new-client-config.py`. | ||
|
|
||
| | Service YAML Field | Script Parameter (`--flag`) | Description | | ||
| |---|---|---| | ||
| | `api_short_name` | `api-shortname` | Unique identifier for the service (e.g., `alloydb`). | | ||
| | (extracted from proto) | `proto-path` | Path from the root of the googleapis repository to the versioned proto directory (e.g., `google/cloud/alloydb/v1alpha`). Root-level paths like `google/cloud/alloydb` are not supported. | | ||
| | `title` | `name-pretty` | Human-friendly name (e.g., `AlloyDB API`). | | ||
| | `documentation_uri` | `product-docs` | Product documentation URL. Must start with `https://`. | | ||
lqiu96 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| | `rest_reference_documentation_uri` | `rest-docs` | REST documentation URL. Optional. | | ||
| | `proto_reference_documentation_uri` | `rpc-docs` | RPC/Proto documentation URL. Optional. | | ||
| | `documentation.summary` or `documentation.overview` | `api-description` | Concise summary of the service. Use the first sentence. | | ||
|
|
||
| ## Example Mapping | ||
|
|
||
| **Service YAML File excerpt:** | ||
| ```yaml | ||
| title: "Discovery Engine API" | ||
| api_short_name: "discoveryengine" | ||
| documentation_uri: "https://cloud.google.com/generative-ai-app-builder/docs/introduction" | ||
| rest_reference_documentation_uri: "https://cloud.google.com/generative-ai-app-builder/docs/reference/rest" | ||
| proto_reference_documentation_uri: "https://cloud.google.com/generative-ai-app-builder/docs/reference/rpc" | ||
| documentation: | ||
| summary: "Discovery Engine for Search and Recommendations" | ||
| ``` | ||
|
|
||
| **Protos in `googleapis` repository:** | ||
| `google/cloud/discoveryengine/v1/` | ||
|
|
||
| **Generated Command:** | ||
| ```bash | ||
| python3 generation/new_client_hermetic_build/add-new-client-config.py add-new-library \ | ||
| --api-shortname="discoveryengine" \ | ||
| --name-pretty="Discovery Engine API" \ | ||
| --proto-path="google/cloud/discoveryengine/v1" \ | ||
| --product-docs="https://cloud.google.com/generative-ai-app-builder/docs/introduction" \ | ||
| --rest-docs="https://cloud.google.com/generative-ai-app-builder/docs/reference/rest" \ | ||
| --rpc-docs="https://cloud.google.com/generative-ai-app-builder/docs/reference/rpc" \ | ||
| --api-description="Discovery Engine for Search and Recommendations" | ||
| ``` | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.