Skip to content

Commit 733ed00

Browse files
andrewnesterjuliacrawf-db
andrewnester
and
juliacrawf-db
authored
Added an example how to deploy resources only for specific targets (#96)
Added an example of how to deploy resources only for specific targets This example illustrates how to achieve the same configuration often requested by using conditional includes/target includes or variables in the `include` section. See databricks/cli#2878 databricks/cli#2872 databricks/cli#2903 as examples of such use cases. There are some key aspects in this implementation 1. In `databricks.yml` file we include (see `include` section) all configuration files for all targets. This does not impact which resources will be deployed for which target. 2. For each job in the corresponding configuration file, like `resources/set1/job_1.yml`, we define in which targets this job should be deployed. We use YAML anchors to avoid duplication between targets. --------- Co-authored-by: Julia Crawford (Databricks) <julia.crawford@databricks.com>
1 parent ec62463 commit 733ed00

8 files changed

Lines changed: 248 additions & 0 deletions

File tree

‎knowledge_base/target_includes/.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
.databricks
2+
*.jar
3+
*.class

‎knowledge_base/target_includes/README.md‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
# Target Includes Example
2+
3+
This example demonstrates the concept of using `target_includes` (or similar include mechanisms) in Databricks Asset Bundles to organize job configurations across different environments without duplication.
4+
5+
It addresses the use case described in [GitHub Issue #2878](https://github.com/databricks/cli/issues/2878), which requests the ability to include specific resource files based on target configurations.
6+
7+
## Directory Structure
8+
9+
```
10+
target_includes/
11+
├── databricks.yml # Main bundle configuration with 3 targets
12+
├── resources/
13+
│ ├── set1/ # Jobs for dev and staging environments
14+
│ │ ├── job_1.yml
15+
│ │ └── job_2.yml
16+
│ └── set2/ # Jobs for staging and prod environments
17+
│ ├── job_1.yml
18+
│ └── job_2.yml
19+
└── README.md
20+
```
21+
22+
## Target Configuration
23+
24+
The bundle defines three targets:
25+
26+
- **dev**: Includes only `resources/set1/*.yml`
27+
- Contains: set1-job-1, set1-job-2
28+
- Environment: development
29+
30+
- **staging**: Includes both `resources/set1/*.yml` and `resources/set2/*.yml`
31+
- Contains: set1-job-1, set1-job-2, set2-job-1, set2-job-2
32+
- Environment: staging
33+
34+
- **prod**: Includes only `resources/set2/*.yml`
35+
- Contains: set2-job-1, set2-job-2
36+
- Environment: production
37+
38+
39+
## Notes
40+
41+
There are some important aspects of this implementation:
42+
- The `databricks.yml` file includes all configuration files for all targets (see `include` section). This does not impact which resources will be deployed to each target.
43+
- For each job in a corresponding configuration file, such as `resources/set1/job_1.yml`, targets are defined where the job should be deployed. YAML anchors are used to avoid duplications between targets.
44+
45+
46+
## Usage
47+
48+
```bash
49+
databricks bundle summary -p u2m -t dev
50+
```
51+
52+
Output:
53+
```bash
54+
Name: target-includes-example
55+
Target: dev
56+
Workspace:
57+
User: ***
58+
Path: ***
59+
Resources:
60+
Jobs:
61+
set1-job-1:
62+
Name: Set1 Job 1 - foo
63+
URL: (not deployed)
64+
set1-job-2:
65+
Name: Set1 Job 2 - foo
66+
URL: (not deployed)
67+
```
68+
69+
```bash
70+
databricks bundle summary -p u2m -t staging
71+
```
72+
73+
Output:
74+
```bash
75+
Name: target-includes-example
76+
Target: staging
77+
Workspace:
78+
User: ***
79+
Path: ***
80+
Resources:
81+
Jobs:
82+
set1-job-1:
83+
Name: Set1 Job 1 - bar
84+
URL: (not deployed)
85+
set1-job-2:
86+
Name: Set1 Job 2 - bar
87+
URL: (not deployed)
88+
set2-job-1:
89+
Name: Set2 Job 1 - bar
90+
URL: (not deployed)
91+
set2-job-2:
92+
Name: Set2 Job 2 - bar
93+
URL: (not deployed)
94+
```
95+
96+
```bash
97+
databricks bundle summary -p u2m -t prod
98+
```
99+
100+
Output:
101+
```bash
102+
Name: target-includes-example
103+
Target: prod
104+
Workspace:
105+
User: ***
106+
Path: ***
107+
Resources:
108+
Jobs:
109+
set2-job-1:
110+
Name: Set2 Job 1 - baz
111+
URL: (not deployed)
112+
set2-job-2:
113+
Name: Set2 Job 2 - baz
114+
URL: (not deployed)
115+
```

‎knowledge_base/target_includes/databricks.yml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
bundle:
2+
name: target-includes-example
3+
4+
include:
5+
- resources/set1/*.yml
6+
- resources/set2/*.yml
7+
8+
variables:
9+
name_suffix:
10+
description: "Target specific suffix for the job name"
11+
12+
targets:
13+
dev:
14+
default: true
15+
variables:
16+
name_suffix: foo
17+
18+
staging:
19+
variables:
20+
name_suffix: bar
21+
22+
prod:
23+
variables:
24+
name_suffix: baz

‎knowledge_base/target_includes/resources/set1/job_1.yml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# Job 1 for set1 - used in dev and staging targets
2+
# Using YAML anchors to avoid duplication
3+
4+
job-config: &job-config
5+
set1-job-1:
6+
name: "Set1 Job 1 - ${var.environment}"
7+
tasks:
8+
- task_key: "process_data"
9+
notebook_task:
10+
notebook_path: ../../src/notebook.py
11+
max_concurrent_runs: 1
12+
timeout_seconds: 3600
13+
14+
targets:
15+
dev:
16+
resources:
17+
jobs:
18+
<<: *job-config
19+
staging:
20+
resources:
21+
jobs:
22+
<<: *job-config

‎knowledge_base/target_includes/resources/set1/job_2.yml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# Job 2 for set1 - used in dev and staging targets
2+
# Using YAML anchors to avoid duplication
3+
4+
job-config: &job-config
5+
set1-job-2:
6+
name: "Set1 Job 2 - ${bundle.target}"
7+
tasks:
8+
- task_key: "process_data"
9+
notebook_task:
10+
notebook_path: ../../src/notebook.py
11+
max_concurrent_runs: 1
12+
timeout_seconds: 3600
13+
trigger:
14+
type: "cron"
15+
cron:
16+
quartz_cron_expression: "0 0 12 * * ?"
17+
timezone_id: "UTC"
18+
19+
targets:
20+
dev:
21+
resources:
22+
jobs:
23+
<<: *job-config
24+
staging:
25+
resources:
26+
jobs:
27+
<<: *job-config

‎knowledge_base/target_includes/resources/set2/job_1.yml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# Job 1 for set2 - used in staging and prod targets
2+
# Using YAML anchors to avoid duplication
3+
4+
job-config: &job-config
5+
set2-job-1:
6+
name: "Set2 Job 1 - ${bundle.target}"
7+
tasks:
8+
- task_key: "process_data"
9+
notebook_task:
10+
notebook_path: ../../src/notebook.py
11+
max_concurrent_runs: 1
12+
timeout_seconds: 3600
13+
trigger:
14+
type: "cron"
15+
cron:
16+
quartz_cron_expression: "0 0 12 * * ?"
17+
timezone_id: "UTC"
18+
19+
targets:
20+
staging:
21+
resources:
22+
jobs:
23+
<<: *job-config
24+
prod:
25+
resources:
26+
jobs:
27+
<<: *job-config

‎knowledge_base/target_includes/resources/set2/job_2.yml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# Job 2 for set2 - used in staging and prod targets
2+
# Using YAML anchors to avoid duplication
3+
4+
job-config: &job-config
5+
set2-job-2:
6+
name: "Set2 Job 2 - ${bundle.target}"
7+
tasks:
8+
- task_key: "process_data"
9+
notebook_task:
10+
notebook_path: ../../src/notebook.py
11+
max_concurrent_runs: 1
12+
timeout_seconds: 3600
13+
trigger:
14+
type: "cron"
15+
cron:
16+
quartz_cron_expression: "0 0 12 * * ?"
17+
timezone_id: "UTC"
18+
19+
targets:
20+
staging:
21+
resources:
22+
jobs:
23+
<<: *job-config
24+
prod:
25+
resources:
26+
jobs:
27+
<<: *job-config

‎knowledge_base/target_includes/src/notebook.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# Databricks notebook source
2+
3+
print("Hello From Notebook!")

0 commit comments

Comments
 (0)