Skip to content

Commit cccb3ae

Browse files
MikeRayMSFTMikeRayMSFT
committed
Merge branch 'master' of https://github.com/MicrosoftDocs/sql-docs-pr into release-remove-sscurrent-tag
2 parents 8378d55 + c59895c commit cccb3ae

43 files changed

Lines changed: 547 additions & 212 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎docs/big-data-cluster/cluster-manage-notebooks.md‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ Once all dependencies are installed, but **Run all cells** fails, each notebook
2626
This section contains a set of notebooks useful for installing and uninstalling command-line tools and packages needed to manage SQL Server Big Data Clusters.
2727

2828
|Name |Description |
29-
|---|---|---|---|
29+
|---|---|
3030
|SOP010 - Upgrade a big data cluster|Use this notebook to upgrade a Big Data Cluster using azdata. |
3131
|SOP012 - Install unixodbc for Mac|Use this notebook when getting errors while using brew install the odbc for SQL Server.|
3232
|SOP036 - Install kubectl command-line interface|Use this notebook to install kubectl command-line interface regardless your OS.|
@@ -44,13 +44,24 @@ This section contains a set of notebooks useful for installing and uninstalling
4444
|SOP064 - Uninstall azdata CLI (using package manager)|Use this notebook to uninstall azdata CLI (using package manager).|
4545
|SOP069 - Install ODBC for SQL Server|Use this notebook to install ODBC driver since some subcommands in azdata require the SQL Server ODBC driver.|
4646

47+
## Encryption at Rest utilities on Big Data Cluster (BDC)
48+
49+
This section contains a set of notebooks useful for managing Encryption at Rest features on BDC.
50+
51+
|Name |Description |
52+
|---|---|
53+
|SOP0124 - List Keys for Encryption at Rest|Use this notebook list all HDFS keys.|
54+
|SOP0128 - Enable HDFS Encryption zones in Big Data Clusters|Use this notebook to enable HDFS Encryption Zones when upgrading to CU8 from CU6 or previous. Not required on new deployments of CU8+ or when upgrading to CU9.|
55+
|SOP0125 - Delete Key For Encryption at Rest|Use this notebook to delete HDFS encryption zone keys. __Caution!__|
56+
|SOP0126 - Backup Keys For Encryption at Rest|Use this notebook to backup HDFS encryption zone keys.|
57+
|SOP0127 - Restore Keys For Encryption at Rest|Use this notebook to restore HDFS encryption zone keys.|
4758

4859
## Managing Certificates on Big Data Clusters (BDC)
4960

5061
A set of notebooks to run a notebook for managing Certificates on Big Data Clusters.
5162

5263
|Name |Description |
53-
|---|---|---|---|
64+
|---|---|
5465
|CER001 - Generate a Root CA certificate|Generate a Root CA certificate. Consider using one Root CA certificate for all non-production clusters in each environment, as this technique reduces the number of Root CA certificates that need to be uploaded to clients connecting to these clusters. |
5566
|CER002 - Download existing Root CA certificate|Use this notebook to download a generated Root CA certificate from a cluster.|
5667
|CER003 - Upload existing Root CA certificate|CER003 - Upload existing Root CA certificate.|

‎docs/big-data-cluster/encryption-at-rest-concepts-and-configuration.md‎

Lines changed: 37 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -24,9 +24,9 @@ SQL Server Big Data Clusters stores data in the following two locations:
2424
To be able to transparently encrypt data in SQL Server Big Data Clusters, there are two possible approaches:
2525

2626
* __Volume encryption__. This approach is supported by the Kubernetes platform and is expected as a best practice for Big Data Clusters deployments. This guide does not cover volume encryption. Consult your Kubernetes platform or appliance documentation for guides on how to properly encrypt volumes that will be used for SQL Server Big Data Clusters.
27-
* __Application level encryption__. This architecture refers to the encryption of data by the application handling the data before it is written to disk. In case the volumes are exposed, an attacker wouldn’t be able to restore data artifacts elsewhere, unless the destination system also has been configured with the same encryption keys.
27+
* __Application level encryption__. This architecture refers to the encryption of data by the application handling the data before it is written to disk. In case the volumes are exposed, an attacker wouldn’t be able to restore data artifacts elsewhere, unless the destination system also has been configured with the same encryption keys.
2828

29-
The Encryption at Rest feature set of SQL Server Big Data Clusters supports the core scenario of application level encryption for the SQL Server and HDFS components.
29+
The __Encryption at Rest feature set of SQL Server Big Data Clusters__ supports the core scenario of __application level encryption__ for the __SQL Server__ and __HDFS__ components.
3030

3131
The following capabilities are provided:
3232

@@ -43,9 +43,6 @@ A Controller hosted service responsible for managing keys and certificates for t
4343
* Hadoop KMS compatibility. It acts as the key management service for HDFS component on BDC.
4444
* SQL Server TDE certificate management.
4545

46-
The following feature is not supported at this time:
47-
* *Keys Versioning support*.
48-
4946
We will reference this service as __BDC KMS__ throughout the rest of this document. Also the term __BDC__ is used to refer to the __SQL Server Big Data Clusters__ computing platform.
5047

5148
### System-managed keys and certificates
@@ -60,9 +57,9 @@ User provided keys and certificates to be managed by BDC KMS, commonly known as
6057

6158
External key solutions compatible with BDC KMS for external delegation. This capability isn't supported at this time.
6259

63-
## Encryption at rest on SQL Server Big Data Clusters CU8
60+
## Encryption at rest on SQL Server Big Data Clusters
6461

65-
SQL Server Big Data Clusters CU8 is the initial release of the Encryption at rest feature set. Read this document carefully to completely assess your scenario.
62+
Read this document carefully to completely assess your scenario.
6663

6764
The feature set introduces the __BDC KMS controller service__ to provide system-managed keys and certificates for data encryption at rest on both SQL Server and HDFS. Those keys and certificates are service-managed and this documentation provides operational guidance on how to interact with the service.
6865

@@ -75,7 +72,7 @@ The feature set introduces the __BDC KMS controller service__ to provide system-
7572
* Master instance BDC provisioned databases and user databases won’t be encrypted automatically. DBAs may use the installed certificate to encrypt any database.
7673
* Compute pool and storage pool will be automatically encrypted using the system-generated certificate.
7774
* Data pool encryption, albeit technically possible using T-SQL `EXECUTE AT` commands, is discouraged and unsupported at this time. Using this technique to encrypt data pool databases might not be effective and encryption may not be happening at the desired state. It also creates an incompatible upgrade path towards next releases.
78-
* There is no certificate rotation at this time. It is supported to decrypt and then encrypt with a new certificate using T-SQL commands if not on HA deployments.
75+
* SQL Server key rotation is achieved using standard T-SQL administrative commands. Please read [SQL Server Big Data Clusters transparent data encryption (TDE) at rest usage guide](encryption-at-rest-sql-server-tde.md) for complete instructions.
7976
* Encryption monitoring happens through existing standard SQL Server DMVs for TDE.
8077
* It is supported to back up and restore a TDE enabled database into the cluster.
8178
* HA is supported. If a database on the primary instance of SQL Server is encrypted, then all secondary replica of the database will be encrypted as well.
@@ -86,12 +83,20 @@ The feature set introduces the __BDC KMS controller service__ to provide system-
8683
* A system-generated key will be provisioned in Hadoop KMS. The key name is `securelakekey`. On CU8 the default key is 256-bit and we support 256-bit AES encryption.
8784
* A default encryption zone will be provisioned using the above system-generated key on a path named `/securelake`.
8885
* Users can create additional keys and encryption zones using specific instructions provided in this guide. Users will be able to choose the key size of 128, 192, or 256 during key creation.
89-
* In place key rotation for HDFS is not possible in CU8. As an alternative, the data can be moved from one encryption zone to another using distcp.
86+
* HDFS Encryption Zones key rotation is achieved using azdata. Please read [SQL Server Big Data Clusters HDFS Encryption Zones usage guide](encryption-at-rest-hdfs-encryption-zones.md) for complete instructions.
9087
* It's not supported to perform HDFS Tiering mounting on top of an encryption zone.
9188

92-
## Configuration guide
89+
## Encryption at Rest Administration
90+
91+
The following list contains the administration capabilities for Encryption at Rest
9392

94-
SQL Server Big Data Clusters encryption at rest is a service-managed feature and, depending on your deployment path, may require additional steps.
93+
* [SQL Server TDE](encryption-at-rest-sql-server-tde.md) management is performed using standard T-SQL commands.
94+
* [HDFS Encryption Zones](encryption-at-rest-hdfs-encryption-zones.md) and HDFS key management is performed using azdata commands.
95+
* The following administration features are performed using [Operational Notebooks](cluster-manage-notebooks.md):
96+
- HDFS key backup and recover
97+
- HDFS key deletion
98+
99+
## Configuration guide
95100

96101
During __new deployments of SQL Server Big Data Clusters__, CU8 onwards, __encryption at rest will be enabled and configured by default__. That means:
97102

@@ -101,14 +106,11 @@ During __new deployments of SQL Server Big Data Clusters__, CU8 onwards, __encry
101106

102107
Requirements and default behaviors described in the previous section apply.
103108

104-
If __upgrading your cluster to CU8__, __read carefully the next section__.
109+
__If performing a new deployment of SQL Server BDC CU8 onwards or upgrading directly to CU9, no additional steps are required__.
105110

106-
### Upgrading to CU8
111+
### Upgrade scenarios
107112

108-
> [!CAUTION]
109-
> Before upgrading to SQL Server Big Data Clusters CU8 perform a complete backup of your data.
110-
111-
On existing clusters, the upgrade process won't enforce encryption on user data and won't configure HDFS encryption zones. This behavior is by design and the following needs to be considered per component:
113+
On existing clusters, the upgrade process won't enforce new encryption or re-encryption on user data that was not already encrypted. This behavior is by design and the following needs to be considered per component:
112114

113115
* __SQL Server__
114116

@@ -119,17 +121,31 @@ On existing clusters, the upgrade process won't enforce encryption on user data
119121
* __HDFS__
120122

121123
1. __HDFS__. The upgrade process won't touch HDFS files and folders outside encryption zones.
122-
1. __Encryption Zones won't be configured__. The Hadoop KMS component won't be configured to use BDC KMS. In order to configure and enable HDFS encryption zones feature after upgrade follow the next section.
123124

124-
### Enable HDFS encryption zones after upgrade
125+
### Upgrading to CU9 from CU8 or earlier
126+
127+
No additional steps are required.
128+
129+
### Upgrading to CU8 from CU6 or earlier
130+
131+
> [!CAUTION]
132+
> Before upgrading to SQL Server Big Data Clusters CU8 perform a complete backup of your data.
133+
134+
135+
__Encryption Zones won't be configured__. The Hadoop KMS component won't be configured to use BDC KMS. In order to configure and enable HDFS encryption zones feature after upgrade follow instructions of the next section.
136+
137+
#### Enable HDFS encryption zones after upgrade to CU8
138+
139+
If you upgraded your cluster to CU8 (`azdata upgrade`) and want to enable HDFS encryption zones there are two options available:
125140

126-
Execute the following actions if you upgraded your cluster to CU8 (`azdata upgrade`) and want to enable HDFS encryption zones.
141+
* Execution of the Azure Data Studio [Operational Notebook](cluster-manage-notebooks.md) named __SOP0128 - Enable HDFS Encryption zones in Big Data Clusters__ to perform the configuration.
142+
* Script execution as described bellow.
127143

128144
Requirements:
129145

130146
- [Active Directory](active-directory-prerequisites.md) integrated cluster.
131147

132-
- [!INCLUDE [azure-data-cli-azdata](../includes/azure-data-cli-azdata.md)] configured and logged into the cluster in AD mode.
148+
- [!INCLUDE[azdata](../includes/azure-data-cli-azdata.md)] configured and logged into the cluster in AD mode.
133149

134150
Follow the following procedure to reconfigure the cluster with encryption zones support.
135151

‎docs/big-data-cluster/encryption-at-rest-hdfs-encryption-zones.md‎

Lines changed: 60 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -15,49 +15,92 @@ ms.technology: big-data-cluster
1515

1616
[!INCLUDE[SQL Server 2019](../includes/applies-to-version/sqlserver2019.md)]
1717

18-
This guide demonstrates how to use Encryption at Rest capabilities of SQL Server Big Data Clusters to encrypt HDFS folders using Encryption Zones.
18+
This guide demonstrates how to use Encryption at Rest capabilities of SQL Server Big Data Clusters to encrypt HDFS folders using Encryption Zones. It also covers HDFS key management tasks.
1919

20-
Note that there is already a default encryption zone mounted at __```/securelake```__ ready to be used. It was created with a system generated 256-bit key named __securelakekey__. This key can be used to create additional encryption zones.
20+
Note that there is a default encryption zone mounted at __```/securelake```__ ready to be used. It was created with a system generated 256-bit key named __securelakekey__. This key can be used to create additional encryption zones.
2121

2222
## <a id="prereqs"></a> Prerequisites
2323

24-
- [SQL Server Big Data Cluster CU8+](release-notes-big-data-cluster.md) with Active Directory Integration.
24+
- [SQL Server Big Data Cluster CU8+](release-notes-big-data-cluster.md) with [Active Directory](active-directory-prerequisites.md) Integration.
2525
- User with administrative privileges.
26-
27-
## Login into the name node
28-
29-
Use [Active Directory connection instructions](active-directory-connect.md) to perform cluster login. Log into the namenode (nmnode-0-0) to issue key and encryption zones commands.
30-
31-
```console
32-
kubectl exec -it -c hadoop -n <cluster_namespace> nmnode-0-0 -- /bin/bash
33-
```
26+
- [!INCLUDE[azdata](../includes/azure-data-cli-azdata.md)] configured and logged into the cluster in AD mode.
3427

3528
## Create an encryption zone using the provided system managed key
3629

3730
1. Create a HDFS folder
3831

3932
```console
40-
hdfs dfs -mkdir -p /user/zone/folder
33+
azdata bdc hdfs mkdir -p /user/zone/folder
4134
```
4235

4336
1. Issue the encryption zone create command to encrypt the folder using the __securelakekey__ key.
4437

4538
```console
46-
hdfs crypto -createZone -keyName securelakekey -path /user/zone/folder
39+
azdata bdc hdfs encryption-zone create --path /user/zone/folder --keyname securelakekey
4740
```
4841

4942
## Create a custom new key and encryption zone
5043

5144
1. Use the following pattern to create a 256-bit key.
5245

5346
```console
54-
kinit hdfs
55-
hadoop key create mydatalakekey -size 256
47+
azdata bdc hdfs key create -n mydatalakekey
5648
```
5749

5850
1. Create and encrypt a new HDFS path using the user key.
5951

6052
```console
61-
hdfs dfs -mkdir -p /user/mydatalake
62-
hdfs crypto -createZone -keyName mydatalakekey -path /user/mydatalake
53+
azdata bdc hdfs encryption-zone create --path /user/mydatalake --keyname mydatalakekey
54+
```
55+
56+
## HDFS Key rotation and encryption zone re-encryption
57+
58+
1. This creates a new version of the __securelakekey__ with new key material.
59+
60+
```console
61+
azdata hdfs bdc key roll -n securelakekey
62+
```
63+
64+
1. Re-encrypt the encryption zone associated with the key above
65+
66+
```console
67+
azdata bdc hdfs encryption-zone reencrypt --path /securelake --action start
68+
```
69+
70+
## HDFS Key and encryption zone monitoring
71+
72+
1. To monitor the status of a encryption zone re-encryption
73+
74+
```console
75+
azdata bdc hdfs encryption-zone status
6376
```
77+
78+
1. To get the encryption information about a file in an encryption zone
79+
80+
```console
81+
azdata bdc hdfs encryption-zone get-file-encryption-info --path /securelake/data.csv
82+
```
83+
84+
1. Listing all encryption zones
85+
86+
```console
87+
azdata bdc hdfs encryption-zone list
88+
```
89+
90+
1. To list all the available keys for HDFS
91+
92+
```console
93+
azdata bdc hdfs key list
94+
```
95+
96+
1. In order to create a custom key for HDFS encryption. Sizes possible are 128, 192 256. Default is 256
97+
98+
```console
99+
azdata hdfs key create --name key1 --size 256
100+
```
101+
102+
## Next steps
103+
104+
Use azdata with Big Data Clusters, see [What are [!INCLUDE[big-data-clusters-2019](../includes/ssbigdataclusters-ver15.md)]?](big-data-cluster-overview.md).
105+
106+
Use azdata with [Azure Arc enabled data services](/azure/azure-arc/data/)

‎docs/big-data-cluster/encryption-at-rest-sql-server-tde.md‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,8 +21,6 @@ Experience is in general the same as SQL Server on Linux and [standard TDE docum
2121

2222
__Unsupported features:__
2323
* Data pool encryption
24-
* Encryption key rotation for databases in a contained availability group in an [HA deployment](deployment-high-availability.md).
25-
2624

2725
## <a id="prereqs"></a> Prerequisites
2826

‎docs/big-data-cluster/release-notes-big-data-cluster.md‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -92,8 +92,8 @@ SQL Server 2019 CU9 for SQL Server Big Data Clusters, includes important capabil
9292

9393
Clusters using `mssql-conf` for SQL Server master instance configurations require additional steps after upgrading to CU9. Follow the instructions [here](bdc-upgrade-configuration.md).
9494

95-
- Improved experience for encryption at rest.
96-
- Ability to install Python packages at Spark job submission time.
95+
- Improved [!INCLUDE[azdata](../includes/azure-data-cli-azdata.md)] experience for encryption at rest.
96+
- Ability to dynamically install Python Spark packages using virtual environments.
9797
- Upgraded software versions for most of our OSS components (Grafana, Kibana, FluentBit, etc.) to ensure BDC images are up to date with the latest enhancements and fixes. See [Open-source software reference](reference-open-source-software.md).
9898
- Other miscellaneous improvements and bug fixes.
9999

@@ -248,7 +248,7 @@ SQL Server 2019 General Distribution Release 1 (GDR1) - introduces general avail
248248

249249
### HA SQL Server Database Encryption key encryptor rotation
250250

251-
- **Affected releases**: All big data cluster HA deployments irrespective of the release.
251+
- **Affected releases**: All version up to CU8. Resolved for CU9.
252252

253253
- **Issue and customer impact**: With SQL Server deployed with HA, the certificate rotation for the encrypted database fails. When the following command is executed on the master pool, an error message will appear:
254254
```
@@ -257,7 +257,13 @@ SQL Server 2019 General Distribution Release 1 (GDR1) - introduces general avail
257257
CERTIFICATE <NewCertificateName>
258258
```
259259
There is no impact, the command fails and the target database encryption is preserved using the previous certificate.
260-
260+
261+
### Enabling HDFS Encryption Zones support on CU8
262+
263+
- **Affected releases**: This scenario surfaces when upgrading specifically to CU8 release from CU6 or previous. This won't happen on new deployments of CU8+ or when upgrading directly to CU9.
264+
265+
- **Issue and customer impact**: HDFS Encryption Zones support is not enabled by default in this scenario and need to be configured using the steps provided in the [configuration guide](encryption-at-rest-concepts-and-configuration.md).
266+
261267
### Empty Livy jobs before you apply cumulative updates
262268
263269
- **Affected releases**: All version up to CU6. Resolved for CU8.

0 commit comments

Comments
 (0)