¶ Introduction
1. In the VMware Cloud Director Object Storage Extension Solution Add-On Deployment, we covered how to deploy a Solutions-Add Deployment with a Single Region.
2. In this guide, we will be demonstrating how to add an additional Cloudian HyperStore region to VCD via the VMware Cloud Director Object Storage Extension Solution Add-On Deployment.
- This guide assumes you have successfully deployed a single site VCD deployment with an operational Solutions Add-On Deployment.
¶ Requirements
| Requirement | Notes |
| Cloudian HyperStore Multi-Region Deployment | The multi-region must be configured in the same cluster. |
| Kubernetes Cluster | A separate kubernetes cluster must be used. |
| Ingress Controller in Kubernetes | The Kubernetes Cluster has an ingress controller and a LoadBalancer service deployed. |
| Cert Manager | The Kubernetes Cluster has Cert Manager installed to process and orchestrate self-signed certificates. |
| PostgreSQL |
|
| VCD OSE Solutions Add-On Instance | A new solutions-Add-On must be deployed in VCD. |
¶ Single Site Multi-Region Topology
When you deploy multi-region object storage clusters, you can deploy one VMware Cloud Director Object Storage Extension server to each object storage cluster. You can connect multiple VMware Cloud Director Object Storage Extension servers to the same VMware Cloud Director site.
With single site multi-region topology, you can deploy multiple instances of the VMware Cloud Director Object Storage Extension solution add-on and configure those instances (deployments) from the VMware Cloud Director Object Storage Extension provider portal.
The following diagram illustrates the architecture.
¶ Cloudian HyperStore Multi-Region Example
¶ OSE Solutions Add-On Deployment (POC)
Note:
- This deployment methodology should not be used in production, as this is leveraging (2) External Kubernetes Node and (2) Cloudian HyperStore Nodes.
- This article was developed so VCSP partners with staging environments can POC the Solutions Add-On deployment methodology.
- In this deployment example, we will be using the VMware Cloud Director Object Storage Extension add-on deployment to an external Kubernetes cluster.
¶ Environment
| Product Name | Version | Number of Instances |
| VMware Cloud Director | 10.5.1.23400185 | 1 |
| vCenter Server | 7.0.3 (22837322) | 1 |
| vSphere ESXi | 7.0.3, 23307199 | 3 |
| Solutions-Add On | 1.2.0-22957452 | 2 |
| Kubernetes | 1.28.2 | 2 |
| PostgreSQL | 14.11 | 2 |
| Cloudian HyperStore | 8.0.1 GA | 2 |
¶ Create a New Instance
1. Go to More > Solution Add On Management, and select NEW INSTANCE
2. Accept Licenses
3. An API Token is required to deploy the OSE Add-On.
4. In the input parameters, the following variables are required:
- Add-On Instance Name
- Kubernetes Cluster Location
- The Kubernetes cluster location is used to specify where Object Storage Extension will be installed.
- By SLZ, you specify an existing TKG cluster in the Solutions organization.
- Object Storage Extension Kubernetes Operator will be automatically installed by SLZ.
- By EXTERNAL, you manually install Object Storage Extension Kubernetes Operator onto a CNCF compliant Kubernetes cluster afterwards. Defaults to SLZ.
5. Review and Complete the Deployment
6. Wait for the Solutions Add-On Instance finish executing it's creation process.
7. You can also review the Tasks while the Solutions Add-On instance deploying is executing the required processes:
8. Ensure all the Tasks completed ✓ Successfully and that the status = ✓ Ready
¶ Configuring the Object Storage Extension
1. Select More > Object Storage Extension
2. In the Object Storage Extension, you will need to input the configuration parameters. Select CONFIGURE:
- You should have (2) Object Storage Deployments in the OSE UI.
¶ Install Operator
1. In the Configure Deployment wizard, select and input the following parameters.
- Select Install a new operator
- Cloud Director API Token
- Cluster Name
2.Select GENERATE INSTALL SCRIPT AND DOWNLOAD
- Download the YAML file and
scpit to the Kubernetes cluster - After downloading the YAML file, copy the file to a machine that supports kubectl.
- Then execute the following command to install the operator to the Kubernetes cluster. Last, click the NEXT button after the operator is installed successfully.
kubectl apply -f ose-install-ose-ext-k8s-2.yml3. Install the Kubernetes Operator
kubectl apply -f ose-install-ose-ext-k8s-2.yml
namespace/vcd-ose created
serviceaccount/vcd-ose-controller-manager created
role.rbac.authorization.k8s.io/vcd-ose-leader-election-role created
clusterrole.rbac.authorization.k8s.io/vcd-ose-manager-role created
clusterrole.rbac.authorization.k8s.io/vcd-ose-metrics-reader created
clusterrole.rbac.authorization.k8s.io/vcd-ose-proxy-role created
rolebinding.rbac.authorization.k8s.io/vcd-ose-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/vcd-ose-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/vcd-ose-proxy-rolebinding created
configmap/vcd-ose-manager-config created
secret/vcd-ose-image-creds created
secret/vcd-ose-vcd-creds created
service/vcd-ose-controller-manager-metrics-service created
deployment.apps/vcd-ose-controller-manager created4. Ensure that the pods are running:
kubectl get pods --namespace=vcd-ose
kubectl get pods --namespace=vcd-ose
NAME READY STATUS RESTARTS AGE
vcd-ose-controller-manager-c56c599bc-xkv92 2/2 Running 0 18s
vcd-ose-controller-manager-c56c599bc-xn27q 2/2 Running 0 18s5. Click NEXT
¶ Configure Service
1. API Token Name
2.CPU (2 by default)
3. Memory (2 Gi by default)
4. Replicas
5. Region
- Must match the Cloudian HyperStore Region Name
6. S3 Endpoint
7. TLS Certificate
¶ Configure Database
Note: The OSE Extension supports the following PostgreSQL DB sources:
- Specify an existing PostgreSQL instance
- PostgreSQL High Availability from VMware Data Solutions
In this deployment, I have elected to use an external PostgreSQL DB.
¶ Configure Storage Cluster
Note: The OSE Extension supports the following Object Storage Platforms:
- Amazon S3
- Cloudian Hyperstore
- Dell ECS
- OSIS (Object Storage Interoperability Service)
In this deployment, I'm leveraging Cloudian Hyperstore
1. Choose the platform (CLOUDIAN)
2. Configure S3 Service
- Default SSL Service Port:
443
3. Configure IAM Service
- Default Service Port:
16443
4. Configure Management Console
- Default SSL Service Port:
8443 - SSO Password
hsctl config get cmc.sso.sharedKey5. Configure Administration Service
- Default SSL Service Port:
19443 - Username:
sysadmin - Password: S3 Admin API Password
hsctl config get admin.auth.password¶ Method to Retrieve the SSL Signature Algorithm
#!/bin/bash
read -p "Enter the FQDN of the DNS record: " fqdn
read -p "Enter the port for $fqdn: " port
echo | openssl s_client -showcerts -servername $fqdn -connect $fqdn:$port 2>/dev/null | openssl x509 -inform pem -noout -text
¶ Deployment Configuration
Once the installer script finalizes the deployment, the OSE UI will illustrate the Configuration has been Applied and that the Server is Running
¶ Add S3 OSE Endpoint
1. Go to Platform and select ADD REGION.
2.Enter the OSE Public Endpoint and select VALIDATE
- Note: A DNS
A RecordorCNAMEshould be created before executing this step.
3. After the OSE Public Endpoint has been validated successfully, select Save to add the Region.
4. Once the Region is saved, you will be presented with (2) regions in the Storage Platform and Regions UI.
¶ Activating Region to Tenant Org
1. Select Tenants and locate the new region.
2. Select Activate
3. Select the Storage Policy
- Note: It's not recommended to Allow tenants to change storage policies – enabled by default
- This depends on the tenant's use-case.
4. Select ACTIVATE to enable the secondary region for the tenant org.
5. The tenant will now have (2) regions configured to use for the S3 service.
¶ Testing Tenant Access
1. Login into the OSE Tenant Portal
2. Ensure that you can access the new region.
3. In the main dashboard there should be (2) regions parsed in the stats.
4. Go to the Buckets tab, select the new region, and select GET STARTED:
5. . Create Bucket and SAVE
- Note: The Object Lock feature requires an object lock license (Compatible or Certified) that is required in order to create object lock buckets on the Cloudian HyperStore cluster.
- For more more information – How to Leverage the Object Lock Service with Cloudian HyperStore and OSE
6. After the bucket has been created, the bucket will list its availability to store objects/data.
¶ Testing Cross Region Replication
What is Cross Region Replication? (Cloudian HyperStore Implementation)
Like Amazon S3, HyperStore supports cross-region replication (CRR). This feature may be valuable if your HyperStore system consists of multiple service regions. With cross-region replication, a bucket in one service region can be configured so that all objects uploaded into the bucket are replicated to a chosen destination bucket in a different service region within the same HyperStore system. This feature enables a bucket owner to enhance the protection of data by having it stored in two geographically dispersed service regions. The feature is also useful in cases where a bucket owner wants to have the same set of data stored in two different regions in order to minimize read latency for users in those regions.
If you wish, you can also use the CRR feature within a single HyperStore service region, so that objects uploaded into one bucket are replicated to a different bucket in the same service region.
As is the case with Amazon S3's implementation of this feature, with HyperStore both the source bucket and the destination bucket must have "versioning" enabled in order to activate cross-region replication.
Object metadata — including any access permissions assigned to an object, and any user-defined object metadata or object tags — is replicated to the destination bucket along with the object data itself. (The exception is that if an object in the source bucket has an x-amz-expiration header, HyperStore does not replicate this header.)
As with Amazon S3, HyperStore’s implementation of the cross-region replication feature does not replicate:
- Objects that were already in the source bucket before the bucket was configured for cross-region replication (except as noted in Handling of Pre-Existing Objects)
- Objects that are encrypted with user-managed encryption keys (SSE-C) or AWS KMS managed encryption keys
- Objects that are themselves replicas from other source buckets. If you configure "bucket1" to replicate to "bucket2", and you also configure "bucket2" to replicate to "bucket3", then an object that you upload to "bucket1" will get replicated to "bucket2" but will not get replicated from there on to "bucket3". Only objects that you directly upload into "bucket2" will get replicated to "bucket3".
- Deletions of specific object versions.
- In the case of an object deletion request that specifies the object version, the object version is deleted from the source bucket but is not deleted from the destination bucket.
- In the case of an object deletion request that does not specify the object version, the deletion marker that gets added to the source bucket is replicated to the destination bucket.
¶ Replicate a Bucket
You can protect your data by automatically replicating S3 objects from one bucket to another within the same region, or across different regions.
Bucket replication copies newly created or updated objects from a source S3 bucket to a destination S3 bucket. The copied objects have the same key names and metadata (creation time, owner, or user-defined metadata).
The objects, which already exist in the bucket are not copied after you set up bucket replication.
¶ Create a Bucket Replication Rule
You can create multiple replication rules for the source bucket. Each rule indicates an individual replication regulation.
You cannot set a bucket replication rule if the object lock feature is enabled.
If you are using Cloudian HyperStore, all the replication rules of one source bucket must use the same destination bucket.
If you are using AWS S3, you can replicate one source bucket to different destination buckets.
¶ Prerequisites
- Verify that you have enabled versioning both for the source and the destination buckets.
- If you are using AWS S3, verify that you are assigned with an IAM role
- Source: Using VMware Cloud Director Object Storage Extension as a Tenant User
¶ Procedure
1. Log in to the VMware Cloud Director tenant portal.
2. From the More drop-down menu, select Object Storage.
3. From the Buckets tab, click the bucket that you want to replicate.
4. In the Bucket Replication tab, click Create a Replication Rule.
5. Enter the necessary details, and click Create.
¶ Testing CRR Across Regions
1. Select the bucket that you created in one of the regions and upload objects to it:
2. After the objects have been successfully uploaded to the bucket, you will see the list of objects that pertain to that bucket:
3. Check the bucket on the secondary service region that pertains to the Replication Rule to ensure that the objects have been replicated.
¶ Using the AWS S3 CLI
Syntax
aws --profile={PROFILE} --no-verify-ssl --endpoint-url=https://{S3-ENDPOINT}:443 s3 ls s3://{bucket} --recursive --summarizeExample Region 1
- Endpoint =
https://s3-nested-vcd.philbendeck.com - Bucket =
nested-region-bucket-crr
aws --profile=org1nested --no-verify-ssl --endpoint-url=https://s3-nested-vcd.philbendeck.com:443 s3 ls s3://nested-region-bucket-crr --recursive --summarize
2024-05-14 13:39:30 80744 25_Create_Bucket.png
2024-05-14 13:39:30 98623 26_Bucket_Created_Successfully.png
2024-05-14 13:39:30 115627 27_Bucket_CRR.png
2024-05-14 13:39:30 106362 28_Bucket_CRR_Rules.png
2024-05-14 13:39:34 93022 29_CRR_Rules_Configured.png
2024-05-14 13:39:34 12717 ose-install-ose-ext-k8s-2.yml
Total Objects: 6
Total Size: 507095Syntax
aws --profile={PROFILE} --no-verify-ssl --endpoint-url=https://{S3-ENDPOINT}:443 s3 ls s3://{bucket} --recursive --summarizeExample Region 2
- Endpoint =
https://s3-nested-vcd-1.philbendeck.com - Bucket =
nested1-bucket-crr
aws --profile=org1nested --no-verify-ssl --endpoint-url=https://s3-nested-vcd-1.philbendeck.com:443 s3 ls s3://nested1-bucket-crr --recursive --summarize
2024-05-14 13:39:30 80744 25_Create_Bucket.png
2024-05-14 13:39:30 98623 26_Bucket_Created_Successfully.png
2024-05-14 13:39:30 115627 27_Bucket_CRR.png
2024-05-14 13:39:30 106362 28_Bucket_CRR_Rules.png
2024-05-14 13:39:34 93022 29_CRR_Rules_Configured.png
2024-05-14 13:39:34 12717 ose-install-ose-ext-k8s-2.yml
Total Objects: 6
Total Size: 507095