Z4K Administration
Workflow
After deploying Zerto for Kubernetes, create a VPG, configure one-to-many (optional), tag checkpoints, and then test failover.
- Create a VPG
- Update Existing VPGs
- Configure One-to-Many
- Tag a Checkpoint
- Test Failover
Then, you can perform one of the following:
- Perform a Failover
- Restore a Single VPG
- Move operation
- Configure Long-term Retention in Kubernetes Environments > Z4K supports backing up Kubernetes workloads and their data to a Long-term Repository and restoring them from the Long-term Repository to the original site, or to a different site/namespace.
- Log Retention > Log collection occurs automatically, and the logs are uploaded to Amazon S3. You can also collect logs ad hoc.
- Protect Ingress Controller Resources > Z4K supports replicating Ingress Controller Resources so networking configuration can be replicated and easily deployed on the recovery site.
- Taints and Tolerations > Z4K supports taints and tolerations configuration for nodes and pods.
- Tweaks > How to view and configure tweaks and values.
Creating a VPG
- Create a .yaml file to represent a VPG. In the following example the VPG webApp1:
- Is configured to self replicate to its source cluster.
- Will use the storage class goldSC.
- SLA is 12 hours of history.
- The Journal can expand up to 160 GB to meet the history requirement.
Note: It is not mandatory to configure the Journal disk size (JournalDiskSizeInGb) and history (JournalHistoryInHours); they have default values of 2 GB and 8 hours respectively.
---
apiVersion: z4k.zerto.com/v1
kind: vpg
spec:
JournalDiskSizeInGb: 160
JournalHistoryInHours: 12
Name: “webApp1”
RecoveryStorageClass: GoldSC
SourceSite:
Id: prod_site
TargetSite:
Id: dr_site
- Annotate Kubernetes entities to include them in the VPG.
- A VPG can contain a selection of entities like stateful sets, deployments, services, secrets and configmaps.
- Applications consisting of several components with inter-dependencies (for example secrets and deployments), should all be tagged with the same VPG annotation in order for the Failover operation to succeed.
- To include an entity in a VPG, you must annotate the entity with the VPG name.
Use the following example of deployment protection for guidelines.
---
kind: Deployment
metadata:
annotations:
vpg: "webApp1 /<VPG name as configured in VPG.yaml>"
labels:
app: debian
name: debian
spec:
replicas: 1
selector:
matchLabels:
app: debian
spec:
containers:
-
image:
command:
- /usr/bin/tail
- "-f"
- /dev/null
debian: stable
volumeMounts:
-
mountPath: /var/gil1
name: external1
name: debian1
volumes:
-
claimName: my-vol1-debian-5to6
name: external1
persistentVolumeClaim: ~
template:
metadata:
labels:
app: debian
Updating Existing VPGs
In case you need to update an existing VPG you need to create a yaml file as you did it for create VPG, with additional section: "metadata". The "metadata" section should contain the VPG name and the namespace id:
---
apiVersion: z4k.zerto.com/v1
kind: vpg
spec:
JournalDiskSizeInGb: 160
JournalHistoryInHours: 12
RecoveryStorageClass: GoldSC
SourceSite:
Id: prod_site
TargetSite:
Id: dr_site
metadata:
name: <VPG name>
namespace: <z4k namespace>
Update kubernetes command:
Configuring One-to-Many
Z4K can protect the same entity multiple times.
The recommended configuration is to replicate no more than 3 VPGs (inbound & outbound). Exceeding this configuration is possible, bearing in mind that performance degradation may appear if POD/Cluster resources are being overutilized.
One-to-Many serves two main goals: 1. Enables high availability (HA) for protected VPGs. 2. Enables Continuous Data Protection (CDP) during migration with a local and a remote copy.
To protect the same entity multiple times:
- Add all of the VPG names in the annotation field with a ';' delimiter sign:
---
kind: Deployment
metadata:
annotations:
vpg: "webApp1Self; webApp1Site2; webApp1Site3 /<VPG names as configured in VPG yaml files>"
labels:
app: debian
name: debian
* * *
- Create the VPG by running the command:
Viewing VPG Status
To display the VPG status as well as an overview of which entities are protected within the VPG, the VPG’s SLA and it's settings, run the command:
Tagging a Checkpoint
To tag a checkpoint, run the command:
Displaying Available Checkpoints
- To display a list of all available checkpoints for a VPG, including properties like the checkpoint ID, run the command:
Where:
| Parameter | Value | Mandatory Y/N |
|---|---|---|
| vpg | VPG name | Y |
| minAge | Minutes: m (example: 5m), Hours: h (example: 5h) , Days: d (example: 5d) | N |
| maxAge | Minutes: m (example: 5m), Hours: h (example: 5h) , Days: d (example: 5d) | N |
Testing Failover
- To test failover, run the command:
Where [checkpoint ID] can be either an ID, or enter "latest" for the latest checkpoint.
- To stop the test run, run the command:
Performing a Failover
- To failover run the command:
Where [checkpoint-id] can be either an ID, or enter "latest" for the latest checkpoint.
- To commit the failover, run the command:
- To rollback the failover, run the command:
Restoring a Single VPG
On a single cluster deployment, only the restore and failover test operations are available. Failover is not available on single cluster deployments.
- To restore a single VPG, run the command:
Where [checkpoint-id] can be either an ID, or enter latest, for the latest checkpoint.
- To commit the restore, run the command:
- To rollback the restore, run the command:
Move Operation
The move operation allows moving the deployments to it's recovery site without preserving them on the protected site. The move operation consists of 3 possible commands: 1. move - Start move live (test before commit) 2. commit-move - Commit move 3. rollback-move - Rolling back Move test before commit
The following is used for the above move commands:
At that point the VPG state will be updated to StartingMove If a checkpoint is not tagged use the value 'latest' Once move operation is completed, the VPG status will be updated to MoveBeforeCommit The VPG will go back into protecting state in it's protected site without being commited to the recovery site. VPG status will be changed to CommittingMove. Onc completed, the VPG will be commited, deployment will now exist on the recovery site and VPG will be removed from the environment.Long-term Retention in Kubernetes Environments
Z4K supports backing up Kubernetes workloads and their data to a long-term repository (LTR) and restoring them from the LTR to the original site, or to a different site.
Supported Repository Types
Z4K supports two LTR repository types:
- AWS S3
- Azure Blob Storage
To configure LTR for your Kubernetes environment, use the following procedures:
- Back up the VPG
- Manually trigger a Backup
- Schedule LTR Backups
- Restore the VPG from a LTR
Backing Up a VPG
To backup a VPG to a target LTR repository, create the VPG and update the VPG yaml file (vpg.yaml) with the LTR repository type.
Use the following examples as guidelines.
Example vpg.yaml File - Backing Up to AWS S3
---
apiVersion: z4k.zerto.com/v1
kind: vpg
spec:
BackupSettings:
IsCompressionEnabled: true
RepositoryInformation:
AwsBackupRepositoryInformation:
Bucket: <BucketName>
CredentialsSecretReference:
Site:
Id: "<site id where is secret installed>"
Id:
Name: mysecret
NamespaceId: ~
NamespaceName: <Secret Namespace>
Region: <BucketRegion>
BackupTargetType: AmazonS3
JournalDiskSizeInGb: 8
JournalHistoryInHours: 8
Name: "<VPG Name>"
RecoveryStorageClass: <RecoveryStorageClass>
SourceSite:
Id: <SourceSite>
TargetSite:
Id: <TargetSite>
Example vpg.yaml File - Backing Up to Azure Blob Storage
---
apiVersion: z4k.zerto.com/v1
kind: vpg
spec:
BackupSettings:
IsCompressionEnabled: true
RepositoryInformation:
AzureBackupRepositoryInformation:
CredentialsSecretReference:
Site:
Id: "<site id where is secret installed>"
Id:
Name: mysecret
NamespaceId: ~
NamespaceName: <Secret Namespace>
DirectoryId: c659fda3-cf53-43ad-befe-776ee475dcf5
StorageAccountName: <Storageaccount>
BackupTargetType: AzureBlob
Name: <VPG Name>
RecoveryStorageClass: <RecoveryStorageClass>
SourceSite:
Id: <SourceSite>
TargetSite:
Id: <TargetSite>
- The AWS S3 access key and secret key should be captured as a Kubernetes secret, whose name appears in the vpg.yaml file. In the example above, this is mysecret.
-
The secret must contain a data item for the AccessKey and a data item for the SecretKey, and can be created in any site to which Zerto Kubernetes Manager has access. In the example above, this is site1. Example vpg.yaml File - Backing Up to Azure Blob Storage
-
The Azure application id and client secret should be captured as a Kubernetes secret, whose name appears in the vpg.yaml file. In the example above, this is mysecret.
- The secret must contain a data item for the ApplicationId and a data item for the ClientSecret, and can be created in any site to which Zerto Kubernetes Manager has access. In the example above, this is site1.
Manually Trigger a Backup
You must manually trigger a backup to create a retention set.
- To manually trigger a backup, run the command:
A backup task is triggered. When the task completes successfully, a retention set is created.
- To access the generated retention set ID (backupset ID), run the command:
Scheduling LTR Backups
To schedule LTR backups, add SchedulingAndRetentionSettings to the VPGs BackupSettings.
Use the following example as a guideline.
---
apiVersion: z4k.zerto.com/v1
kind: vpg
spec:
BackupSettings:
IsCompressionEnabled: true
RepositoryInformation:
AwsBackupRepositoryInformation:
Bucket: <BucketName>
CredentialsSecretReference:
Site:
Id: "<site id where is secret installed>"
Id:
Name: mysecret
NamespaceId: ~
NamespaceName: <Secret Namespace>
Region: <BucketRegion>
BackupTargetType: AmazonS3
SchedulingAndRetentionSettings:
PeriodsSettings:
-
ExpiresAfterUnit: Years
ExpiresAfterValue: 1
Method: Full
PeriodType: Yearly
-
ExpiresAfterUnit: Months
ExpiresAfterValue: 12
Method: Full
PeriodType: Monthly
-
ExpiresAfterUnit: Weeks
ExpiresAfterValue: 4
Method: Full
PeriodType: Weekly
-
ExpiresAfterUnit: Days
ExpiresAfterValue: 7
Method: Incremental
PeriodType: Daily
JournalDiskSizeInGb: 8
JournalHistoryInHours: 8
Name: <VPG Name>
RecoveryStorageClass: <RecoveryStorageClass>
SourceSite:
Id: <SourceSite>
TargetSite:
Id: <TargetSite>
Important Considerations for SchedulingAndRetentionSettings
- There must be at least one Full Method.
- There can be no more than one Incremental Method.
- A Full Method cannot be followed by an Incremental Method. In other words, if there is a Full Method, it should be the last in the chain.
- Zerto Kubernetes Manager schedules backups and expirations as needed.
Restoring a VPG from LTR
To restore a VPG from LTR, run the command:
Where:
| Parameter | Comment |
|---|---|
| site-id | Identifier of the site to which you want to restore. |
| storage-class | Storage class, with which persistent volumes for the restored data is created. |
| namespace | namespace in which you want the restored Kubernetes entities to be created. |
These parameters give you flexibility in restoring a VPG to the original site, or to a different site or namespace. You can also restore from a repository that has VPGs backupsets from a different site and restore them to a new site using the parameter backupset-id.
When the restore task completes successfully, Kubernetes entities with the prefix "res-" are created in the specified site and namespace.
Log Retention
Zerto for Kubernetes uses Apache log4net™ logging framework to generate, collect and output log statements.
Log Volume Size
- A ZKM service can have up to 200 log files.
- A ZKM-PX service can have up to 50 log files.
Log Rotation
- Log files are automatically archived when the log file size exceeds 10MB.
- Archived log files can be deleted manually.
Required Storage
- A log file is 0.6MB after automatic archiving.
Therefore: - A ZKM service needs 200 x 0.6MB = ~120MB - A ZKM-PX service needs 50 x 0.5MB = ~30MB
Log Location
Logs are stored locally on zkm/zkm-px pods /logs. The ad hoc logs are uploaded to Amazon and stored in S3 bucket.
Ad Hoc Log Collection
Use one of the following methods to generate logs.
Ad Hoc Log Collection Method 1
Run the following command, which runs a script on the ZKM-PX in the background:
Ad Hoc Log Collection Method 2
- Connect to the pod using the command:
- Run one of the following scripts:
Protecting Ingress Controller Resources
There are 2 options to configure Ingress Controller Resources with a deployment for protection:
- Configure protection when creating a new deployment, as illustrated in the YAML example below:
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
vpg: website-vpg1
name: minimal-ingress
spec:
ingressClassName: nginx-example
rules:
-
http: ~
paths:
-
backend:
name: test
port:
number: 80
service: ~
path: /testpath
pathType: Prefix
-OR-
- Configure protection using an existing deployment by running the command:
Protecting Ingress Controller Resources
To display Ingress controller annotation run the following command:
$ kubectl get pod <ingress_pod_id> -n <namespace> -o json | jq .metadata.annotations
{
"kubernetes.io/psp": "eks.privileged",
"vpg": "website-vpg1"
}
$ kubectl get vpg -n <namespace>
NAME STATE SYNC SOURCE TARGET STATEFULSETS DEPLOYMENTS SERVICES CONFIGMAPS SECRETS INGRESSES NUMCP RPO JOURNALHISTORY JOURNALSIZEMB
website-vpg1 Protecting node-1 node-1 0 1 0 0 0 1 10 6 seconds 49 seconds 2
Taints and Tolerations
Z4K supports taints and tolerations configuration for nodes and pods.
You can add a node annotation called “zertorole” to force VRA installation behaviour.
- Add the value “worker” to force VRA installation.
- Add the value “master” to force skipping VRA installation.
Adding and Updating Zertorole Annotation
- To force VRA installation, add or update the annotation on the current node with the command:
- To skip VRA installation, add or update the annotation on the current node with the command:
Note: Use --overwrite to update an annotation that already exists.
Removing Zertorole Annotation
To remove a zertorole annotation use a minus - sign at the end of the annotation:
<span class="Note">Note: Nodes that are tainted with a taint that does not allow VRA installation ("NoSchedule" effect) cannot have it's deployments protected.</span>
- Taints and tolerations are not replicated.
- When Taints and Tolerations are in use they must be predefined in pods and nodes for VPG protection, before recovery oeprations take place.
- For more info, see Taints and Tolerations
Tweaks
Z4K allows users to view and customize tweaks for Z4K components using the following commands:
- To view existing tweaks and values use the command: Example
- To set a new value for an existing tweak, use the command:
Example
- To review the tweaks value and confirm settings, use the command: Example