Link Search Menu Expand Document

Troubleshooting

Table of Contents

In this section you can find the most frequent failure modes and how to work around them.

Default EBS encryption enabled with a customer-managed KMS key Elastio does not have permission to use

EBS volume backup fails if it is encrypted with a customer-managed KMS key that Elastio does not have access to. This also affects EC2 backups, as well as scans of EC2 instances, EBS volumes, and AWS Backup recovery points, in case they are encrypted in a similar way.

This manifests as the following error message in backup or scan jobs that you can see in your Tenant on Jobs page or in Elastio CLI:

"Ebs operation has failed: Failed to access volume. Most likely there is a problem with encrypted volume and permissions. Check permissions for KMS key: {KMS key ARN}"

Same issue would manifest differently in different job kinds, for instance an example of AWS EBS snapshot scan error:

"Mount of AWS EBS snapshots '{snapshot-id}' failed. Details: Failed to mount '{snapshot-id}' device due to error: Failed to decrypt volume. KMS Key: {KMS key ARN}"

It happens because Elastio’s background job task roles don’t have permission to use the KMS key used for the EBS encryption. It might occur if there is an encryption policy for all newly created EBS volumes or if a single volume was created with a KMS key that Elastio cannot access.

In case such errors are observed in your environment, you would receive an email with the list of affected assets and KMS keys that you will need to adjust. Alternatively, you can locate the KMS key ID on the Jobs / Cloud Connector page, accessible through the navigation bar in the top right corner of the Elastio Tenant page. Filter the jobs list to only display Failed jobs, then locate the one(s) with the errors like in the example(s) above and copy KMS key ID(s). Save them somewhere for the following steps.

Now you’ll need to locate the KMS key used for encryption in your AWS console. In the AWS console, go to the Key Management Service (KMS) section, Customer managed keys tab. You can search the keys by alias, key ID or tags. The instructions differ depending on the way the key was created. In case the KMS key was created manually, when viewing the key, it will have Key Users section that needs to be modified. In case the Key was created programmatically (via AWS CLI, Terraform etc), only Key policy can be viewed and modified.

Manually created KMS keys

Once the key is located, press its alias or key ID to view key details. Scroll down the Key policy tab until you reach “Key Users”.

In the “Key users” section press the “Add” button.

Figure 1.1 Key users

Figure 1.1 Key users

If you are doing EBS/EC2 backups, in the list search for the 2 roles beginning with (you’ll need to add the one without “Execution” in the name):

elastio-account-level-stack-ebsBgJobs
elastio-account-level-stack-ec2BgJobs

For AWS backup imports, search for the role beginning with:

  • EBS:
elastio-account-level-sta-awsBackupRpImportBgJob
  • EC2:
elastio-account-level-sta-awsBackupRpEc2ImportBgJob

For scan jobs of AWS Backup RPs the roles are as follows (you’ll need to add the one(s) without “Execution” in the name):

  • EC2:
elastio-account-level-sta-awsBackupRpEc2ScanBgJob
  • EBS:
elastio-account-level-sta-awsBackupRpEbsScanBgJob
  • EFS
elastio-account-level-sta-awsBackupRpEfsScanBgJob
  • S3
elastio-account-level-stack-awsS3ScanBgJob

For direct EC2 scan:

elastio-account-level-stack-awsEc2ScanBgJob

For direct EBS scan:

elastio-account-level-stack-awsEbsScanBgJob

Add the role(s) to the key users like so.

Figure 1.2 Add key users

Figure 1.2 Add key users

Once you add these users, the Key policy section will look like this:

Figure 1.3 Updated key policy

Figure 1.3 Updated key policy

Programmatically created KMS keys

If the key was created programmatically, only its policy can be viewed and modified. An example of what the KMS key policy looks like is displayed below:

{
    "Id": "KMS-KEY-ID",
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Enable IAM User Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "IAM-ARN"
            },
            "Action": "kms:*",
            "Resource": "*"
        }
    ]
}

In order to remediate the permission issue, you will need to modify the KMS key policy to grant the appropriate Elastio IAM role the necessary permissions to use the key. Pick the role depending on the operation you need to perform from the list below.

If you are doing EBS/EC2 backups, in the list search for the 2 roles beginning with (you’ll need to add the one without “Execution” in the name):

elastio-account-level-stack-ebsBgJobs
elastio-account-level-stack-ec2BgJobs

For AWS backup imports, search for the role beginning with:

  • EBS:
elastio-account-level-sta-awsBackupRpEbsImportBgJob
  • EC2:
elastio-account-level-sta-awsBackupRpEc2ImportBgJob

For scan jobs of AWS Backup RPs the roles are as follows (you’ll need to add the one(s) without “Execution” in the name):

  • EC2:
elastio-account-level-sta-awsBackupRpEc2ScanBgJob
  • EBS:
elastio-account-level-sta-awsBackupRpEbsScanBgJob
  • EFS
elastio-account-level-sta-awsBackupRpEfsScanBgJob
  • S3
elastio-account-level-stack-awsS3ScanBgJob

For direct EC2 scan:

elastio-account-level-stack-awsEc2ScanBgJob

For direct EBS scan:

elastio-account-level-stack-awsEbsScanBgJob

Add the role with the permissions to use the key to the policy, using the policy language below as a template:

Please, note: You must replace the $ELASTIO_ROLE_ARN placeholders in the example below with the actual IAM role name, before adding it to your key’s policy.

{
            "Sid": "Allow use of the key by Elastio",
            "Effect": "Allow",
            "Principal": {
                "AWS": "$ELASTIO_ROLE_ARN"
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Allow Elastio to attach persistent resources encrypted by with this key",
            "Effect": "Allow",
            "Principal": {
                "AWS": "$ELASTIO_ROLE_ARN"
            },
            "Action": [
                "kms:CreateGrant",
                "kms:ListGrants",
                "kms:RevokeGrant"
            ],
            "Resource": "*",
            "Condition": {
                "Bool": {
                    "kms:GrantIsForAWSResource": "true"
                }
            }
        }

Once these steps are completed, this error should not occur anymore for the keys that were modified. Please, keep in mind, that roles for different types of jobs are different, so you must pay careful attention to job kind before modifying the KMS key. It’s possible depending upon your use case that multiple job roles must be granted permissions to use a given key.

VPC misconfigurations

Deploying a vault in a private VPC without an Internet Gateway for all subnets in the VPC with “auto-assign public IPv4” address enabled results in the AWS Batch job for the backup getting stuck in Runnable state. An error message isn’t delivered.

Currently Elastio shows the VPC misconfiguration warning upon deployment if the compute environment of the background job has a subnet with public_ip_on_launch toggle disabled. If the subnet has public_ip_on_launch enabled, the VPC misconfiguration warning is not displayed. Additionally, Amazon ECS (and AWS Batch respectively) doesn’t work as expected if the public subnet doesn’t have IGW interface attached to the subnet and a routing rule for the IGW.

Note: The VPC misconfiguration warning displayed is The Cloud Connector will be deployed into a private subnet and will need manual configuration. Please see documentation for more details. One of our cloud team members will reach out with assistance. The warning can be safely ignored if the subnets you deploy the vault into have a NAT gateway with access to the Internet, in particular access to all AWS API endpoints.

To resolve the networking issues, create a private subnet with a NAT gateway. The instructions below will help you to do that:

  1. Create a public NAT in the public subnet.
  2. Go to the private subnet route table and route 0.0.0.0/0 through the NAT.
  3. The public subnet route table should have a route 0.0.0.0/0 to IGW interface.

Note: The selected VPC should have a subnet with “auto-assign public IPv4” address enabled, as well as a subnet in every Availability Zone within the region. If you plan to use Elastio only within the VPC, deploy the vault entirely in private subnets with NAT gateways. In this case the vault will only be accessed from within the private subnet. Mount backups from your workstations will become possible only after setting up a VPN tunnel into the VPC with a network path from the VPN tunnel to the private subnets, where the vault is running.

Adding URLs accessed by Elastio to the whitelist

If the VPC in the AWS organization is configured in a way that requires whitelisting addresses accessed from within the environment, the following URLs need to be whitelisted:

To allow Elastio communicate to AWS APIs and pull Elastio container images:

  • *.amazonaws.com

  • public.ecr.aws

  • *.cloudfront.net

To grant access to the ransomware database Elastio uses for iScan:

  • iscan.elastio.com/

To enable connection to malware definitions used by iScan:

  • oemeval001-94027265-32ef-489b-ae6c-27b60ba758fd.2d7dd.cdn.bitdefender.net

  • ocsp.digicert.com/

  • elastio-df5cb85e-a1f5-44a4-8e15-4566fdfbf679.2d7dd.cdn.bitdefender.net

  • cs1197.wpc.alphacdn.net

For all but the first two URLs, port 80 needs to be enabled alongside port 443.

Enable VSS for an AWS EC2 instance

Without the VSS being enabled for the Windows instances, only crash-consistent backups can be created. This kind of backup can create snapshots with inconsistent data on the systems under heavy load, so the data cannot be properly scanned for threats or restored from such recovery points. To avoid such risks it is recommended to enable the VSS in AWS. This instruction will walk you through the necessary steps to configure VSS and enable it on the instances you need it on.

First you’ll need to create a policy with necessary permissions:

  • Go to IAM section of the AWS console.
  • In the navigation pane, select Policies, and then - Create policy.
  • On the Create policy page, choose the JSON tab, and then replace the default content with the following JSON policy.
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "ec2:CreateTags",
            "Resource": [
                "arn:aws:ec2:*::snapshot/*",
                "arn:aws:ec2:*::image/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:CreateSnapshot",
                "ec2:CreateImage",
                "ec2:DescribeImages",
                "ec2:DescribeSnapshots"
            ],
            "Resource": "*"
        }
    ]
}
  • Press Review policy.
  • For Name, enter VssSnapshotRole.
  • Press Create policy.

Then you’ll need to create a role. To create the role do the following:

  • In the navigation pane, select Roles, and then - Create role.
  • Under Select type of trusted entity, choose AWS Service.
  • Choose EC2, and then choose Next: Permissions.
  • In the list of policies, choose the box next to AmazonSSMManagedInstanceCore and press Next: Tags.
  • For Role name, enter the name VssSnapshotRole.
  • Select Create role. The system returns you to the Roles page.
  • Select the role that you just created and press Attach policies.
  • Search for and choose the box next to the policy you created in the previous procedure VssSnapshotRole and press Attach policy.

Alternatively, you can add the AmazonSSMManagedInstanceCore and VssSnapshotRole to the role already attached to the instances.

Then you’ll need to install the VSS components to the Windows EC2 instances:

  • Open the AWS Systems Manager console.
  • In the navigation pane, select Run Command.
  • For Command document, choose AWS-ConfigureAWSPackage.
  • For Command parameters, do the following:
    • Verify that Action is set to Install.
    • For Name, enter AwsVssComponents.
    • For Version, leave the field empty so that Systems Manager installs the latest version.
    • For Targets, identify the instances on which you want to run this operation selecting instances manually.
  • Press Run.

The VSS should become enabled for the instances that you attach the newly created role to.

Wrong AWS account or region

The following error message is displayed when trying to mount a recovery point with a command from the Elastio Tenant in the incorrectly configured CLI that reaches out to the wrong AWS account or region:

ubuntu@ip-172-31-22-46:~$ sudo -E elastio mount rp --rp rp-01g4zdj9c8ay1kx08y39dyq426
Recovery point was not found in vault `{any}` for ID `rp-01g4zdj9c8ay1kx08y39dyq426`
Location:
/home/elastio/.cargo/registry/docs/dl.cloudsmith.io-2dc4edbccd98c64c/cheburashka-4.3.2/docs/logging/mod.rs:592:18

To resolve the issue, run aws sts get-caller-identity to ensure that the account number is the same as the RP’s account number. If those match, run aws configure and check that the region of the CLI matches the region of the recovery point.

Cloud Connector is not present in the selected region

The following error message is displayed when attempting to perform EBS or EC2 backup if the AWS CLI is configured for another region or AWS account:

user@ThinkPad-T15-Gen-1:~$ elastio ec2 backup --instance-id i-0ea486582654d7562
Failed to fetch data for a default vault

Caused by:
   0: Catalog service error
   1: Failed to invoke remote function elastio-catalog-service-read
   2: Function not found: arn:aws:lambda:us-east-1:421555810956:function:elastio-catalog-service-read
   3: Function not found: arn:aws:lambda:us-east-1:421555810956:function:elastio-catalog-service-read

Location:
    /home/elastio/.cargo/registry/docs/dl.cloudsmith.io-2dc4edbccd98c64c/cheburashka-4.2.12/docs/logging/mod.rs:588:18
user@user-ThinkPad-T15-Gen-1:~$

The following error message is displayed when attempting to perform file, block or stream backup in a region that does not have a Cloud Connector installed:

user@user-ThinkPad-T15-Gen-1:~$ elastio file backup ./123.pub
Failed to invoke remote function elastio-jobs-status-service

Caused by:
   0: Function not found: arn:aws:lambda:us-east-1:421555810956:function:elastio-jobs-status-service
   1: Function not found: arn:aws:lambda:us-east-1:421555810956:function:elastio-jobs-status-service

Location:
    /home/elastio/.cargo/registry/docs/dl.cloudsmith.io-2dc4edbccd98c64c/cheburashka-4.2.12/docs/logging/mod.rs:588:18

Run elastio version to ensure that Cloud Connector is deployed in the current region. Run aws sts get-caller-identity to check if the account configured for the AWS CLI is correct. If the region is wrong, change it by running aws configure. In case the wrong account is reached, reconfigure the AWS CLI with the correct keys or profile.

The assets are not displayed in the Elastio Tenant

After logging into an Elastio Tenant and connecting an AWS account, no assets are displayed on the Assets page of the Tenant.

Make sure that there are some EC2 and/or EBS assets in the dedicated account and region - if there are none in the AWS account none will be displayed through Elastio.

A default vault is missing

When attempting to run a backup or a restore through the Elastio CLI, the following error is displayed:

Failed to find default vault. Specify '--vault' option or set the default vault and run again

Location:
    /home/elastio/.cargo/registry/docs/dl.cloudsmith.io-2dc4edbccd98c64c/cheburashka-4.4.0/docs/logging/mod.rs:593:18

Check if the default vault is set by running elastio vault default. If the output is empty, run elastio vault list pick a vault and run elastio vault default vault-name to set some vault as default. Alternatively, you can navigate to this account through Sources page in Elastio Tenant, select a vault and set it as default. In case the vault list comes back empty, create a vault through Sources page in your Tenant.

Note: For the time being, please limit the number of vaults to maximum 4 per region.

Note: You can skip using the --vault flag when you set your default vault. It is no longer necessary to add the vault name every time you run a command that requires the vault. The system will use the default vault if none is specified.

Integrity scan fails

The following error message is displayed when attempting to run iscan against the recovery point that cannot be mounted as the target EBS volume does not have a filesystem.

Figure 1.2: Integrity Scan error message

Figure 1.2: Integrity Scan error message

Ths error is most likely with new EBS volumes created through AWS Console manually that were not formatted. To resolve the issue, attach said EBS volume to an instance and create a filesystem on it.

Deploying the Elastio CFN in the wrong AWS account

When deploying the Elastio CFN in the wrong AWS account, the following error message appears.

Figure 1.3: The Elastio CFN deploy error message

Figure 1.3: The Elastio CFN deploy error message

To resolve the issue, switch to the required resource account on the AWS Management Console.

EC2 backup fails

The following error message is displayed when attempting to perform AWS EC2 backup.

Figure 1.4: AWS EC2 backup error message

Figure 1.4: AWS EC2 backup error message

The same error in the logs is displayed as follows:

{
    "timestamp": "2022-05-28T07:03:05.344Z",
    "level": "ERROR",
    "fields": {
        "error": [
            "Failed to get volume state, reason Unknown(BufferedHttpResponse {status: 400, body: \"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?>\\n<Response><Errors><Error><Code>InvalidVolume.NotFound</Code><Message>The volume 'vol-05e873f1a99eec441' does not exist.</Message></Error></Errors><RequestID>fe978390-f1f7-4d39-9e0f-b71579561d06</RequestID></Response>\", headers: {\"x-amzn-requestid\": \"fe978390-f1f7-4d39-9e0f-b71579561d06\", \"cache-control\": \"no-cache, no-store\", \"strict-transport-security\": \"max-age=31536000; includeSubDomains\", \"vary\": \"accept-encoding\", \"content-type\": \"text/xml;charset=UTF-8\", \"transfer-encoding\": \"chunked\", \"date\": \"Sat, 28 May 2022 07:03:05 GMT\", \"connection\": \"close\", \"server\": \"AmazonEC2\"} })"
        ],
        "message": "VolumeAwsImpl::state() failed"
    },
    "target": "elastio_agentless::aws::volume",
    "filename": "cli/elastio-agentless/docs/aws/volume.rs",
    "line_number": 415,
    "spans": [
        {
            "availability_zone": "us-east-1c",
            "instance_type": "",
            "op_type": "create_volume_from_snapshot",
            "platform": "aws",
            "volume_iops": 100,
            "volume_size": 0,
            "volume_type": "gp3",
            "name": "AgentlessLabels"
        }
    ],
    "threadId": "ThreadId(6)"
}

The AWS EC2 has a volume encrypted with a key, which Elastio does not have access to. The issue is going to be fixed in the upcoming release.

Block mount fails

When attempting to perform block mount on the supported Linux-based OS without the NBD kernel module installed, the following error message is displayed.

[user@linux ~]$ sudo -E elastio mount rp --rp rp-01g8e24sp9twa14brpan41bq5f

Unknown: Failed to load `nbd` module. Ensure your Linux kernel is compiled with NBD support.

Caused by:

command ["modprobe", "nbd"] exited with code 1

Location:

/home/elastio/.cargo/registry/docs/dl.cloudsmith.io-2dc4edbccd98c64c/cheburashka-4.4.0/docs/logging/mod.rs:593:18

To resolve the issue, make sure that the nbd-client package is installed. Furthermore, make sure that the elastio mount command is executed as root or with sudo.