Create an AWS-integrated custom cloud
Create a custom cloud for BYOC in your Aiven organization to better address your specific business needs or project requirements.
To configure a custom cloud in your Aiven organization and prepare your AWS account so that Aiven can access it:
- In the Aiven Console or with the Aiven CLI client, you specify new cloud details to generate a Terraform infrastructure-as-code template.
- You download the generated template and deploy it in your AWS account to acquire IAM Role ARN (Amazon Resource Name).
- You deploy your custom cloud resources supplying the acquired IAM Role ARN to the Aiven platform, which gives Aiven the permissions to securely access your AWS account, create resources, and manage them onward.
- You select projects that can use your new custom clouds for creating services.
- You add contact details for individuals from your organization that Aiven can reach out to in case of technical issues with the new cloud.
Before you start
Prerequisites
- You have enabled the BYOC feature.
- You have an active account with your cloud provider.
- Depending on the tool to use for creating a custom cloud:
- Console: Access to the Aiven Console or
- CLI:
- Aiven CLI client installed
- Aiven organization ID from the output of the
avn organization listcommand or from the Aiven Console > User information > Organizations.
- You have the organization admin role in your Aiven organization.
- You have Terraform installed.
- You have required IAM permissions.
IAM permissions
You need cloud account credentials set up on your machine so that your user or role has required Terraform permissions to integrate with your cloud provider.
Show permissions required for creating resources for bastion and workload networks
{
"Statement": [
{
"Action": [
"iam:AttachRolePolicy",
"iam:CreateRole",
"iam:DeleteRole",
"iam:DeleteRolePolicy",
"iam:GetRole",
"iam:GetRolePolicy",
"iam:ListAttachedRolePolicies",
"iam:ListInstanceProfilesForRole",
"iam:ListRolePolicies",
"iam:PutRolePolicy",
"iam:TagRole",
"iam:UpdateAssumeRolePolicy"
],
"Effect": "Allow",
"Resource": "arn:aws:iam::*:role/cce-*-iam-role"
},
{
"Action": [
"ec2:DescribeAddresses",
"ec2:DescribeAddressesAttribute",
"ec2:DescribeAvailabilityZones",
"ec2:DescribeInternetGateways",
"ec2:DescribeNatGateways",
"ec2:DescribeNetworkInterfaces",
"ec2:DescribePrefixLists",
"ec2:DescribeRouteTables",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSecurityGroupRules",
"ec2:DescribeStaleSecurityGroups",
"ec2:DescribeSubnets",
"ec2:DescribeVpcs",
"ec2:DescribeVpcEndpoints",
"ec2:DescribeVpcAttribute",
"ec2:DescribeTags"
],
"Effect": "Allow",
"Resource": [
"*"
],
"Sid": "Describe"
},
{
"Action": [
"ec2:CreateTags"
],
"Condition": {
"StringEquals": {
"ec2:CreateAction": [
"AllocateAddress",
"CreateInternetGateway",
"CreateNatGateway",
"CreateRoute",
"CreateRouteTable",
"CreateSecurityGroup",
"CreateSubnet",
"CreateVpc",
"CreateVpcEndpoint"
]
}
},
"Effect": "Allow",
"Resource": [
"*"
],
"Sid": "CreateTag"
},
{
"Action": [
"ec2:DeleteTags"
],
"Effect": "Allow",
"Resource": [
"arn:aws:ec2:*:*:elastic-ip/*",
"arn:aws:ec2:*:*:internet-gateway/*",
"arn:aws:ec2:*:*:natgateway/*",
"arn:aws:ec2:*:*:route-table/*",
"arn:aws:ec2:*:*:security-group/*",
"arn:aws:ec2:*:*:security-group-rule/*",
"arn:aws:ec2:*:*:subnet/*",
"arn:aws:ec2:*:*:vpc/*"
],
"Sid": "DeleteTag"
},
{
"Action": [
"ec2:AllocateAddress",
"ec2:CreateInternetGateway",
"ec2:CreateVpc"
],
"Condition": {
"StringLike": {
"aws:RequestTag/Name": "cce-*"
}
},
"Effect": "Allow",
"Resource": [
"*"
],
"Sid": "Create"
},
{
"Action": [
"ec2:CreateNatGateway"
],
"Condition": {
"StringNotLike": {
"ec2:ResourceTag/Name": "cce-*"
}
},
"Effect": "Deny",
"Resource": [
"arn:aws:ec2:*:*:elastic-ip/*",
"arn:aws:ec2:*:*:subnet/*"
],
"Sid": "CreateNGWAllowCCESubnetOnly"
},
{
"Action": [
"ec2:CreateNatGateway"
],
"Condition": {
"StringNotLike": {
"aws:RequestTag/Name": "cce-*"
}
},
"Effect": "Deny",
"Resource": [
"arn:aws:ec2:*:*:natgateway/*"
],
"Sid": "CreateNGWAllowCCEOnly"
},
{
"Action": [
"ec2:CreateNatGateway"
],
"Effect": "Allow",
"Resource": [
"arn:aws:ec2:*:*:elastic-ip/*",
"arn:aws:ec2:*:*:natgateway/*",
"arn:aws:ec2:*:*:subnet/*"
],
"Sid": "CreateNGW"
},
{
"Action": [
"ec2:CreateRouteTable",
"ec2:CreateSecurityGroup",
"ec2:CreateSubnet"
],
"Condition": {
"StringNotLike": {
"ec2:ResourceTag/Name": "cce-*"
}
},
"Effect": "Deny",
"Resource": [
"arn:aws:ec2:*:*:vpc/*"
],
"Sid": "CreateSubAllowCCEVPCOnly"
},
{
"Action": [
"ec2:CreateRouteTable"
],
"Condition": {
"StringNotLike": {
"aws:RequestTag/Name": "cce-*"
}
},
"Effect": "Deny",
"Resource": [
"arn:aws:ec2:*:*:route-table/*"
],
"Sid": "CreateRTAllowCCEOnly"
},
{
"Action": [
"ec2:CreateRouteTable"
],
"Effect": "Allow",
"Resource": [
"arn:aws:ec2:*:*:route-table/*",
"arn:aws:ec2:*:*:vpc/*"
],
"Sid": "CreateRT"
},
{
"Action": [
"ec2:CreateSecurityGroup"
],
"Condition": {
"StringNotLike": {
"aws:RequestTag/Name": "cce-*"
}
},
"Effect": "Deny",
"Resource": [
"arn:aws:ec2:*:*:security-group/*"
],
"Sid": "CreateSGsAllowCCEOnly"
},
{
"Action": [
"ec2:CreateSecurityGroup"
],
"Effect": "Allow",
"Resource": [
"arn:aws:ec2:*:*:security-group/*",
"arn:aws:ec2:*:*:vpc/*"
],
"Sid": "CreateSG"
},
{
"Action": [
"ec2:CreateSubnet"
],
"Condition": {
"StringNotLike": {
"aws:RequestTag/Name": "cce-*"
}
},
"Effect": "Deny",
"Resource": [
"arn:aws:ec2:*:*:subnet/*"
],
"Sid": "CreateSubAllowCCEOnly"
},
{
"Action": [
"ec2:CreateSubnet"
],
"Effect": "Allow",
"Resource": [
"arn:aws:ec2:*:*:subnet/*",
"arn:aws:ec2:*:*:vpc/*"
],
"Sid": "CreateSubnets"
},
{
"Action": [
"ec2:CreateVpcEndpoint"
],
"Effect": "Allow",
"Resource": [
"*"
],
"Sid": "CreateVpcEndpoint"
},
{
"Action": [
"ec2:AssociateAddress",
"ec2:AssociateRouteTable",
"ec2:AssociateSubnetCidrBlock",
"ec2:AssociateVpcCidrBlock",
"ec2:AssignPrivateNatGatewayAddress",
"ec2:AttachInternetGateway",
"ec2:AuthorizeSecurityGroupEgress",
"ec2:AuthorizeSecurityGroupIngress",
"ec2:CreateRoute",
"ec2:ModifySecurityGroupRules",
"ec2:ModifySubnetAttribute",
"ec2:ModifyVpcAttribute",
"ec2:ModifyVpcEndpoint",
"ec2:ReplaceRoute",
"ec2:ReplaceRouteTableAssociation",
"ec2:UpdateSecurityGroupRuleDescriptionsEgress",
"ec2:UpdateSecurityGroupRuleDescriptionsIngress"
],
"Condition": {
"StringLike": {
"ec2:ResourceTag/Name": "cce-*"
}
},
"Effect": "Allow",
"Resource": [
"*"
],
"Sid": "Modify"
},
{
"Action": [
"ec2:DisassociateAddress"
],
"Condition": {
"StringNotLike": {
"ec2:ResourceTag/Name": "cce-*"
}
},
"Effect": "Deny",
"Resource": [
"arn:aws:ec2:*:*:elastic-ip/*"
],
"Sid": "DisassociateEIPAllowCCEOnly"
},
{
"Action": [
"ec2:DisassociateAddress"
],
"Effect": "Allow",
"Resource": [
"arn:aws:ec2:*:*:*/*"
],
"Sid": "DisassociateEIP"
},
{
"Action": [
"ec2:DetachInternetGateway",
"ec2:DisassociateNatGatewayAddress",
"ec2:DisassociateRouteTable",
"ec2:DisassociateSubnetCidrBlock",
"ec2:DisassociateVpcCidrBlock",
"ec2:DeleteInternetGateway",
"ec2:DeleteNatGateway",
"ec2:DeleteNetworkInterface",
"ec2:DeleteRoute",
"ec2:DeleteRouteTable",
"ec2:DeleteSecurityGroup",
"ec2:DeleteSubnet",
"ec2:DeleteVpc",
"ec2:DeleteVpcEndpoints",
"ec2:ReleaseAddress",
"ec2:RevokeSecurityGroupEgress",
"ec2:RevokeSecurityGroupIngress",
"ec2:UnassignPrivateNatGatewayAddress"
],
"Condition": {
"StringLike": {
"ec2:ResourceTag/Name": "cce-*"
}
},
"Effect": "Allow",
"Resource": [
"*"
],
"Sid": "Delete"
},
{
"Action": [
"s3:*"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::cce-*"
]
}
],
"Version": "2012-10-17"
}
Create a custom cloud
Create a custom cloud either in the Aiven Console or with the Aiven CLI.
- Aiven Console
- Aiven CLI
Launch the BYOC setup
- Log in to the Aiven Console, and go to an organization.
- Click Admin in the top navigation, and click Bring your own cloud in the sidebar.
- In the Bring your own cloud view, select Create custom cloud.
Generate an infrastructure template
In this step, an IaC template is generated in the Terraform format. In the next step, you'll deploy this template in your AWS account to acquire Role ARN (Amazon Resource Name), which Aiven needs for accessing your AWS account.
In the Create custom cloud wizard:
-
Specify cloud details:
- Cloud provider
- Region
- Custom cloud name
- Infrastructure tags
-
Click Next.
-
Specify deployment and storage details:
-
Choose between:
- Private model, which routes traffic through a proxy for additional security utilizing a bastion host logically separated from the Aiven services.
- Public model, which allows the Aiven control plane to connect to the service nodes via the public internet.
-
CIDR for BYOC resources
The CIDR block defines the IP address range of the VPC that Aiven creates in your own cloud account. Any Aiven service created in the custom cloud will be placed in the VPC and will get an IP address within this address range.
In the CIDR field, specify an IP address range for the BYOC VPC using a CIDR block notation, for example:
10.0.0.0/16,172.31.0.0/16, or192.168.0.0/20.Make sure that an IP address range you use meets the following requirements:
-
IP address range is within the private IP address ranges allowed in RFC 1918.
-
CIDR block size is between
/16(65536 IP addresses) and/24(256 IP addresses). -
CIDR block is large enough to host the desired number of services after splitting it into per-availability-zone subnets.
For example, the smallest
/24CIDR block might be enough for a few services but can pose challenges during node replacements or maintenance upgrades if running low on available free IP addresses. -
CIDR block of your BYOC VCP doesn't overlap with the CIDR blocks of VPCs you plan to peer your BYOC VPC with. You cannot change the BYOC VPC CIDR block after your custom cloud is created.
-
-
Object storage Limited availability
By default, the following data is stored in the BYOC object storage in your own cloud account:
note- Data is stored in your BYOC object storage using one S3 bucket per custom cloud.
- Permissions for S3 bucket management will be included in the Terraform infrastructure template to be generated upon completing this step.
-
-
Click Generate template.
Your IaC Terraform template gets generated based on your inputs. You can view, copy, or download it. Now, you can use the template to acquire Role ARN.
Deploy the template
Role ARN is an identifier of the role created when running the infrastructure template in your AWS account. Aiven uses Role ARN to assume the role and run operations such as creating VMs for service nodes in your BYOC account.
Use the generated Terraform template to create your Role ARN by deploying the template in your AWS account.
-
Copy or download the template and the variables file from the Create custom cloud wizard.
-
Optionally, modify the template as needed.
noteTo connect to a custom-cloud service from different security groups (other than the one dedicated for the custom cloud) or from IP address ranges, add specific ingress rules before you apply a Terraform infrastructure template in your AWS account in the process of creating a custom cloud resources.
Before adding ingress rules, see the examples provided in the Terraform template you generated and downloaded from Aiven Console.
-
Set up Terraform to authenticate with AWS.
Configure your AWS credentials using one of the following methods:
-
Environment variables (quick setup for testing):
export AWS_ACCESS_KEY_ID="your_access_key"
export AWS_SECRET_ACCESS_KEY="your_secret_key"
export AWS_DEFAULT_REGION="your_region" -
AWS CLI profile (recommended for local development):
-
Configure credentials using the AWS CLI:
aws configure --profile your-profile-name -
Reference the profile when running Terraform:
export AWS_PROFILE=your-profile-name
-
-
IAM roles (recommended for production and CI/CD environments):
If running on an EC2 instance, in AWS CloudShell, or in a CI/CD pipeline, use IAM roles attached to the compute resource instead of static credentials.
tipFor enhanced security, consider using aws-vault to store encrypted credentials or AWS Single Sign-On (SSO) for centralized identity management.
For more authentication options and configuration details, see the AWS Provider authentication documentation.
-
-
Deploy the infrastructure template using Terraform:
terraform init
terraform plan -var-file=FILE_NAME.tfvars
terraform apply -var-file=FILE_NAME.tfvarsReplace
FILE_NAME.tfvarswith the name of the variables file you downloaded.importantThe
-var-fileoption is required to pass the configuration variables to Terraform. -
Find the role identifier (Role ARN) in the Terraform output after running
terraform apply. -
Enter Role ARN into the IAM role ARN field in the Create custom cloud wizard.
-
Click Next to proceed or park your cloud setup and save your current configuration as a draft by selecting Save draft. You can resume creating your cloud later.
Set up your custom cloud's availability
Select projects where the new custom cloud will be available for hosting your services. These projects will support creating new services in the custom cloud or migrating your existing services to the custom cloud if your service and networking configuration allows it. For more information on migrating your existing services to the custom cloud, contact your account team.
Your cloud can be available in:
- All the projects in your organization
- Selected organizational units
- Specific projects only
To set up your cloud's availability in the Create custom cloud wizard > the Assign BYOC to projects section, select one of the two following options:
- By default for all projects to make your custom cloud available in all existing and future projects in the organization
- By selection to pick specific projects or organizational units where you want your custom cloud to be available.
By selecting an organizational unit, you make your custom cloud available from all the projects in this unit.