Sign upSign InSign upSign InArseny Zinchenko (setevoy)FollowITNEXT--ListenShareSo we have an AWS EKS cluster built with AWS CDK and Python — AWS: CDK and Python — building an EKS cluster, and general impressions of CDK, and we have an idea of how IRSA works — AWS: EKS, OpenID Connect, and ServiceAccounts.The next step after deploying the cluster is to configure the OIDC Identity Provider in AWS IAM and to add two controllers — ExternalDNS to work with Route53, and AWS ALB Controller to create AWS Load Balancers when creating Ingress in an EKS cluster.For authentication in AWS, both controllers will use the IRSA model — IAM Roles for ServiceAccounts, that is, in the Kubernetes Pod with the controller we’ll connect a ServiceAccount that will allow the use of an IAM role with an IAM Policies with the necessary permissions.The WorkerNodes autoscaling controller will be added later: previously, I’ve used the Cluster AutoScaler, but this time I want to try Karpenter, so I’ll make a separate post for that.We continue to eat cactus use the AWS CDK with Python. It will be used to create IAM resources and deploy Helm charts with controllers directly from the CloudFormation stack of the cluster.I tried to deploy the controllers as a separate stack but spent an hour or trying to figure out how to get the AWS CDK to pass values ​​from one stack to another via CloudFormation Exports and Outputs, but finally, I gave up and did it all in one stack class. May be will try another time.Creating a cluster is described in one of the previous posts — AWS: CDK and Python — building an EKS cluster, and general impressions of CDK.What do we have now?A class to create a stack:The aws_account is passed from the app.py when creating an AtlasEksStack() class object:And we will continue to use it for the AWS IAM configuration.We also have a separate VPC:And the EKS cluster itself:Next, we need to add the creation of OIDC in IAM, and the deployment of Helm charts with controllers.We’ll use boto3 (this is one of the things that I don't really like about AWS CDK - that a lot of things have to be done not with the methods/constructs of the CDK itself, but with "crutches" in the form of boto3 or other modules/libraries).We need to get the OIDC Issuer URL, and get its thumbprint — then we can use the create_open_id_connect_provider.OIDC Provider URL can be obtained using boto3.client('eks'):Next, with the help of the libraries ssl and comhashlib we get the thumbprint of the oidc.eks.us-east-1.amazonaws.com endpoint's certificate :And now with boto3.client('iam') and create_open_id_connect_provider() we can create the IAM OIDC Identity Provider:Here, we wrap everything in a the try/except, because during further updates of the stack boto3.client('iam') sees that the Provider already exists, and it crashes with an error EntityAlreadyExists.Let’s add the ExternalDNS first — it has a fairly simple IAM Policy, so we’ll test how CDK works with Helm charts.Here, the first step is to create an IAM Role that our ServiceAccount can assume for ExternalDNS, and which will allow ExternalDNS to perform actions with the domain zone in Route53 because now ExternalDNS has a ServiceAccount, but it gives an error:msg=”records retrieval failed: failed to list hosted zones: WebIdentityErr: failed to retrieve credentials\ncaused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity\n\tstatus code: 403In the Trust relationships of this role, we must specify a Principal in the form of the ARN created by the OIDC Provider, in the Action — sts:AssumeRoleWithWebIdentity, and in the Condition - if the request comes from a ServiceAccount that will be created by the ExternalDNS Helm-chart.Let’s create a couple of variables:The oidc_provider_arn is formed from the variable oidc_provider_url obtained earlier in response = eks_client.describe_cluster(name=cluster_name).Describe the creation of a role using iam.Role():As a result, we should get a role with the following Trust relationships:The next step is an IAM Policy.If you deploy the stack now, ExternalDSN will get a permissions error:msg=”records retrieval failed: failed to list hosted zones: AccessDenied: User: arn:aws:sts::492***148:assumed-role/eks-dev-1–26-EksExternalDnsRoleB9A571AF-7WM5HPF5CUYM/1689063807720305270 is not authorized to perform: route53:ListHostedZones because no identity-based policy allows the route53:ListHostedZones action\n\tstatus code: 403So we need to describe two iam.PolicyStatement() - one for working with the domain zone, and the second for accessing route53:ListHostedZones API call.Make them separate, because for route53:ChangeResourceRecordSets in the resources we have to have restrictions on only one specific zone but for permission on route53:ListHostedZones the resources block should be in the form of "*". i.e. "all":Now we can add the ExternalDNS Helm chart itself.Here we use the aws-cdk.aws-eks.add_helm_chart().In the values enable the serviceAccount, and in its annotations pass the 'eks.amazonaws.com/role-arn': external_dns_role.role_arn:Let’s deploy and look under ExternalDNS — we can see both our domain-filter and environment variables for the IRSA work:Check the logs:And let’s test if it’s working.To check — let’s create a simple Service with the Loadbalancer type, in its annotations add the external-dns.alpha.kubernetes.io/hostname to trigger the ExternalDNS to create a DNS record in the Route53:Check ExternalDNS logs:And check the domain itself:“It works!” ©All the code together now looks like this:Let’s go to the ALB Controller.In general, everything is the same here, the only thing I had to mess with was the IAM Policy, because if we have only two permissions for ExternalDNS, and we can describe them directly when creating this Policy, for the ALB Controller the policy must be taken from GitHub, because it is quite large.Here we use requests (crutches again):Here, we receive the policy file, form it in JSON, and then from this JSON we form the policy document itself.Next, we create an IAM Role with the similar to ExternalDNS Trust relationships, only change its conditions - specify the ServiceAccount that will be created for the AWS ALB Controller:Now, install the Helm chart itself with the necessary values - enable a ServiceAccount, in its annotations specify the ARM role that was created above, and set the clusterName:Let’s create a simple Pod, a Service, and an Ingress which must trigger the ALB Controller to create an AWS ALB LoadBalancer:Deploy and check the Ingress resource:The only thing here that didn’t work the first time is the aws-iam-token attach to the Pod: that's why in the values I've set the 'automountServiceAccountToken': True, although it already has a default value true.But after several redeploys with cdk deploy, the token was created and connected to the Pod:In general, that’s all.As usual with CDK it’s a pain and suffering due to the lack of proper documentation and examples, but with the help of ChatGPT and the tutorials it did work.Also, it would probably be good to move the creation of resources at least to separate functions instead of doing everything with the AtlasEksStack.__init__(), but that can be done later.The next step is to launch VictoriaMetrics in Kubernetes, and then we will start working on Karpenter.Originally published at RTFM: Linux, DevOps, and system administration.----ITNEXTCloud Infrastructure and DevOps engineerArseny Zinchenko (setevoy)inITNEXT--Juntao QiuinITNEXT--10Daniel GlejznerinITNEXT--11Arseny Zinchenko (setevoy)inITNEXT--Rico FritzscheinLevel Up Coding--22Ignacio de Gregorio--31Ahmed Elfakharany--4Manusha ChethiyawardhanainAWS in Plain English--3Mojabi Rafi (Albraftitan)--ManojSainiinAWS Tip--HelpStatusWritersBlogCareersPrivacyTermsAboutText to speechTeams