IAM roles for service account on AWS EKS¶
Applications in a pod's containers can use an AWS SDK or the AWS CLI to make API requests to AWS services using AWS Identity and Access Management (IAM) permissions. Applications must sign their AWS API requests with AWS credentials. IAM roles for service accounts provide the ability to manage credentials for your applications. To achieve this, you can read the following articles:
- IAM roles for service accounts: This is the official AWS documentation that explains IAM roles for service accounts step-by-step.
- Understanding IAM roles for service accounts, IRSA, on AWS EKS.: Good article to provide an easy-to-understand explanation.
- It's worth noting that this pitfall occurs in Ray images prior to version 2.5.0.
For example, users may want to download their files from their S3 bucket with AWS Python SDK (
boto3) in Ray Pods. However, there is a pitfall in the Ray images. When you execute the boto3_example_1.py in a Ray Pod, you will get an error like
An error occurred (403) when calling the HeadObject operation: Forbidden even if your pod is attached to a service account which has an IAM role that is able to access the S3 bucket.
# boto3_example_1.py import boto3 s3 = boto3.client('s3') bucket_name = YOUR_BUCKET_NAME key = YOUR_OBJECT_KEY filename = YOUR_FILENAME s3.download_file(bucket_name, key, filename)
The root cause is that the version of
boto3 in the Ray image is too old. To elaborate,
rayproject/ray:2.3.0 provides boto3 version 1.4.8 (Nov. 21, 2017),
a more recent version (1.26) is currently available as per https://pypi.org/project/boto3/#history. The
boto3 1.4.8 does not support to initialize the security credentials automatically in some cases (e.g.
# image: rayproject/ray:2.5.0 pip freeze | grep boto # boto3==1.4.8 # botocore==1.8.50
Another issue that users may encounter is related to RayService.
working_dir for RayService is set to a zip file located in a private S3 bucket, it can prevent the Ray Serve application from starting. Users can confirm this by executing
serve status in the head Pod, which will return an error like
An error occurred (AccessDenied) when calling the GetObject operation: Access Denied. In this case, users can build their custom images with upgrading the
boto3 package (i.e. Solution 2).
Solution 1: Setup the credentials explicitly¶
# boto3_example_2.py import os import boto3 def assumed_role_session(): role_arn = os.getenv('AWS_ROLE_ARN') with open(os.getenv("AWS_WEB_IDENTITY_TOKEN_FILE"), 'r') as content_file: web_identity_token = content_file.read() role = boto3.client('sts').assume_role_with_web_identity(RoleArn=role_arn, RoleSessionName='assume-role', WebIdentityToken=web_identity_token) credentials = role['Credentials'] aws_access_key_id = credentials['AccessKeyId'] aws_secret_access_key = credentials['SecretAccessKey'] aws_session_token = credentials['SessionToken'] return boto3.session.Session(aws_access_key_id=aws_access_key_id, aws_secret_access_key=aws_secret_access_key, aws_session_token=aws_session_token) session = assumed_role_session() s3 = session.client("s3") bucket_name = YOUR_BUCKET_NAME key = YOUR_OBJECT_KEY filename = YOUR_FILENAME s3.download_file(bucket_name, key, filename)
Solution 2: Upgrade the boto3 package¶
pip install --upgrade boto3 python3 -m pip install -U pyOpenSSL cryptography python3 boto3_example_1.py # success