So my bucket Alpha is in a region called REGION1.My bucket Beta is in a different region, let's call it REGION2.I go into my EC2 instance, and use this command:aws s3 cp s3://Alpha/meh.txt./meh.txtIt works! So far so good. It works perfectly.But, when I do this command:aws s3 cp s3://Beta/qwe.txt./qwe.txtI get the following error message: fatal error: An error occurred (400) when calling the HeadObject operation: Bad RequestThe error above can only be fixed if I specify the region flag. Why is that?Because I thought bucket names are unique regardless of region, so why should I need to specify a region?Is there a way to stop this from happening without specifying a region? Maybe a special policy or something?Thank you.
Conda install -c anaconda boto3. EC2 Client and Response. Now that the Boto3 Library is all set to use, let us start. EC2 Client Introduction. Boto3’s client interface allows the user to query against the existing resources and minimal functionality to modify some aspects of these resources. Using Boto3, you can operate on DynamoDB stores in pretty much any way you would ever need to. You can create new tables, read and write data either individually or in bulk, you can delete tables, change table capacities, set up auto-scaling, etc. Creating the pricing client pricing = boto3.client('pricing') Filters. The filters I used were: Operating System; Region Name; Instance Type; Tenancy; Product Family; Usage Type; OS: The os filter takes the values ‘SUSE’, ‘RHEL’, ‘Windows’, ‘Linux’ only. Region: The regions filter must use the actual region name.
Only AWS knows the answer to this. AWS could remove this requirement, but I guess they want you to be explicit.It would be simple enough to do automatically. When the client gets a request to a bucket a central lookup service could determine the bucket region and send the request there. That however creates a single point of failure. You could work around that by running the lookup service in every region, and having all the endpoints in the client, but that's more code, more configuration, more services to run, more services taking CPU.If they didn't want to run a lookup service then the requests would have to go via a central point, proxied to your region.
That takes CPU and bandwidth.It's probably easier and more efficient to specify the region than the other options. I could have been clearer. My intention was that you were correct about the inefficiency of proxying, and to mention that there are other complications of that as well. While it's possible to access the directory, that, too, is inefficient.
But I believe the cli will quietly do it anyway if you don't give it a region explicitly or in config. I never configure aws-cli with a region, and always specify it explicitly, since it keeps me mindful about where my assets are.–Aug 27 '17 at 2:36.
Configuring CredentialsThere are two types of configuration data in boto3: credentials and non-credentials. Credentials include items such as awsaccesskeyid, awssecretaccesskey, and awssessiontoken. Non-credential configuration includes items such as which region to use or which addressing style to use for Amazon S3. The distinction between credentials and non-credentials configuration is important because the lookup process is slightly different. Boto3 will look in several additional locations when searching for credentials that do not apply when searching for non-credential configuration.The mechanism in which boto3 looks for credentials is to search through a list of possible locations and stop as soon as it finds credentials. The order in which Boto3 searches for credentials is:. Passing credentials as parameters in the boto.client method.
Passing credentials as parameters when creating a Session object. Environment variables. Shared credential file ( /.aws/credentials). AWS config file ( /.aws/config). Assume Role provider. Boto2 config file ( /etc/boto.cfg and /.boto). Instance metadata service on an Amazon EC2 instance that has an IAM role configured.Each of those locations is discussed in more detail below.
Import boto3 client = boto3. Client ( 's3', awsaccesskeyid = ACCESSKEY, awssecretaccesskey = SECRETKEY, awssessiontoken = SESSIONTOKEN, ) # Or via the Session session = boto3. Session ( awsaccesskeyid = ACCESSKEY, awssecretaccesskey = SECRETKEY, awssessiontoken = SESSIONTOKEN, )where ACCESSKEY, SECRETKEY and SESSIONTOKEN are variables that contain your access key, secret key, and optional session token.
Note that the examples above do not have hard coded credentials. We do not recommend hard coding credentials in your source code. Environment VariablesBoto3 will check these environment variables for credentials: AWSACCESSKEYID The access key for your AWS account. AWSSECRETACCESSKEY The secret key for your AWS account. AWSSESSIONTOKEN The session key for your AWS account. This is only needed when you are using temporary credentials. The AWSSECURITYTOKEN environment variable can also be used, but is only supported for backwards compatibility purposes.
AWSSESSIONTOKEN is supported by multiple AWS SDKs besides python. Shared Credentials FileThe shared credentials file has a default location of /.aws/credentials. You can change the location of the shared credentials file by setting the AWSSHAREDCREDENTIALSFILE environment variable.This file is an INI formatted file with section names corresponding to profiles. With each section, the three configuration variables shown above can be specified: awsaccesskeyid, awssecretaccesskey, awssessiontoken. These are the only supported values in the shared credential file.Below is an minimal example of the shared credentials file. NoteThis is a different set of credentials configuration than using IAM roles for EC2 instances, which is discussed in a section below.Within the /.aws/config file, you can also configure a profile to indicate that boto3 should assume a role.
When you do this, boto3 will automatically make the corresponding AssumeRole calls to AWS STS on your behalf. It will handle in memory caching as well as refreshing credentials as needed.You can specify the following configuration values for configuring an IAM role in boto3:. rolearn – The ARN of the role you want to assume. sourceprofile – The boto3 profile that contains credentials we should use for the initial AssumeRolecall. externalid – A unique identifier that is used by third parties to assume a role in their customers’ accounts. This maps to the ExternalId parameter in the AssumeRole operation. This is an optional parameter.
mfaserial – The identification number of the MFA device to use when assuming a role. This is an optional parameter.
Specify this value if the trust policy of the role being assumed includes a condition that requires MFA authentication. The value is either the serial number for a hardware device (such as GAHT12345678) or an Amazon Resource Name (ARN) for a virtual device (such as arn:aws:iam::12:mfa/user). rolesessionname – The name applied to this assume-role session. This value affects the assumed role user ARN (such as arn:aws:sts::12:assumed-role/rolename/rolesessionname).
This maps to the RoleSessionName parameter in the AssumeRole operation. This is an optional parameter. If you do not provide this value, a session name will be automatically generated.If you do not have MFA authentication required, then you only need to specify a rolearn and a sourceprofile.When you specify a profile that has IAM role configuration, boto3 will make an AssumeRole call to retrieve temporary credentials. Subsequent boto3 API calls will use the cached temporary credentials until they expire, in which case boto3 will automatically refresh credentials. Boto3 does not write these temporary credentials to disk.
This means that temporary credentials from the AssumeRole calls are only cached in memory within a single Session. All clients created from that session will share the same temporary credentials.If you specify an mfaserial, then the first time an AssumeRole call is made, you will be prompted to enter the MFA code. Your code will block until you enter your MFA code. You’ll need to keep this in mind if you have an mfaserial configured but would like to use boto3 in some automated script.Below is an example configuration for the minimal amount of configuration needed to configure an assume role profile. IAM RoleIf you are running on Amazon EC2 and no credentials have been found by any of the providers above, boto3 will try to load credentials from the instance metadata service. In order to take advantage of this feature, you must have specified an IAM role to use when you launched your EC2 instance. For more information on how to configure IAM roles on EC2 instances, see the guide.Note that if you’ve launched an EC2 instance with an IAM role configured, there’s no explicit configuration you need to set in boto3 to use these credentials. Boto3 will automatically use IAM role credentials if it does not find credentials in any of the other places listed above.
Aws Boto3 Client Specify Region
Environment Variable Configuration AWSACCESSKEYID The access key for your AWS account. AWSSECRETACCESSKEY The secret key for your AWS account. AWSSESSIONTOKEN The session key for your AWS account. This is only needed when you are using temporary credentials. The AWSSECURITYTOKEN environment variable can also be used, but is only supported for backwards compatibility purposes.
AWSSESSIONTOKEN is supported by multiple AWS SDKs besides python. AWSDEFAULTREGION The default region to use, e.g. Us-west-1, us-west-2, etc. AWSPROFILE The default profile to use, if any. If no value is specified, boto3 will attempt to seach the shared credentials file and the config file for the default profile.
AWSCONFIGFILE The location of the config file used by boto3. By default this value is /.aws/config.
You only need to set this variable if you want to change this location. AWSSHAREDCREDENTIALSFILE The location of the shared credentials file. By default this value is /.aws/credentials.
You only need to set this variable if you want to change this location. BOTOCONFIG The location of the boto2 credentials file. This is not set by default. You only need to set this variable if want to use credentials stored in boto2 format in a location other than /etc/boto.cfg or /.boto.
AWSCABUNDLE The path to a custom certificate bundle to use when establishing SSL/TLS connections. Boto3 includes a bundled CA bundle it will use by default, but you can set this environment variable to use a different CA bundle. AWSMETADATASERVICETIMEOUT The number of seconds before a connection to the instance metadata service should time out. When attempting to retrieve credentials on an EC2 instance that has been configured with an IAM role, a connection to the instance metadata service will time out after 1 second by default. If you know you are running on an EC2 instance with an IAM role configured, you can increase this value if needed. AWSMETADATASERVICENUMATTEMPTS When attempting to retrieve credentials on an EC2 instance that has been configured with an IAM role, boto3 will only make one attempt to retrieve credentials from the instance metadata service before giving up.
If you know your code will be running on an EC2 instance, you can increase this value to make boto3 retry multiple times before giving up. AWSDATAPATH A list of additional directories to check when loading botocore data. You typically do not need to set this value. There’s two built in search paths: /data/ and /.aws/models. Setting this environment variable indicates additional directories to first check before falling back to the built in search paths. Multiple entries should be separated with the os.pathsep character which is: on linux and; on windows.
Configuration FileBoto3 will also search the /.aws/config file when looking for configuration values. You can change the location of this file by setting the AWSCONFIGFILE environment variable.This file is an INI formatted file that contains at least one section: default. You can create multiple profiles (logical groups of configuration) by creating sections named profile profile-name. If your profile name has spaces, you’ll need to surround this value in quotes: profile 'my profile name'. Below are all the config variables supported in the /.aws/config file: region The default region to use, e.g. Us-west-1, us-west-2, etc.
When specifying a region inline during client initialization, this property is named regionname awsaccesskeyid The access key to use. Awssecretaccesskey The secret access key to use. Awssessiontoken The session token to use. This is typically only needed when using temporary credentials. Note awssecuritytoken is supported for backwards compatibility. Cabundle The CA bundle to use.
See the docs above on AWSCABUNDLE for more information. Metadataservicetimeout The number of seconds before timing out when retrieving data from the instance metadata service. See the docs above on AWSMETADATASERVICETIMEOUT for more information. Metadataservicenumattempts The number of attempts to make before giving up when retrieving data from the instance metadata service.
See the docs above on AWSMETADATASERVICENUMATTEMPTS for more information. Parametervalidation Disable parameter validation (default is true; parameters are validated by default). This is a boolean value that can have a value of either true or false. Whenever you make an API call using a client, the parameters you provide are run through a set of validation checks including (but not limited to): required parameters provided, type checking, no unknown parameters, minimum length checks, etc. You generally should leave parameter validation enabled. Rolearn The ARN of the role you want to assume. Sourceprofile The profile name that contains credentials we should use for the initial AssumeRole call.
Externalid Unique identifier to pass when making AssumeRole calls. Mfaserial Serial number of ARN of an MFA device to use when assuming a role. Rolesessionname The role name to use when assuming a role. If this value is not provided, a session name will be automatically generated. S3Set S3 specific configuration data. You typically will not need to set these values.
Boto3 will automatically switching signature versions and addressing styles if necessary. This is a nested configuration value. See the Nested Configuration section for more information on the format. The sub config keys supported for s3 are:. addressingstyle: Specifies which addressing style to use.
This controls if the bucket name is in the hostname or part of the URL. Value values are: path, virtual, and auto. signatureversion: Which AWS signature version to use when signing requests. Value values are: s3 and s3v4.