cfn deploy & package break with aws-cli version 1.41.9

[JosephBrooksbank]

Started here: localstack/localstack#12933 (comment)
Then moved here: aws/aws-cli#9629
Which helped me figure out the issue. awslocal patches create_client to configure the s3 endpoint specifically when running certain commands.

It looks like v1.41.9 of aws-cli wraps create_client in create_nested_client. I'm not an avid python user so I'm not 100% sure what is going on here but it seems like this overrides the wrapping that awslocal does.

I think the solution should be pretty simple. There is an AWS_ENDPOINT_URL that can be set per service or for all services. Setting this environment variable in prepare_environment should replace the necessity of a lot of the rest of the wrapper I think

[alexrashed]

Hi @JosephBrooksbank!
Is there a chance you could provide a simple reproducer? For me, awslocal with S3 works just fine with the latest versions of the package:

localstack start -d
localstack wait
cd /tmp
python3 -m venv awslocal-test-venv
source awslocal-test-venv/bin/activate.fish
python3 -m pip install --upgrade awscli[v1] awscli-local
which awslocal
awslocal s3 mb s3://test-bucket
localstack stop

All works just fine for me with awscli==1.42.0 and awscli-local==0.22.0.

[JosephBrooksbank]

Hi @alexrashed!
Sure, I'll get something put together. awslocal with s3 directly works fine for me as well- the issue is the 'side effect' caused by using cloudformation package, which is a cloudformation service call that itself makes calls to s3. When you pass in --endpoint-url to aws it only applies to the direct call to the service. Someone submitted a PR about this a while ago, but instead localstack went with a patch on your side. Using the AWS_ENDPOINT_URL applies 'globally' to all service calls.

[JosephBrooksbank]

Here it is: https://github.com/JosephBrooksbank/localstack-issue-91
I'm on windows (🤢)

.\localstack.exe start -d
python3 -m venv awslocal-test-venv
.\awslocal-test-venv\activate
pip install awscli==1.41.16 awscli-local aws-sam-cli
awslocal s3 mb s3://lambdas
sam build -t .\template.yaml
awslocal cloudformation package --template .\.aws-sam\build\template.yaml --region us-east-1 --output-template packaged_template.yaml --s3-bucket lambdas

Gives

Unable to upload artifact TestLambda referenced by CodeUri parameter of TestLambda resource.
An error occurred (InvalidAccessKeyId) when calling the PutObject operation: The AWS Access Key Id you provided does not exist in our records.

[alexrashed]

Thanks a lot for the detailed reproducer and the thorough research on the issue!
You were right, the patch broke with the wrapping of the client in botcore and respectively in awscli, starting with version 1.41.9.
While I generally agree that we should switch towards the endpoint configuration using the standard tools, I wanted to move fast on this one and just fix the patch for now.
Once #93 is merged, I'll create and publish a new release, which should fix your issues.

[alexrashed]

FYI: Your issue should be fixed with 0.22.1. Thanks again for the detailed report, please let us know if you face any other issues.