ec2tag watcher fails if metadata service is unavailable during a credential refresh

[ianbamforth]

The DefaultProvider takes aws keys from environment variables / config if available, otherwise falling back to ec2 metadata IAM role. If you are using keys from the latter, they refresh automatically when they get near to expiry. If the metadata service is down at this point, Synapse enters a broken state, where it has no credentials and cannot recover. We have seen this in our production environment.

I'm currently looking at a patch whereby I specifically select the EC2Provider if no keys are provided by the environment / config, inserting the following into ec2tag.rb before the call to AWS::EC2.new:

unless ((@discovery['aws_access_key_id'] || ENV['aws_access_key_id']) \ && (@discovery['aws_secret_access_key'] || ENV['aws_secret_access_key'] )) AWS.config(:credential_provider => AWS::Core::CredentialProviders::EC2Provider.new(:retries => 0)) end

Does this seem like a reasonable approach? Happy to submit a PR.

[jolynch]

@ianbamforth that seems reasonable to me. If it's been working in production for you I'm happy to merge it :-)