On: How to port your OS to EC2 (2018)

[zdw]

> Due to some bizarre breakage in EC2 — which I've been complaining about for ten years — the serial console is very "laggy". If you find that you're not getting any output, wait five minutes and try again.

This is the single most frustrating bug with EC2 I've experienced. In my case, I have Jenkins spinning up temporary executors in EC2, then getting the SSH host key from serial log to verify the connection (I appreciate the "importance of SSH host keys compared with flossing" comparison in this article...)

Machines take <1m to boot, then sit there for 4-8m just waiting for the log message to show up.

[oppositelock → zdw]

There's a new console feature which addresses this, but yes, the previous incarnation was very frustrating. I'd like to see this on all instance types.

https://aws.amazon.com/about-aws/whats-new/2021/03/introduci...

[paulddraper → zdw]

You should add the fingerprint as an EC2 tag or something from a boot script.

[kelnos → paulddraper]

Doesn't that require giving the instance elevated privileges around setting tags? Some people might not want that.

[paulddraper → kelnos]

It does, though you can limit it to certain tag name like "SSH Fingerprint".

And if course you need the AWS CLI installed (which AWS built AMIs have).

All in all seems fine for a Jenkins worker

[cperciva → paulddraper]

Two problems with doing this:

1. If you provide an EC2 Role which allows setting that tag, any process on the instance which can access the Instance Metadata Store can record a different SSH fingerprint.

2. You can only have one EC2 Role attached at once, so doing this prevents you from using other roles.

[jen20 → cperciva]

It’s worse than that with respect to (1): any process on _any_ instance which has tag setting permissions can set the tag for _any other_ instance, since conditions don’t support scoping to instance ID.

Re (2) technically you cannot have any roles attached directly, but instead attach an instance profile (the distinction is clearer via the API than the console). The shape of the API for an instance profile clearly was designed to support multiple roles, but in practice is limited to one. It’s typical to create a role with many policies attached for each functional type of instance, so in practice it does’t matter too much.

[klohto → jen20]

1) CreateTags does support scoping using StringEquals with ec2:SourceInstanceARN and comparing that with the aws:arn

  "aws:ARN": "${ec2:SourceInstanceARN}"

It will give you a warning when creating through web console, but the condition works.

[jen20 → klohto]

TIL - thanks for the heads up. Is this a documented substitution?

[klohto → jen20]

I don't think so... I have been trying to find it for a few years now, but it just won't turn up anywhere.

FYI, I personally use resource tags as well on top this, just to limit the scope (in case it magically disappears one day).

[paulddraper → cperciva]

> any process on the instance which can access the Instance Metadata Store can record a different SSH fingerprint.

A security issue really only if the process is able to pair it with a MITM attack.

> You can only have EC2 Role attached at once.

Yes, so attach this policy to that IAM role.

[klohto → cperciva]

1. You can limit access to IMS to certain processes inside linux 2. Just combine the policies

If you’re worried about malicious processes on the EC2 altering the fingerprint, you got bigger issues. You can also sign the fingerprint with gpg.

[znpy → zdw]

You might want to look at SSH certificate authority and stuff like that.

No more checking SSH host keys, validation is automatic and stuff only fail if something is wrong.