Lambda Support

Lambda provides for powerful realtime event based code execution in response to infrastructure and application behavior. A number of different Amazon services can be used as event sources.

CloudWatch Events

CloudWatch Events (CWE) is a general event bus for AWS infrastructure. Currently, it covers several major sources of information:

  1. CloudTrail API calls over a poll period on CloudTrail delivery,
  2. real-time instance status events,
  3. autoscale group notifications, and
  4. scheduled/periodic events.

CloudTrail provides a very rich data source over the entire range of AWS services exposed via the audit trail that allows Custodian to define effective realtime policies against any AWS product. Additionally, for EC2 instances we can provide mandatory policy compliance - this means the non-compliant resources never became available.

Cloud Custodian Integration

Custodian provides for policy level execution against any CWE event stream. Each Custodian policy can be deployed as an independent Lambda function. The only difference between a Custodian policy that runs in Lambda and one that runs directly from the CLI in poll mode is the specification of the subscription of the events in the mode config block of the policy.

Internally Custodian will reconstitute current state for all the resources in the event, execute the policy against them, match against the policy filters, and apply the policy actions to matching resources.

CloudTrail API Calls

Lambdas can receive CWE over CloudTrail API calls with delay of 90s at P99.

policies:
  - name: ec2-tag-running
    resource: ec2
    mode:
      type: cloudtrail
      events:
       - RunInstances
    actions:
      - type: mark
        tag: foo
        msg: bar

Because the total AWS API surface area is so large most CloudTrail API event subscriptions need two additional fields:

  1. For CloudTrail events we need to reference the source API call.
  2. To work transparently with existing resource policies, we also need to specify how to extract the resource IDs from the event via JMESPath so that the resources can be queried.

For very common API calls for policies, some shortcuts have been defined to allow for easier policy writing as for the RunInstances API call above, which expands to:

events:
 - source: ec2.amazonaws.com
   event: RunInstances
   ids: "responseElements.instancesSet.items[].instanceId"

EC2 Instance State Events

Lambdas can receive EC2 instance state events in real time (seconds delay).

policies:
  - name: ec2-require-encrypted-volumes
    resource: ec2
    mode:
      type: ec2-instance-state
      events:
      - pending
    filters:
      - type: ebs
        key: Encrypted
        value: False
    actions:
      - mark
      - terminate

Periodic Function

We support both rate per unit time and cron expressions, per scheduler syntax.

policies:
  - name: s3-bucket-check
    resource: s3
    mode:
      type: periodic
      schedule: "rate(1 day)"

Config Rules

AWS Config rules allow you to invoke logic in response to configuration changes in your AWS environment, and Cloud Custodian is the easiest way to write and provision Config rules. Delay here is typically 1-15m (though the SLA on tag-only changes is a bit higher).

In this section we’ll look at how we would deploy the quickstart example using Config. Before you proceed, make sure you’ve removed the Custodian tag from any EC2 instance left over from the quickstart.

First, modify custodian.yml to specify a mode type of config-rule. You’ll also need the ARN of an IAM role to assume when running the Lambda that Custodian is going to install for you.

policies:
  - name: my-first-policy
    mode:
        type: config-rule
        role: arn:aws:iam::123456789012:role/some-role
    resource: ec2
    filters:
      - "tag:Custodian": present
    actions:
      - stop

Now deploy the policy:

custodian run -s . custodian.yml

That should give you log output like this:

2017-01-25 05:43:01,539: custodian.policy:INFO Provisioning policy lambda my-first-policy
2017-01-25 05:43:04,683: custodian.lambda:INFO Publishing custodian policy lambda function custodian-my-first-policy

Go check the AWS console to see the Lambda as well as the Config rule that Custodian created. The Config rule should be listed as “Compliant” or “No results reported” (if not, be sure you removed the Custodian tag from any instance left over from the quickstart).

Now for the fun part! With your new policy installed, go ahead and create an EC2 instance with a Custodian tag (any non-empty value), and wait (events from Config are effectively delayed 15m up to 6hrs on tag changes). If all goes well, you should eventually see that your new custom Config rule notices the EC2 instance with the Custodian tag, and stops it according to your policy.

Congratulations! You have now installed your policy to run under Config rather than from your command line.

Execution Options

When running in Lambda you may want policy execution to run using particular options corresponding to those passed to the custodian CLI.

Execution in lambda comes with a default set of configuration which is different from the defaults you might set when running through the command line:

  • Metrics are enabled
  • Output dir is set to a random /tmp/ directory
  • Caching of AWS resource state is disabled
  • Account ID is automatically set with info from sts
  • Region is automatically set to the region of the lambda (using the AWS_DEFAULT_REGION environment variable in lambda)

When you want to override these settings, you must set ‘execution-options’ with one of the following keys:

  • region
  • cache
  • profile
  • account_id
  • assume_role
  • log_group
  • metrics_enabled
  • output_dir
  • cache_period
  • dryrun

One useful thing we can do with these options is to make a policy execute in a different account using assume_role. A policy definition for this looks like:

policies:
  - name: my-first-policy-cross-account
    mode:
        type: periodic
        schedule: "rate(1 day)"
        role: arn:aws:iam::123456789012:role/lambda-role
        execution-options:
          assume_role: arn:aws:iam::210987654321:role/target-role
          metrics_enabled: false
    resource: ec2
    filters:
      - "tag:Custodian": present
    actions:
      - stop

A couple of things to note here:

  1. Metrics are pushed using the assumed role which may or may not be desired
  2. The mode must be periodic as there are restrictions on where policy executions can run according to the mode:
    Config:May run in a different region but not cross-account
    Event:Only run in the same region and account
    Periodic:May run in a different region and different account