AWS Threat Hunting within Cloud Functions and Microservices¶

Use these command-line hunts to investigate AWS Lambda tampering, IAM reconnaissance, credential misuse, S3-triggered input fuzzing, and related VPC Flow Log activity.

Working assumptions¶

The examples assume the following evidence layout:

cloudtrail.ndjson — one CloudTrail JSON event per line
raw/lambda/ — Lambda logs in .log or compressed .gz form
raw/vpcflow/ — compressed VPC Flow Logs
GNU-compatible awk/gawk, grep, sed, sort, column, and jq

Adjust filenames, target IPs, and field positions to match the collection. Do not expose secret values while investigating credential access.

Module 1 — Function tampering and backdoors¶

Review Lambda operational events¶

Review incoming, ping, and beacon events, including JSON op fields with the value beacon. Sort chronologically and format each line as timestamp, event type, and remaining message.

Review Lambda operational events

zcat raw/lambda/000000.gz \
| grep -E 'Incoming event:|Ping event:|Beacon event:|"op":[[:space:]]*"beacon"' \
| sort -k1,1 \
| awk '
{
  ts=$1
  event="-"
  if (match($0, /"op":[[:space:]]*"([^"]+)"/, a)) { event=a[1] }
  sub(/^[^]]+\][[:space:]]+[^[:space:]]+[[:space:]]+/, "", $0)
  printf "%-24s %-8s %s\n", ts, event, $0
}'

Count beacon destinations¶

Extract and count target URLs so repeated command-and-control or beacon destinations stand out.

Count Lambda beacon destinations

zgrep -h '"op":[[:space:]]*"beacon"' raw/lambda/*.gz \
| sed -n 's/.*"target":[[:space:]]*"\([^"]\+\)".*/\1/p' \
| sort \
| uniq -c

Enumerate Lambda CloudTrail events¶

Show event time, event name, and calling identity type.

Enumerate Lambda CloudTrail events

jq -r '
select(.eventSource=="lambda.amazonaws.com")
| [.eventTime,.eventName,.userIdentity.type]
| @tsv
' cloudtrail.ndjson \
| head -20 \
| sort

Hunt Lambda configuration updates¶

Review environment-variable, description, runtime, role, and other configuration changes.

Find Lambda configuration updates

jq -r '
select(
  .eventSource=="lambda.amazonaws.com"
  and (.eventName|test("^UpdateFunctionConfiguration"))
)
| "\(.eventTime) \(.eventName) fn=\(.requestParameters.functionName // "-")"
' cloudtrail.ndjson

Inspect configuration-change context¶

Display identity and request parameters for attribution and change context.

Inspect Lambda configuration changes

jq '
select(
  .eventSource=="lambda.amazonaws.com"
  and (.eventName|test("^UpdateFunctionConfiguration"))
)
| {
    eventTime,
    eventName,
    userIdentity,
    requestParameters
  }
' cloudtrail.ndjson \
| head -80

Hunt Lambda code updates¶

Detect function-code replacement or modification events.

Find Lambda code updates

jq -r '
select(
  .eventSource=="lambda.amazonaws.com"
  and (.eventName|test("^UpdateFunctionCode"))
)
| "\(.eventTime) \(.eventName) fn=\(.requestParameters.functionName // "-")"
' cloudtrail.ndjson

Pivot on a beacon IP in VPC Flow Logs¶

Show direction, sockets, protocol, packet and byte counts, and the ACCEPT or REJECT action. The final sed expression highlights the target socket in a compatible terminal.

Pivot on a suspected beacon IP

zcat raw/vpcflow/*.gz \
| awk -v ip="54.166.174.253" '
($5==ip || $6==ip) && ($14=="ACCEPT" || $14=="REJECT") {
  dir = ($5==ip ? "OUT" : "IN")

  src = sprintf("%-21s", $5 ":" $7)
  dst = sprintf("%-21s", $6 ":" $8)

  printf "%s %s %s -> %s proto=%-2s pkts=%-5s bytes=%-7s action=%s\n",
    dir, $1, src, dst, $9, $10, $11, $14
}
' \
| sed $'s/54\\.166\\.174\\.253:80/\\x1b[38;2;231;3;221m&\\x1b[0m/g'

Module 2 — IAM reconnaissance and discovery¶

Normalize and filter Lambda reconnaissance logs¶

Remove duplicated Lambda metadata, blank lines, and unrelated records. Retain operational events, authorization failures, exceptions, tracebacks, and reconnaissance activity.

Filter Lambda reconnaissance activity

zcat raw/lambda/000000.gz \
| sed -E 's/^([0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9:.]+Z[[:space:]]+\[[A-Z]+\])[[:space:]]+[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9:.]+Z[[:space:]]+[0-9a-f-]{36}[[:space:]]+/\1 /' \
| sed -E '/^[[:space:]]*$/d' \
| grep -E 'Incoming event:|Ping event:|Op event:|Recon probe:|AccessDenied|Unauthorized|Denied|Exception|Traceback|recon' \
| head -n 500

Count Lambda operation values¶

Identify dominant operations and unusual reconnaissance-related operation types.

Count Lambda op values

zcat raw/lambda/000000.gz \
| grep -oE '"op":[[:space:]]*"[^"]+"' \
| sort \
| uniq -c \
| sort -nr \
| head -n 15

Count CloudTrail event sources¶

Reveal which AWS control-plane services dominate the activity.

Count CloudTrail services

CT="cloudtrail.ndjson"

jq -r '.eventSource' "$CT" \
| sort \
| uniq -c \
| sort -nr \
| head -n 12

Inspect STS identity and role activity¶

Review GetCallerIdentity and AssumeRole, extracting assumed-role and session names when present and retaining the source IP for correlation.

Inspect STS calls

jq -r '
select(
  .eventSource=="sts.amazonaws.com"
  and (
    .eventName=="GetCallerIdentity"
    or .eventName=="AssumeRole"
  )
)
| [
    .eventTime,
    .eventName,
    (
      if (.userIdentity.arn // "") | test("^arn:aws:sts::")
      then (
        .userIdentity.arn
        | capture("assumed-role/(?<role>[^/]+)/(?<sess>[^/]+)")
        | "\(.role)/\(.sess)"
      )
      else (.userIdentity.arn // "-")
      end
    ),
    (.sourceIPAddress // "-")
  ]
| @tsv
' "$CT" \
| sort \
| head -n 25 \
| column -t -s $'\t'

Count IAM API calls¶

Look for enumeration patterns involving ListRoles, ListPolicies, ListAttachedRolePolicies, ListAccessKeys, and GetRole.

Count IAM API calls

jq -r '
select(.eventSource=="iam.amazonaws.com")
| .eventName
' "$CT" \
| sort \
| uniq -c \
| sort -nr \
| head -n 15

Extract CloudTrail failures¶

Shorten long STS role and session names for readable output. This shell wrapper was reconstructed from a partially cropped source; validate the shortening rules against the local naming convention.

Extract CloudTrail failures

jq -r '
def shorten_role:
  split("-") as $p
  | ($p | index("lambda")) as $i
  | if $i != null and ($i + 1) < ($p | length)
    then "\($p[0])..\($p[$i])..\($p[$i+1])"
    else "\($p[0])..\($p[-1])"
    end;

def shorten_sess:
  split("-")
  | "\(.[0])...\(.[-1])";

select(.errorCode != null)
| [
    .eventTime,
    .eventSource,
    .eventName,
    .errorCode,
    (
      if (.userIdentity.arn // "") | test("^arn:aws:sts::")
      then (
        .userIdentity.arn
        | capture("assumed-role/(?<role>[^/]+)/(?<sess>[^/]+)")
        | "\(
            .role | shorten_role
          )/\(
            .sess | shorten_sess
          )"
      )
      else (.userIdentity.arn // "-")
      end
    )
  ]
| @tsv
' "$CT" \
| column -t -s $'\t'

Show accepted HTTPS flow records¶

Filter for accepted traffic where either the source or destination port is 443.

Show accepted HTTPS flows

zcat raw/vpcflow/000000.gz \
| awk '
$14=="ACCEPT" && ($7==443 || $8==443) {
  print $1, $5, "->", $6, "sport=" $7, "dport=" $8, $14
}
' \
| LC_ALL=C sort -k1,1 \
| head -n 25

Module 3 — Credential theft and misuse¶

Extract secret-read outcomes¶

Show the transition from probes for nonexistent secrets to repeated successful reads without printing secret values.

Extract secret-read outcomes

grep -E 'Secret read (failed|succeeded)' raw/lambda/000000.log \
| sed -E '
s/^([0-9TZ:\.\-]+).*Secret read failed for ([^:]+).*/\1\tFAILED\t\2/;
s/^([0-9TZ:\.\-]+).*Secret read succeeded:.*"secret_id":"([^"]+)".*/\1\tSUCCESS\t\2/
' \
| sort \
| column -t

Hunt Secrets Manager access¶

Display event time, requested secret identifier, and failure code or SUCCESS.

Find GetSecretValue operations

jq -r '
select(
  .eventSource=="secretsmanager.amazonaws.com"
  and .eventName=="GetSecretValue"
)
| [
    .eventTime,
    (.requestParameters.secretId // "unknown"),
    (.errorCode // "SUCCESS")
  ]
| @tsv
' cloudtrail.ndjson \
| sort \
| column -t

Inspect successful-read metadata¶

Extract the secret identifier, version stage, and JSON key names without displaying actual values.

Inspect secret metadata without values

grep 'Secret read succeeded:' raw/lambda/000000.log \
| sed -E 's/^.*Secret read succeeded:[[:space:]]*//' \
| jq -r '
[
  .secret_id,
  .version_stage,
  (.json_keys | join(","))
]
| @tsv
' \
| column -t

Identify S3 staging attempts¶

Spot post-secret-access staging or exfiltration attempts.

Find S3 theft staging attempts

grep 'Theft staging attempt:' raw/lambda/000000.log \
| sed -E '
s/^([0-9TZ:\.\-]+).*bucket=([^ ]+).*key=([^ ]+).*/\1\tATTEMPT\ts3:\/\/\2\/\3/
' \
| sort \
| column -t

Module 4 — Injection and input fuzzing¶

Extract S3-trigger events¶

Pull the first Records[] entry from each incoming S3 event and show event time, event name, object URI, and source IP.

Extract S3-trigger events

grep 'Incoming event: {"Records"' raw/lambda/000000.log \
| sed -E 's/^.*Incoming event: //' \
| jq -r '
.Records[0]
| [
    .eventTime,
    .eventName,
    ("s3://" + .s3.bucket.name + "/" + .s3.object.key),
    .requestParameters.sourceIPAddress
  ]
| @tsv
' \
| sort -u \
| sort \
| column -t

Inspect one S3-trigger event¶

Understand the event structure before building additional jq filters.

Inspect an S3-trigger event

grep 'Incoming event: {"Records"' raw/lambda/000000.log \
| head -n 1 \
| sed -E 's/^.*Incoming event: //' \
| jq '
.Records[0]
| {
    eventSource,
    eventName,
    eventTime,
    s3: {
      bucket: .s3.bucket.name,
      key: .s3.object.key
    },
    sourceIPAddress: .requestParameters.sourceIPAddress
  }
'

Extract received objects¶

Determine which object the Lambda function processed and its approximate input size.

Extract received S3 objects

grep -Rhs --include="*.log" "Injection: received object" raw/lambda \
| sed -E '
s/^([0-9TZ:\.\-]+).*Injection:[[:space:]]+received[[:space:]]+object[[:space:]]+/\1\tRECEIVED\t/;
s/[[:space:]]+bytes=/\tbytes=/
' \
| sort \
| column -t \
| head -n 12

Correlate objects with parse failures¶

Use the Lambda request UUID to associate each received object with its parser error.

Correlate received objects and parse failures

grep -Rhs --include="*.log" -E \
'Injection:[[:space:]]+received[[:space:]]+object|Injection:[[:space:]]+parse[[:space:]]+failed' \
raw/lambda \
| awk '
function tidy(s) {
  gsub(/[[:space:]]+/, " ", s)
  sub(/^ /, "", s)
  sub(/ $/, "", s)
  return s
}

function short_s3(s) {
  sub(/^s3:\/\/[^\/]+\//, "s3://.../", s)
  return s
}

{
  match($0, /^[0-9TZ:\.\-]+/)
  ts = substr($0, RSTART, RLENGTH)

  match(
    $0,
    /[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/
  )
  id = substr($0, RSTART, RLENGTH)

  if ($0 ~ /Injection:[[:space:]]+received[[:space:]]+object/) {
    match($0, /s3:\/\/[^[:space:]]+/)
    obj = substr($0, RSTART, RLENGTH)

    obj = short_s3(obj)
    recv[id] = obj

    print ts "\tRECEIVED\t" obj
  }

  else if ($0 ~ /Injection:[[:space:]]+parse[[:space:]]+failed/) {
    msg = $0

    sub(
      /^.*Injection:[[:space:]]+parse[[:space:]]+failed[[:space:]]*/,
      "",
      msg
    )

    msg = tidy(msg)

    obj = (id in recv) ? recv[id] : "-"

    print ts "\tPARSE_FAILED\t" obj "\t" msg
  }
}
' \
| sort \
| column -t \
| head -n 80

Correlate objects with tracebacks¶

Read all Lambda .log files, track invocation IDs and associated S3 objects, detect traceback blocks, extract handler.py locations, normalize JSONDecodeError messages, and emit TRACEBACK or UNHANDLED rows.

The beginning of the original command was not visible. The file enumeration and xargs -0 cat wrapper below are a faithful reconstruction, not a guaranteed byte-for-byte copy. Validate it against the local log format before operational use.

Correlate S3 objects and Lambda tracebacks

find raw/lambda -type f -name '*.log' -print0 \
| xargs -0 cat \
| gawk '
function tidy(s) {
  gsub(/[[:space:]]+/, " ", s)
  sub(/^ /, "", s)
  sub(/ $/, "", s)
  return s
}

function short_s3(s) {
  sub(/^s3:\/\/[^\/]+\//, "s3://.../", s)
  return s
}

function clean_exc(s) {
  s = tidy(s)
  sub(
    /^json\.decoder\.JSONDecodeError:[[:space:]]*/,
    "JSONDecodeError: ",
    s
  )
  sub(/: line .*/, "", s)
  return s
}

# Track the most recent timestamp.
# Lambda traceback lines may not all repeat the timestamp.
/^[0-9]{4}-[0-9]{2}-[0-9]{2}T/ {
  match($0, /^[0-9TZ:\.\-]+/)
  last_ts = substr($0, RSTART, RLENGTH)
}

# Capture the current Lambda invocation request ID.
/START RequestId:/ {
  if (match($0, /RequestId: [0-9a-f-]{36}/)) {
    cur_id = substr($0, RSTART, RLENGTH)
    sub(/^RequestId: /, "", cur_id)
  }
}

# Associate each request UUID with the S3 object being processed.
/Injection:[[:space:]]+received[[:space:]]+object/ {
  if (match($0, /[0-9a-f-]{36}/)) {
    id = substr($0, RSTART, RLENGTH)
  } else {
    id = cur_id
  }

  if (match($0, /s3:\/\/[^[:space:]]+/)) {
    obj = substr($0, RSTART, RLENGTH)
    recv[id] = short_s3(obj)
  }
}

# Start traceback collection.
/Traceback \(most recent call last\):/ {
  in_tb = 1
  tb_file = "-"
  tb_exc = ""
  tb_id = cur_id
  next
}

# Capture Python source locations such as handler.py:232.
in_tb && /File "/ {
  if (
    match(
      $0,
      /File "[^"]*handler\.py", line [0-9]+/
    )
  ) {
    loc = substr($0, RSTART, RLENGTH)

    sub(/^.*handler\.py", line /, "handler.py:", loc)

    tb_file = loc
  }
  next
}

# Capture the final exception line after a traceback.
in_tb && /^[A-Za-z_][A-Za-z0-9_.]*Error:/ {
  tb_exc = clean_exc($0)

  obj = (tb_id in recv) ? recv[tb_id] : "-"

  print last_ts "\tTRACEBACK\t" obj "\t" tb_file "\t" tb_exc

  in_tb = 0
  tb_file = "-"
  tb_exc = ""
  next
}

# Handle Lambda runtime unhandled-exception markers.
/Runtime\.Unhandled/ {
  obj = (cur_id in recv) ? recv[cur_id] : "-"

  print last_ts "\tUNHANDLED\t" obj "\t-\tRuntime.Unhandled"

  in_tb = 0
}

# Also handle generic unhandled wording if it appears.
/Unhandled exception/ {
  obj = (cur_id in recv) ? recv[cur_id] : "-"

  print last_ts "\tUNHANDLED\t" obj "\t-\t" tidy($0)

  in_tb = 0
}
' \
| sort \
| column -t \
| head -n 60

High-value CloudTrail fields¶

Field	Investigation value
`eventTime`	When the API action occurred
`eventSource`	AWS service that handled the request, such as Lambda, IAM, STS, or Secrets Manager
`eventName`	Specific API operation
`userIdentity`	Identity that performed the action
`userIdentity.type`	Identity class, including `Root`, `IAMUser`, and `AssumedRole`
`userIdentity.arn`	ARN of the calling identity or STS session
`sourceIPAddress`	Source IP associated with the request
`errorCode`	CloudTrail error when the operation failed
`requestParameters`	Target role, function, secret, resource, or other API parameters

High-value AWS API and event names¶

Activity	API or event names
Lambda tampering	`UpdateFunctionConfiguration`, `UpdateFunctionCode`
STS and identity discovery	`GetCallerIdentity`, `AssumeRole`
IAM reconnaissance	`ListRoles`, `ListPolicies`, `ListAttachedRolePolicies`, `ListAccessKeys`, `GetRole`
Secrets Manager access	`GetSecretValue`
S3 and Lambda processing	`ObjectCreated:Put`