Operator On The Wire
Join
← Back to Knowledge Base
BLUE TEAM / THREAT HUNT / CLOUD

AWS CloudTrail

This note documents detection patterns related to AWS Account & Cloud Resource Abuse using CloudTrail and supporting telemetry.

Cloud attacks are not about binaries — they are about identity and API abuse.

Authentication → Identity Abuse → Privilege Escalation → Resource Access → Exfiltration → Cleanup


Triage with Splunk

# Install
docker pull splunk/splunk:latest

# Start (LOCALHOST ONLY - SAFE)
docker run -d --name splunk `
-p 127.0.0.1:8000:8000 `
-p 127.0.0.1:8089:8089 `
-e SPLUNK_GENERAL_TERMS=--accept-sgt-current-at-splunk-com `
-e SPLUNK_START_ARGS=--accept-license `
-e SPLUNK_PASSWORD="password" `
splunk/splunk

# Stop / Cleanup (burn after use mindset)
docker stop splunk  
docker start splunk  
docker rm splunk

Normalize JSON Records

# CloudTrail stores multiple events inside Records[]
# MUST flatten or Splunk will break parsing

# For non-compressed logs
find CloudTrail/ -name '*.json' -exec cat {} + | jq -c '.Records[]' > cloudtrail.json

# For .gz logs 
find CloudTrail/ -name '*.json.gz' -exec zcat {} + | jq -c '.Records[]' > cloudtrail.json

Ingest with Splunk

# Enable delete (useful during iteration)
"Settings" -> "Roles" -> "admin" -> Tick "can_delete"

# Copy logs into container
docker cp TARGET_LOGS/ splunk:/

# Verify container
docker exec -it splunk bash

# Add data (IMPORTANT: ignore Digest logs)
"Home" -> "Add Data" -> "Monitor" -> "(Files & Directories)" -> "Finish"

# Timestamp Fix (CloudTrail = UTC)
"Timestamp" -> "Advanced" ->
    "Time zone" = "GMT"
    "Timestamp format" = "%Y-%m-%dT%H:%M:%SZ"
    "Timestamp fields" = "eventTime"

# Prevent truncation of JSON
"Advanced" -> "New setting" ->
    "TRUNCATE" = "0"

"Save As" -> "CloudTrail"

# Reset index during testing
* | delete

Key Fields

FieldWhy It MattersWhat to Look For
userIdentity.typeIdentity contextIAMUser, AssumedRole, AWSService
userIdentity.userNameActor identityUnexpected users, service accounts
userIdentity.accessKeyIdCredential trackingPivot compromised keys
userIdentity.sessionContext.sessionIssuer.userNameRole executionLambda / EC2 acting as role
sourceIPAddressOriginRare IPs, geo changes, non-AWS IPs
userAgentExecution methodaws-cli, Boto3, custom scripts
eventSourceServiceIAM, S3, Lambda, CloudWatch
eventNameActionCore of detection
awsRegionLocationUnusual regions
requestParameters.roleRole assignmentDetect PassRole abuse
requestParameters.policyArnPrivilege changesAdministratorAccess
requestParameters.bucketNameData targetExfil location
requestParameters.keyObject accessedFile-level visibility
requestParameters.prefixTargeted accessDirectory-based exfil
responseElementsOutcomeSuccess vs failure

Direct Indicators

Log SourceEvent NameMeaningForensic ValueNotes
CloudTrailConsoleLoginSuccessful AWS loginCriticalCheck MFAUsed, sourceIPAddress, responseElements.ConsoleLogin ("Success"), unusual geo/IP
CloudTrailCreateUserNew IAM user createdCriticalLook at userIdentity, sourceIPAddress, immediate follow-up actions (keys/policies)
CloudTrailCreateAccessKeyAccess key createdCriticalCheck userIdentity.userName, pivot on accessKeyId, often followed by CLI usage
CloudTrailDeleteAccessKeyAccess key removedHighCould indicate cleanup or key rotation
CloudTrailAttachUserPolicyPolicy attached to userCriticalInspect requestParameters.policyArn, look for AdministratorAccess
CloudTrailPutUserPolicyInline policy addedCriticalCheck requestParameters.policyDocument, often stealth escalation
CloudTrailSetDefaultPolicyVersionPolicy version switchedCriticalClassic privesc, compare old vs new policy permissions
CloudTrailAddUserToGroupUser added to groupCriticalCheck group privileges (admin groups = escalation)
CloudTrailCreateRoleNew role createdHighCheck trust policy (AssumeRolePolicyDocument)
CloudTrailUpdateAssumeRolePolicyTrust policy modifiedCriticalEnables role abuse / cross-account access
CloudTrailPassRole (implicit)Role assigned to resourceCriticalNOT logged directly → infer via requestParameters.role in resource creation
CloudTrailAssumeRoleRole assumedCriticalInspect userIdentity.sessionContext.sessionIssuer.userName, track lateral movement
CloudTrailGetCallerIdentityIdentity checkMediumOften attacker first action after gaining creds
CloudTrailListUsers / ListRolesIAM enumerationHighRecon phase, often follows initial access
CloudTrailGetPolicyVersionPolicy inspectionHighAttacker reading permissions before escalation
CloudTrailRunInstancesEC2 instance launchedCriticalCheck requestParameters.iamInstanceProfile, attacker compute
CloudTrailStartInstancesInstance startedHighCould indicate reusing existing infra
CloudTrailStopInstancesInstance stoppedMediumAnti-forensics / cost control
CloudTrailTerminateInstancesInstance deletedHighCleanup / evidence destruction
CloudTrailCreateFunction20150331Lambda createdCriticalCheck requestParameters.role, handler, runtime, small codeSize suspicious
CloudTrailInvokeLambda executedCriticalExecution event → pivot to actions done by role
CloudTrailUpdateFunctionCodeLambda code modifiedCriticalPayload update / persistence
CloudTrailGetObjectObject downloaded from S3CriticalCheck requestParameters.bucketName, object key, spike = exfil
CloudTrailPutObjectObject uploaded to S3HighCheck if external IP, staging/exfil
CloudTrailDeleteObjectObject deletedHighAnti-forensics
CloudTrailListBucketsS3 enumerationHighOften first step before exfil
CloudTrailListObjects / ListObjectsV2Object enumerationCriticalPre-exfil step, check prefix
CloudTrailDeleteBucketReplicationReplication removedHighCould disable persistence
CloudTrailPutBucketReplicationReplication addedCriticalData exfil persistence mechanism
CloudTrailDeleteTrailCloudTrail deletedCriticalLogging disabled → attacker covering tracks
CloudTrailStopLoggingLogging disabledCriticalImmediate blindspot
CloudTrailCreateTrailNew trail createdMediumCould be attacker-controlled logging
CloudTrailDeleteLogGroupLogs deleted (CloudWatch)CriticalAnti-forensics
CloudTrailDescribeLogGroupsLog discoveryHighAttacker mapping logging
CloudTrailDeleteDetectorGuardDuty disabledCriticalDetection evasion
CloudTrailListDetectorsGuardDuty enumerationHighRecon
CloudTrailPutRolePolicyInline role policy addedCriticalPrivilege escalation via roles
CloudTrailAttachRolePolicyManaged policy attached to roleCriticalOften used with AdministratorAccess
CloudTrailCreateLoginProfileConsole password createdCriticalEnables interactive login
CloudTrailUpdateAccessKeyAccess key status changeHighMay indicate persistence or cleanup

Indirect Indicators (Behavioral Detection)

IndicatorWhat To Look ForForensic ValueNotes
Login from rare IPNew sourceIPAddress, new ASN, new countryCriticalCompare against historical IPs for that user; cloud attackers often rotate IPs
No MFA loginMFAUsed=false in ConsoleLoginCriticalEspecially suspicious for privileged users or console access
Suspicious UserAgentaws-cli, Boto3, custom scripts, unusual clientsCriticalHuman attackers often use aws-cli; compare vs expected service agents (Lambda, EC2)
Burst of IAM actionsMultiple IAM API calls in short timeframeCriticale.g. ListPolicies → GetPolicyVersion → SetDefaultPolicyVersion = privesc chain
New access key after loginCreateAccessKey shortly after authCriticalIndicates persistence; pivot on accessKeyId immediately
Policy attached to userAttachUserPolicy, PutUserPolicyCriticalLook for AdministratorAccess or broad * permissions
Policy version switchSetDefaultPolicyVersionCriticalClassic escalation — attacker activates more permissive version
Role assigned to resourcerequestParameters.role in Create* eventsCriticalIndicates iam:PassRole abuse (not logged directly)
AssumeRole anomaliesAssumeRole from new IP / unusual userCriticalCheck sessionIssuer, cross-account ARNs = lateral movement
Identity using multiple IPsSame userIdentity.userName with many IPsHighIndicates attacker hopping infrastructure
Same IP across multiple usersSingle IP tied to many identitiesHighCredential reuse / attacker infrastructure
Rapid API chainingMany different eventName in secondsCriticalAutomated attacker scripts (not human clicking)
Unusual region activityEvents in new awsRegionHighAttackers often use different regions
Lambda creation + executionCreateFunction → Invoke close togetherCriticalPayload deployment + execution
Small Lambda payloadcodeSize very small (<1KB)HighOften simple exfil scripts
S3 access spikeMany GetObject in short timeCriticalData exfiltration
S3 prefix targetingrequestParameters.prefix usedCriticalTargeted data theft (e.g. CustomerB/2024-12-09/)
External IP + S3 accessGetObject from non-AWS IPCriticalStrong exfil indicator
Unexpected role usageRole used outside normal serviceCriticale.g. Lambda role used manually
CLI usage on service accountaws-cli for non-human accountCriticalService accounts should not use CLI
GuardDuty activityListDetectors, DeleteDetectorCriticalAttacker discovering/disabling detection
Logging tamperingStopLogging, DeleteTrail, DeleteLogGroupCriticalEquivalent to clearing logs
Resource cleanup after activityTerminateInstances, Delete*HighAnti-forensics
Short-lived sessionsRapid start/stop activityMediumFailed attack or recon
Odd time activityOutside business hoursHighCombine with other signals for confidence

Quick Triage Queries

General

# Active Identities
index=* userIdentity.type!="AWSService"  
| stats count by userIdentity.userName  
| sort -count

# Source IP
index=*  
| stats count by sourceIPAddress  
| sort -count

# User Agent
index=*  
| stats count by userAgent
| sort -count

# Full Timeline
index=* userIdentity.type!="AWSService"  
| eval _time=strptime(eventTime, "%Y-%m-%dT%H:%M:%SZ")  
| table _time userIdentity.userName eventName sourceIPAddress  
| sort _time

Scrutiny

# On bucket
* requestParameters.bucketName="forela-fileshare"
| table eventName, _time, sourceIPAddress, userIdentity.arn
| sort _time

# By denied access failures
* sourceIPAddress="54.242.59.197" errorCode=AccessDenied
| table eventName, _time, userAgent, userIdentity.arn
| sort _time

Common Tools / Tradecraft

Tool / MethodUsage
AWS CLINative attacker interface
boto3 / SDKsProgrammatic abuse
Stolen credentialsInitial access
IAM abusePrivilege escalation
Terraform / scriptsMass changes
PacuAWS exploitation framework
CloudSploit / ScoutSuiteRecon tools
Custom scriptsAutomation

Relevant Artifacts

Cloud Logs

  • CloudTrail (PRIMARY)
  • CloudTrail Insights (anomaly detection)
  • CloudWatch Logs

IAM

  • Users, Roles, Policies
  • Access keys (active/inactive)
  • Trust relationships

Storage (S3)

  • Object access logs
  • Bucket policies
  • Public exposure settings

Compute (EC2)

  • Instance metadata
  • Security groups
  • AMI usage

Network

  • VPC Flow Logs
  • Public IP activity
  • Security group changes

Other

  • AWS Config (resource changes)
  • GuardDuty findings
  • Route53 DNS logs

Decision Tree

  1. Was there a login?

    • ConsoleLogin
    • MFA used?
    • Source IP expected?
  2. Identity abused?

    • CreateAccessKey
    • CreateUser
    • AssumeRole
  3. Privilege escalation?

    • AttachUserPolicy
    • PutUserPolicy
    • UpdateAssumeRolePolicy
  4. Discovery?

    • ListUsers
    • ListBuckets
    • GetCallerIdentity
  5. Resource abuse?

    • RunInstances
    • S3 access (GetObject)
  6. Defense evasion?

    • DeleteTrail
    • StopLogging
  7. Timeline correlation

    • Login → IAM → Resource → Exfil

Attack Patterns

Credential Abuse + Exfil

GetCallerIdentity
→ ListBuckets
→ ListObjects
→ GetObject (mass)

Privilege Escalation via Policy

ListPolicies
→ GetPolicyVersion
→ SetDefaultPolicyVersion
→ AttachUserPolicy (Admin)

Lambda-Based Execution (Stealth)

CreateFunction
→ PassRole
→ Invoke
→ GetObject / IAM actions

Persistence via IAM

CreateUser
→ CreateAccessKey
→ AttachUserPolicy (Admin)

Defense Evasion

ListDetectors
→ DeleteDetector
→ DescribeLogGroups
→ DeleteLogGroup
→ StopLogging


MITRE ATT&CK References

  • T1078 Valid Accounts
  • T1098 Account Manipulation
  • T1069 Permission Groups Discovery
  • T1087 Account Discovery
  • T1021 Remote Services (via AssumeRole)
  • T1530 Data from Cloud Storage
  • T1485 Data Destruction
  • T1562 Impair Defenses