LetsDefend: AWS PerSEStence — building a timeline from CloudTrail

AWS PerSEStence, rated Hard on LetsDefend.

This one was a fun one!

Cloud incident response is a genuinely different discipline, and this challenge is a great demonstration of why. There is no disk to image. There is no memory to capture. There is an enormous pile of JSON, and the whole job is turning it into a story.

The problem with CloudTrail#

CloudTrail logs every API call made in your account. That is exactly what you want during an investigation, and exactly what makes it unreadable raw: thousands of deeply nested JSON events, one per call, and most of them completely routine.

My first instinct was cat. That instinct was wrong.

cat cloudtrail-events.json

A wall of minified JSON scrolling past faster than you can read it, with the one field you care about buried four levels deep.

jq is the actual tool#

jq '.' cloudtrail-events.json

Same data, formatted and indented. Suddenly you can see the structure — and once you can see the structure, you can start asking questions of it instead of reading it.

Then this is the one I kept coming back to:

# What happened, when, by whom, from where
jq -r '.Records[] | [.eventTime, .eventName, .userIdentity.arn, .sourceIPAddress] | @tsv' \
  cloudtrail-events.json

Four columns per event. That is the difference between a file you cannot read and a table you can sort.

And do not skip ls. Before parsing anything, list what you were actually given — how many files, how they are split, what dates they cover. It is very easy to analyse one file thoroughly and never notice there were six.

Reading CloudTrail like an investigator#

Four fields carry most of the story:

• eventName is the API call. Persistence in AWS almost always shows up as identity activity — creating users, creating access keys, attaching policies, creating roles, editing trust relationships.
• userIdentity tells you which principal acted, and whether it was root, an IAM user, or an assumed role. An assumed role turning up somewhere it never has before is worth a second look.
• sourceIPAddress is your pivot. One confirmed malicious event gives you an address, and that address gives you everything else from the same origin.
• errorCode is underrated. Failed calls map the attacker's reconnaissance — what they tried before something worked tells you what they were after.

Building the timeline#

This is the real deliverable, and I think it is the skill the challenge is actually testing.

Sort everything relevant by eventTime and write the sequence out in plain language. Initial access, then what they enumerated, then what they created or changed to keep their access, then what they did with it.

The output is not a list of suspicious events. It is a narrative with an order and a cause — this call succeeded, which made the next one possible. And if two events could genuinely have happened in either order, say so rather than implying a certainty the logs do not support.

Why persistence is the interesting part#

The name is the hint.

Getting in is one event. Staying in is a pattern. In AWS that usually means the attacker created something durable — a new access key, an extra IAM user, a modified trust policy — so that losing the original foothold costs them nothing.

Which is exactly why containment has to be scoped from the timeline and not from the initial alert. Rotate the compromised credential and stop there, and everything they built afterwards is still working.

What did we learn?#

1. Forming a timeline is the core of incident response, not a write-up formality.
2. jq over cat for any JSON at volume — and @tsv to make it sortable.
3. Use ls to see all of your evidence before analysing a subset of it.
4. CloudTrail becomes readable once you know which four fields carry the story.
5. Compiling it into one narrative — how they got in, and what they did to stay in — is the finished product.

If you are working through this one too, I would like to hear how you got on.

Keep practising!

References#

• LetsDefend — where this challenge lives
• AWS CloudTrail record contents — every field, defined
• jq manual — worth thirty minutes of your life