Ramblings of a CyberSecurity Nerd

Splunk Survival Series – Part 2: Getting Analytical

Welcome back to the Splunk Survival Series. In Part 1, we mapped the battlefield — indexes, sourcetypes, and clean field-driven searches. Now, we’re sharpening your blade with logic: using field operators, `eval`, `top`, and triage dashboards to pivot from data to decisions in seconds.

📌 Visibility Context

Logs are only as useful as your ability to filter them. In the SOC, this means two things:

  • Eliminating junk (field-based filters)
  • Surfacing anomalies (what breaks the baseline)

This is the pivot point from Tier 1 to Tier 2 mindset. You’re not searching — you’re slicing.

🔬 Field Operators: IN, NOT, LIKE

Stop writing five ORs in a row. Use these instead:

# Find failed logons for known service accounts
index=wineventlog EventCode=4625 
user IN ("svc-backup", "svc-splunk", "svc-api")

# Filter out internal IP space
index=firewall 
src_ip NOT IN ("10.0.0.0/8", "192.168.0.0/16")

# Match usernames that contain ‘test’
index=wineventlog user LIKE "test%"

These operators aren’t just cleaner — they’re faster. Let Splunk optimize what you would’ve written with a dozen ORs and wildcards.

📎 Analyst Behavior Snapshot

  • Sends: user="svc-backup" OR user="svc-splunk" OR user="svc-api"
  • System does: Parses it inefficiently, slows performance, returns clutter
  • What comes back: Everything, but it’s a pain to expand or reuse later

Replace that with IN, and your search becomes both readable and reusable.

🧪 Top and Rare: Surface Anomalies Fast

When you don’t know what you’re looking for, find what’s most common — or least.

# Top 10 failed login usernames
index=wineventlog EventCode=4625 
| top user

# Rare IPs seen in proxy logs
index=proxy 
| rare src_ip

Both `top` and `rare` are underused but deadly effective. They’re a poor man’s anomaly detector — and often all you need for fast threat surfacing.

🌐 Eval: Logic That Makes Splunk Think

`eval` turns your query into a living script. Use it to bucket time, tag anomalies, clean data, or score risk.

# Label common service accounts
| eval account_type=if(match(user, "svc-"), "service", "user")

# Mark login failures from outside your ASN
| eval risky=if(cidrmatch("10.0.0.0/8", src_ip), "no", "yes")

# Score based on failed count
| eval score=case(count>100,"Critical", count>50,"High", count>10,"Medium", true(),"Low")

Now your dashboard can sort by signal — not just volume.

📊 Splunk Query Examples

# Find unusual usernames triggering failures
index=wineventlog EventCode=4625 
| top user limit=15

# Evaluate login success/failure
index=wineventlog EventCode=4624 OR EventCode=4625 
| eval outcome=if(EventCode==4624,"Success","Fail")
| stats count by user, outcome

🛠️ Dashboard Design Strategy

Don’t build dashboards that just regurgitate logs. Build dashboards that surface triage logic.

  • Panels should answer questions: “Who’s failing?” “Where’s it coming from?” “What’s normal?”
  • Always pre-filter panels: Use `where`, `eval`, or `IN` to constrain searches
  • Use drilldowns for action: Clicking on a spike should pivot to detail view or log trace

🔐 Analyst Workflow Hardening

  • Force use of field operators (`IN`, `NOT`, `LIKE`) — no string filters
  • Train `top` and `rare` as day-one commands for noise hunting
  • Ban dashboards with raw log panels — everything should be summarized
  • Use `eval` to flag risky behavior, abnormal volume, or broken patterns

📋 Queries to Use Right Now

# Top 10 failed users from public IPs
index=wineventlog EventCode=4625 
| where NOT cidrmatch("10.0.0.0/8", src_ip) 
| top user

# Rare user-agent strings
index=proxy 
| rare user_agent

# Failed login triage
index=wineventlog EventCode=4625 
| stats count by src_ip, user 
| where count > 10
| eval priority=if(count>50,"High","Medium")

📚 Suggested Resources

🧾 Final Thoughts

This is the part of Splunk that separates the button-pushers from the analysts. Field logic, filtering, and eval-based triage don’t just make searches more accurate — they give you control. Because when an alert fires, you don’t just need data. You need an answer. And that answer starts here.

In Part 3, we dive into the power of `rex` and regular expressions — extracting artifacts from raw logs and slicing text into evidence like a surgeon.

Published: August 18, 2025

Leave a comment