The Cold Truth: Turn Off All Default Rules First
Let me start with a hard truth. If you just deployed Splunk ES, turn off every single default correlation search. Don’t ask why. Ask the guy on Reddit who said it best: “Splunk ES default correlation searches should not be turned on!”
I’m not being dramatic. Default rules are designed for generic environments — and your environment is anything but generic. Turn them all on, and your SIEM becomes a noise machine, not a security tool.
Last year I worked with a client who ran out-of-box rules for three months. They averaged 2,000+ alerts per day. Less than 10 were actionable. Their SOC team burned out and eventually nuked everything, starting from scratch. That’s three months of wasted effort.
Normalize your data first. Then write rules. This is rule zero.
The “Correlation Sandwich” Structure
Writing Splunk correlation rules is a three-layer problem: gather data, structure information, return knowledge. I call it the “correlation sandwich” — raw logs on the bottom, field extraction and normalization in the middle, logic on top.
Most people jump straight to the search command and end up with a mess. Here’s the right way.
Step 1: Data Normalization (Don’t Skip This!)
Without normalization, your rules are worthless. You need to map logs from different sources to a unified field schema.
index=firewall OR index=proxy OR index=endpoint
| eval src_ip = coalesce(src_ip, clientip, source_ip, src)
| eval dest_ip = coalesce(dest_ip, dest, destination_ip, dst)
| eval user = coalesce(user, username, account_name, sAMAccountName)
Looks simple? 90% of teams fail here. If your field names aren’t consistent, your logic won’t work.
Step 2: Define the Threat Pattern
This is the core. You need to answer one question: What combination of events constitutes a real threat?
Example — detecting brute force:
index=windows EventCode=4625
| bucket _time span=5m
| stats count as failed_attempts by src_ip, TargetUserName
| where failed_attempts > 10
But is that enough? No. This rule will catch legitimate users fat-fingering their passwords too. Better approach:
index=windows EventCode=4625
| bucket _time span=5m
| stats count as failed_attempts by src_ip, TargetUserName
| where failed_attempts > 10
| lookup local=true privileged_accounts TargetUserName OUTPUT privileged
| where isnull(privileged) OR privileged="false"
| eval risk_score = failed_attempts * 10
See what I did there? Exclude normal users and add a risk score. That’s production-grade.
Step 3: Thresholds — Don’t Guess, Calculate
There’s a classic question: “What are the two types of thresholds which may be used in correlation searches?”
Answer: count-based and time-based.
But the real issue isn’t the type — it’s how you set them. I’ve seen people pick thresholds out of thin air: 5 failed logins? 10? 100?
The right way: run historical data and look at the distribution.
index=windows EventCode=4625
| timechart span=1h count by src_ip usenull=f
| eventstats avg(count) as avg_count, stdev(count) as stdev_count
| eval threshold = avg_count + (3 * stdev_count)
| where count > threshold
Using standard deviation to calculate dynamic thresholds is 100x better than guessing.
Best Practices Cheat Sheet
| Practice | Description | My Rating |
|---|---|---|
| Normalize First | Unify field mappings before writing logic | ⭐⭐⭐⭐⭐ |
| Disable Defaults | Start from zero, enable selectively | ⭐⭐⭐⭐⭐ |
Use lookup for Context | Enrich with asset, user, threat intel | ⭐⭐⭐⭐ |
| Dynamic Thresholds | Calculate from historical data, not hardcoded | ⭐⭐⭐⭐⭐ |
| Rule Severity Tiers | Differentiate critical vs informational | ⭐⭐⭐⭐ |
| Monthly Review | Review rule hit rates at least once a month | ⭐⭐⭐⭐⭐ |
| Keep Rules Simple | No single rule should exceed 50 lines | ⭐⭐⭐ |
Leverage risk Command | Splunk ES built-in threat scoring | ⭐⭐⭐⭐ |
Common Pitfalls (I’ve Hit All of Them)
Pitfall 1: Rules Too Broad
Writing overly broad rules generates noise. Example:
index=* "malware" OR "virus"
What does this detect? Everything. And nothing accurately. Narrow your data sources to only the logs that can produce valid alerts.
Pitfall 2: Ignoring Time Windows
Correlation is about event sequences. Without a time window, your rule just checks “if A and B exist anywhere” — completely ignoring order.
index=windows EventCode=4624
| join type=inner src_ip [search index=proxy dest_ip=10.0.0.1 | fields src_ip]
This join matches across all time, not a reasonable window. Use transaction or stats with time buckets instead.
Pitfall 3: Not Handling False Positives
First week after a rule goes live, you’ll get tons of false positives. Don’t rewrite the rule yet. Build an exclusion list first:
| search NOT [| inputlookup exclude_ips | fields src_ip]
Stabilize the exclusion list, then adjust the rule.
FAQ
How to write correlation rules in SIEM?
Three steps: normalize data, define threat patterns, set thresholds. Normalization unifies logs from different sources. Threat patterns define what constitutes an attack. Thresholds decide when to trigger. Don’t skip any step.
How to write correlation rules in Splunk?
Use the search command with stats, transaction, and lookup. Always use the risk command to score alerts instead of triggering directly. Write rules in Splunk ES’s Correlation Search interface, not as scheduled searches in normal search.
Why do we make correlation rules in SIEM?
Because a single log event is meaningless. One failed login could be a typo. Ten failed logins followed by one success — that’s a brute force attack. Correlation rules connect isolated events to reconstruct the attack chain.
What are the two types of thresholds which may be used in correlation searches?
Count-based (event count exceeds threshold) and time-based (events occur within a specific window). Production environments typically use both together.
Final Thoughts
While I was writing this, Hacker News had a thread about “Nano – open core siem built on rust and ClickHouse.” Great tech stack, but here’s the thing: SIEM effectiveness is never about the tech stack. It’s about rule quality and operational maturity.
You can run garbage rules on the fastest SIEM in the world — you’ll just generate more garbage faster.
So don’t chase shiny new toys. Get the fundamentals right. Write good rules. That beats everything else.
Based on community discussions from Reddit and Hacker News over the past 30 days, plus my own field experience. Rule syntax varies by environment — test thoroughly before production deployment.