ArcSight connector/esm flooding DNS servers – How to solve this?

I’m currently working on an ArcSight ESM instance to tune it and make it work so that this becomes a helpful tool. This in total is already an extraordinary time consuming project, because its not all done with just pumping logfiles into the ESM tool, you need to filter and setup rules to ensure what you see is the „unknown“ security threat and not the day-to-day known issues which are already handled like virus xyz cleaned on device A.

However, before I was able to work on the above item I was confronted with a simple DNS issue, which i thought was simple but at the end was causing some trouble and time to fix.

Problem: the ArcSight connectors we running for our firewalls, IPS systems, proxy servers as well as VPN gateways  are configured to resolve forward and reverse all the IPs/hostnames within an log record. In case of millions logs a day from firewalls and proxies this can cause a heavy load on your DNS servers. Especially the reverse lookups and NX answers are not cached per default not at the ArcSight connector and even not at a standard bind which is running as a caching server. Reverse lookup answers will be anyway not cached, if the answer comes from an non-authoritative DNS servers. And turning of name/ip resolution was not an option at all, because this information is very helpful.

So at the end it came down to the following requirements list for a caching DNS server:

• flexible cache size (in memory for high speeed, but also with dump capability), support for several 100 000 records
• Caching support for several DNS record types A, MX …
• Caching support for reverse lookups
• Caching of NX answers from non-auth dns servers
• TTL overwrite to extend the life time of cache entries (this is a pro and con, but many records have very small TTLS today and will make an caching DNS inefficient) 
• split horizon DNS, so that queries for internal DNS domains and IP ranges are send to internal DNS servers and external domains/non-internal and non-internal IPs to external Internet DNS servers
• Statistical output and logging

While caching at all is not a problem and their are several products or even configs/patches available, the caching of reverse lookups from non-auth DNS servers as well as TTL overwrite was the major problem. I did tester several DNS implementations and finally ended up with PDNSD  Dont be afraid that the latest version is from some years ago, however its still part of some Linux distros and can be installed also from package managers e.g. apt-get on Ubuntu and it can provide all the functionality as listed above.

After installing pdnsd you need to change/modify the pdnsd.config file located under /etc

Here an example with some comments:

root@edrsvlab1:~# cat /etc/pdnsd.conf
// Read the pdnsd.conf(5) manpage for an explanation of the options.

global {
perm_cache=2048000;  //we used 2G of cache, which should be ok until end of my life and just bbecause I can ;-)
cache_dir=“/var/cache/pdnsd“; //this will be the cache directory and log/debug file dir
run_as=“pdnsd“;

server_ip = any; //pdnsd will listen in all NICS for queries

status_ctl = on; //pdnsd has an control program called pdnsd-ctl which can be used to modify and control the daemon during run-time

paranoid=on;

proc_limit=10; //only use 10 processes in parallel

min_ttl=1d; // One day of min ttl to keep records in cache for 24 hours min.
max_ttl=1w; // One week, cleanup and clear records older than 1 week
neg_ttl=1d; // One day. NX ttl to one day
timeout=3; // Global timeout option, wait max 3 seconds for an answer

neg_rrs_pol=on; // always cache neg response even if answer comes from non auth server
neg_domain_pol=on;

daemon=on; // we run you as a daemon

verbosity=1; // if it works nobody needs much logging

query_method=udp_only; // UDP queries only, in case the answers become to big or truncated pdnsd will also switch to TCP

debug=off; //if it runs it runs and nobody needs debugging, for configuration check it is still helpfull
}

/* Now define which queries will be send to local/internal DNS servers */

server {
label=“lan“;
ip=<internal server IP>;
uptest=ping; // check whether the server is up via PING
proxy_only=off; // we also need caching, so turn proxy only off
timeout=3; // we wait for max 3 seconds
caching=on; // caching on
include=“.<internal domain #1>“; // dont miss the dot at the beginning to catch also subdomains
include=“.<internal domain #2>“;// dont miss the dot at the beginning to catch also subdomains
include=“.<internal domain #3>“;// dont miss the dot at the beginning to catch also subdomains
include=“.10.10.in-addr.arpa“; //internal IP address range 1
include=“.22.172.in-addr.arpa“; //internal IP address range 2
policy=excluded; // if there is no match in the above rules, default is exclude and dont use this server
}

 

/* Now define which queries will be send to Internet public  DNS servers */

 

server {
label=“public“;
ip=<dns server IP from your ISP>;
proxy_only=off; //same as above
timeout=3; //same as above
caching=on; //same as above
uptest=ping; //same as above

// now exclude all your internal stuff (domains, ip ranges), so that only the remaining lookups are send to your ISP
exclude=“.<internal domain #1>“; // dont miss the dot at the beginning to catch also subdomains
exclude=“.<internal domain #2>“;// dont miss the dot at the beginning to catch also subdomains
exclude=“.<internal domain #3>“;// dont miss the dot at the beginning to catch also subdomains
exclude=“.10.10.in-addr.arpa“; //internal IP address range 1
exclude=“.22.172.in-addr.arpa“; //internal IP address range 2
}

 

/* THATS IT, HAVE FUN */

 

I hope this quick guide might be helpful to you, At least for me it solved a lot of trouble and ArcSight can now resolve forward and reverse until the cache explodes…