Logging: A Vanishing Art Form

In March 2011, the Anti-Phishing Working Group (APWG)’s Internet Policy Committee published the responses to a survey of organizations whose Websites had been compromised and subsequently used to host phishing pages.

Images by zagazou76

I’ve been looking at the reported incidents since March
2011 to see if any trends emerge. While many aspects of defending Websites against attack and compromise still desperately need improvement, logging stands out as one
of the most clearly under-utilized.

One reason to log events on servers, hosts, and switching equipment is to record what is happening on your network so that you can identify unusual or suspicious activities. Yet when asked after an attack, “How was the attack discovered or reported?” only six percent of respondents indicated that they or their colleagues discovered it from Web server logs -- and that percentage appears to have dropped in the most recent sample.

A second reason to log events is to have information available during an incident to help identify and mitigate the attack. Thirty-four percent of respondents indicated that they reviewed system and Web server files as part of their incident response; again, that percentage appears to have dropped in the most recent sample.

It should come as no surprise that compromises are more commonly reported to web site owners than discovered by them. Even conceding that organizations increasingly outsource Web-hosting (and this is also reflected in the survey responses), log collection and analysis appear to be vanishing as arts and practices. Can it really be true that we’re not paying attention to logging in the era of advance persistent threats and state-sponsored cyber-espionage? Apparently so, and the reality is sobering, ironic, comical, and downright depressing.

In baseball parlance, I’m about to serve you a meatball: Educate your partners in the value, art, and practices of log collection. Here are a few recommendations to pass along:

Enable logging on all systems. There is enormous value in having multiple perspectives of the same events or traffic. Data on, for example, which system or executable is generating traffic, whether the same traffic is emanating from multiple systems, common traffic destinations, and commonly resolved domain names is essential when investigating a Website hack, data exfiltration attack, hosts running unapproved (unlicensed) software, or even performance bottlenecks.

Log as much as your pocketbook will spare. What and how much to log may seem like an art form, but many resources are available for you to recommend. Brian Honan’s presentation on log management offers simple, solid advice and examples. As you identify what you will log, keep in mind that you can learn as much from allowed events as denied.

Synchronize time across your network. Correlating events you log from switches and hosts is best accomplished when all systems are operating on the same UTC time. 

Collect logs at a central, secure repository. Centralized logging facilitates troubleshooting and incident response. You can also use log data to assess network performance; performance is often an easier "sell" in an organization than security. Secure, centralized logging can be implemented relatively inexpensively using syslog-ng. Dedicate a server to only collect logs. Physically secure the server, harden it, run it on a separate LAN segment, and restrict access to the server. Depending on your security needs, you can even choose to operate the log server in stealth mode.

Convincing your partners to embrace centralized logging is step one. The next and harder step is to convince and assist them in actually making use of what they’ve logged. Begin by pointing them to the dismal statistics I presented here. Emphasize that frequent log analysis is also beneficial in correcting network problems that affect availability (online presence) or performance. We’ll talk about what to recommend if your partners buy in to this proposition in a future article.

Originally posted at The Champion Community 16 August 2012

Comments

[Seth Hall]

Related to this discussion is logs derived from network traffic which Bro creates (http://www.bro-ids.org/). In my opinion we have turned logging into an art form and we put huge value and attention to our logs, what they contain, and that they're correct. Right now in our 2.1 distribution we do protocol logs for DNS, HTTP, Tunnels (6to4, Teredo, and AYIYA), SOCKS, Syslog, IRC, SSL, FTP, Connections, SMTP, SSH. In development we have Modbus, DNP3, and SMB.

The additional benefit is that the protocol logging is just barely scratching the surface of what Bro provides since Bro is a programming language and you can write your own scripts to analyze network traffic in any way you want.