smarticus-blog - Latest Comments in #yerdoinitwrong episode 1: logging with syslog

Re: #yerdoinitwrong episode 1: logging with syslog

bryanl — Tue, 13 Oct 2009 08:46:35 -0000

I'm sure with little toy apps, "tail -f" works. When you outgrow that, you need something better.

Re: #yerdoinitwrong episode 1: logging with syslog

bryanl — Fri, 25 Sep 2009 22:54:54 -0000

Yes. I'm just working out some content, and then we will have a proper
podcast

======

Re: #yerdoinitwrong episode 1: logging with syslog

chris — Fri, 25 Sep 2009 17:17:54 -0000

Any chance of this becoming a proper podcast so that I can subscribe?

Re: #yerdoinitwrong episode 1: logging with syslog

waloeiii — Tue, 22 Sep 2009 20:13:28 -0000

host = <%= hostname %>
index = <%= environment %>
_blacklist = \.(tgz|gz)$

[monitor:///var/log]
disabled = false

[monitor:///data/onehub/shared/log]
disabled = false

[monitor:///data/onehub/shared/pids]
disabled = false

[monitor:///vol/log/mysql-slow.log]
disabled = false

[monitor:///vol/log/mysqld.log]
disabled = false

Re: #yerdoinitwrong episode 1: logging with syslog

Jerod Santo — Tue, 22 Sep 2009 17:09:52 -0000

Hmm, lightweight forwarders seem like a nice way of going about this. Can you provide an example inputs.conf from one of your forwarding instances?

Re: #yerdoinitwrong episode 1: logging with syslog

joegrossberg — Wed, 16 Sep 2009 14:18:52 -0000

You don't. But the majority of Rails sites have a different level of traffic and using Splunk -- unlike reading your log files -- precludes the option of side-by-side debugging via "load the page while simultaneously seeing what appears in your terminal".

One size does not fit all, and a 10 req/s site is atypical.

Re: #yerdoinitwrong episode 1: logging with syslog

bryanl — Wed, 16 Sep 2009 13:35:12 -0000

How do you have real time when you are doing over 10 requests per second?

Re: #yerdoinitwrong episode 1: logging with syslog

waloeiii — Wed, 16 Sep 2009 13:07:25 -0000

I should also point out that you can configure the forwarders to send data to different indexes. Set the index variable in inputs.conf to coincide with the RAILS_ENV on the machine you are deploying to. Searching through splunk defaults to production (I renamed it from main), but if I want to find something on staging just add index=staging to the query. If you aren't sure of the environment you can just search across all indices.

Re: #yerdoinitwrong episode 1: logging with syslog

waloeiii — Wed, 16 Sep 2009 13:04:39 -0000

When using Splunk you don't have to send your logs to syslog, in fact I find it simpler to not do it. Every single one of my machines runs Splunk in Lightweight Forwarder Mode (http://www.splunk.com/base/... and they all forward to a central Splunk server. The Forwarding instances don't require licenses or anything, they will watch whatever files (or folders!) you configure in inputs.conf and then relay that to your central instance. If the central instance goes down, the Forwarders queue messages (up to a determined size) while waiting for the central server to respond. Forwarding with some aggressive logrotate configs keeps my log volume down on the working instances, and I now have 18 months of logs from 17 machines in one nice organized index.

@joegrossberg Splunk 3.x has a live-tail feature that I find is only ~2 seconds off real-time. Splunk 4.x is considerably faster and the regular search is only ~5 seconds off real-time (but no Live Tail in 4.x yet).

Re: #yerdoinitwrong episode 1: logging with syslog

joegrossberg — Wed, 16 Sep 2009 12:28:29 -0000

This might be outdated, but I think these points are worth mentioning:

* Splunk can also take in logs from non-Rails apps (e.g. have your Rails, Java and Apache logs all in one place) and multiple projects
* Splunk can be overly strict about the logging format it expects :/
* Splunk is not real-time, unlike tail -f

Re: #yerdoinitwrong episode 1: logging with syslog

Toby Crawley — Wed, 16 Sep 2009 12:21:45 -0000

Bryan: thanks for doing this! This is the answer to a problem I was planning on solving in the next couple of weeks. I'm looking forward to seeing what else I'm doing wrong.