DISQUS

Tobias Crawley · 3 months ago

Bryan: thanks for doing this! This is the answer to a problem I was planning on solving in the next couple of weeks. I'm looking forward to seeing what else I'm doing wrong.

joegrossberg · 3 months ago

This might be outdated, but I think these points are worth mentioning:

* Splunk can also take in logs from non-Rails apps (e.g. have your Rails, Java and Apache logs all in one place) and multiple projects
* Splunk can be overly strict about the logging format it expects :/
* Splunk is not real-time, unlike tail -f

bryanl · 3 months ago

How do you have real time when you are doing over 10 requests per second?

joegrossberg · 3 months ago

You don't. But the majority of Rails sites have a different level of traffic and using Splunk -- unlike reading your log files -- precludes the option of side-by-side debugging via "load the page while simultaneously seeing what appears in your terminal".

One size does not fit all, and a 10 req/s site is atypical.

waloeiii · 3 months ago

When using Splunk you don't have to send your logs to syslog, in fact I find it simpler to not do it. Every single one of my machines runs Splunk in Lightweight Forwarder Mode (http://www.splunk.com/base/Documentation/3.3.4/...) and they all forward to a central Splunk server. The Forwarding instances don't require licenses or anything, they will watch whatever files (or folders!) you configure in inputs.conf and then relay that to your central instance. If the central instance goes down, the Forwarders queue messages (up to a determined size) while waiting for the central server to respond. Forwarding with some aggressive logrotate configs keeps my log volume down on the working instances, and I now have 18 months of logs from 17 machines in one nice organized index.

@joegrossberg Splunk 3.x has a live-tail feature that I find is only ~2 seconds off real-time. Splunk 4.x is considerably faster and the regular search is only ~5 seconds off real-time (but no Live Tail in 4.x yet).

waloeiii · 3 months ago

I should also point out that you can configure the forwarders to send data to different indexes. Set the index variable in inputs.conf to coincide with the RAILS_ENV on the machine you are deploying to. Searching through splunk defaults to production (I renamed it from main), but if I want to find something on staging just add index=staging to the query. If you aren't sure of the environment you can just search across all indices.

Jerod Santo · 3 months ago

Hmm, lightweight forwarders seem like a nice way of going about this. Can you provide an example inputs.conf from one of your forwarding instances?

waloeiii · 3 months ago

host = <%= hostname %>
index = <%= environment %>
_blacklist = \.(tgz|gz)$

[monitor:///var/log]
disabled = false

[monitor:///data/onehub/shared/log]
disabled = false

[monitor:///data/onehub/shared/pids]
disabled = false

[monitor:///vol/log/mysql-slow.log]
disabled = false

[monitor:///vol/log/mysqld.log]
disabled = false

chris · 3 months ago

Any chance of this becoming a proper podcast so that I can subscribe?

bryanl · 3 months ago

Yes. I'm just working out some content, and then we will have a proper
podcast


======

Tricon · 2 months ago

It's hard to beat simple tools like tail -f when it comes to observing logs in realtime. I'm more interested in using Splunk to follow up on bug reports, issues, etc. -- things I won't hear about in realtime anyways.

bryanl · 2 months ago

I'm sure with little toy apps, "tail -f" works. When you outgrow that, you need something better.