Shipping Events from Fluentd to Elasticsearch

We use fluentd to process and route log events from our various applications. It’s simple, safe, and flexible. With at-least-once delivery by default, log events are buffered at every step before they’re sent off to the various storage backends. However, there are some caveats with using Elasticsearch as a backend.

Currently, our setup looks something like this:

The general flow of data is from the application, to the fluentd aggregators, then to the backends — mainly Elasticsearch and S3. If a log event warrants a notification, it’s published to a SNS topic, which in turn triggers a Lambda function that sends the notification to Slack.

The fluentd aggregators are placed by an auto-scaling group, but are not load balanced by a load balancer. Instead, a Lambda function connected to the auto-scaling group lifecycle notifications updates a DNS round-robin entry with the private IP addresses of the fluentd aggregator instances.

We use the fluent-plugin-elasticsearch plugin to output log events to Elasticsearch. However, because this plugin uses the bulk insert API and does not validate whether events have actually been successfully inserted in to the cluster, it is dangerous to rely on it exclusively (thus the S3 backup).

Upgrading PostgreSQL on Ubuntu

I recently started using Ubuntu Linux on my main development machine. That means that my PostgreSQL database is running under Ubuntu, as well. I’ve written guides to upgrading PostgreSQL using Homebrew in the past, but the upgrade process under Ubuntu was much smoother.

These steps are assuming that you use Ubuntu 16.04 LTS, and PostgreSQL 9.6 is already installed via apt.

  1. Stop the postgresql service.
    $ sudo service postgresql stop
  2. Move the newly-created PostgreSQL 9.6 cluster elsewhere.
    $ sudo pg_renamecluster 9.6 main main_pristine
  3. Upgrade the 9.5 cluster.
    $ sudo pg_upgradecluster 9.5 main
  4. Start the postgresql service.
    $ sudo service postgresql start

Now, when running pg_lsclusters, you should see something like the following:

9.5 main          5434 online postgres /var/lib/postgresql/9.5/main          /var/log/postgresql/postgresql-9.5-main.log
9.6 main          5432 online postgres /var/lib/postgresql/9.6/main          /var/log/postgresql/postgresql-9.6-main.log
9.6 main_pristine 5433 online postgres /var/lib/postgresql/9.6/main_pristine /var/log/postgresql/postgresql-9.6-main_pristine.log

Verify everything is working as expected, then feel free to remove the 9.5/main and 9.6/main_pristine clusters (pg_dropcluster).

These cluster commands may be available in other distros, but I haven’t been able to check them. YMMV. Good luck!

Podcasts I’m Listening To (November 2015 Edition)

My wife Naoko wrote a reply to this post. It was fun comparing how different the podcasts we listen to are. 🙂

First, I’d like to plug a podcast that I’m a semi-regular guest on, techsTalking(5417), a podcast where technology people just talk about whatever is on our mind.

Here are some other podcasts that I’m currently subscribed to:

  • The Incomparable — a podcast about anything geeky. Star Wars? Check. Star Trek? Check. Silly drafts? Check. Crazy movies? Check.
  • The Incomparable Game Show — born from The Incomparable proper, regular panelists play crazy games for your entertainment. On the podcast.
  • Incomparable Radio Theater — The Incomparable podcast, once upon a time, liked to do funny things on April Fools. Like, say: release a full-length episode in the format of old-time radio drama. Including equally funny sponsors (some fake, some real). Now, they’ve spun it off in to a separate podcast.
  • Random Trek — Incomparable regular Scott McNulty hosts a podcast with non-random guests talking about random episodes of Star Trek.
  • Robot or Not? — Is it a robot? Or not?
  • Astronomy Cast — A weekly “facts-based journey through the cosmos”.
  • Reconcilable Differences — Two of my favorite podcasters, John Siracusa and Merlin Mann, get together on one podcast.

A few other podcasts I listen to occasionally:

And assorted programming-specific podcasts.

bundler gotcha

So, this is a thing:

bundle install --without development:test
Bundle complete! XX Gemfile dependencies, XX gems now installed.
Gems in the groups development and test were not installed.


bundle install
Bundle complete! XX Gemfile dependencies, XX gems now installed.
Gems in the groups development and test were not installed.

Basically — you run bundle install --without <group> once, and that’s saved in .bundle/config. So next time you run bundle install without any arguments, it won’t install gems in the groups you specify.

It looks like it is fixed in Bundler 2.0, though.

Homebrew and PostgreSQL 9.4

Edit 2016/1/9 I have updated these instructions for upgrading from PostgreSQL 9.4 to 9.5.

As you may know, I am a big PostgreSQL user and fan. I also use Homebrew to manage 3rd party software packages on my Mac. PostgreSQL 9.4 was just released a couple days ago with some really cool features — a binary-format JSON datatype for speed and flexibility (indexes on JSON keys? Of course.), and some really good performance improvements. Read the release blog post and release notes for more information.

However, if you’ve used PostgreSQL before, you know that upgrading can be a little difficult. Here’s what you have to do to upgrade your Homebrew-installed PostgreSQL 9.3 to 9.4. Keep in mind, these steps are for a standard Homebrew installation — as long as you haven’t configured custom data directory paths, it should work.

  1. Turn PostgreSQL off first, just in case:
    $ launchctl unload ~/Library/LaunchAgents/homebrew.mxcl.postgresql.plist
  2. Update PostgreSQL itself:
    $ brew update && brew upgrade postgresql
  3. Make a new, pristine 9.4 database:
    $ initdb /usr/local/var/postgres9.4 -E utf8
  4. Migrate the data to the new 9.4 database:
    $ pg_upgrade \
      -d /usr/local/var/postgres \
      -D /usr/local/var/postgres9.4 \
      -b /usr/local/Cellar/postgresql/9.3.5_1/bin/ \
      -B /usr/local/Cellar/postgresql/9.4.0/bin/ \
  5. Move 9.4 data directory back to where PostgreSQL expects it to be:
    $ mv /usr/local/var/postgres /usr/local/var/postgres9.3
    $ mv /usr/local/var/postgres9.4 /usr/local/var/postgres
  6. Start PostgreSQL back up!
    $ launchctl load ~/Library/LaunchAgents/homebrew.mxcl.postgresql.plist

Note: If you’re using the pg gem for Rails, you should recompile:

$ gem uninstall pg
$ gem install pg