AWS English

My Brief Thoughts on the AWS Kinesis Outage

There have been multiple analyses about the recent (2020/11/25) outage of AWS Kinesis and its cascading failure mode, taking a chunk of AWS services with it — including seemingly unrelated Cognito — due to dependencies hidden to the user. If you haven’t read the official postmortem statement by AWS yet, go read it now.

There are an infinite amount of arguments that can made about cascading failure; I’m not here to talk about that today. I’m here to talk about a time a few years ago I was evaluating a few systems to do event logging. Naturally, Kinesis was in consideration, and our team interviewed an AWS Solution Architect about potential design patterns we could implement, what problems they would solve, what hiccups we may encounter on the way, et cetera.

At the time, I didn’t think much of it, but in hindsight it should have been a red flag.

ME: “So, what happens when Kinesis goes down? What kind of recovery processes do you think we need to have in place?”

SA: “Don’t worry about that, Kinesis doesn’t go down.”

The reason I didn’t think much of it at that time was that our workload would have been relatively trivial for Kinesis, and I mentally translated the reply to “don’t worry about Kinesis going down for this particular use case”.

We decided to not go with Kinesis for other reasons (I believe we went with Fluentd).

Maybe my mental translation was correct? Maybe this SA had this image of Kinesis as a system that was impervious to fault? Maybe it was representative of larger problem inside AWS of people who overestimated the reliability of Kinesis? I don’t know. This is just a single data point — it’s not even that strong. A vague memory from “a few years ago”? I’d immediately dismiss it if I heard it.

The point of this post is not to disparage this SA, nor to disparage the Kinesis system as a whole (it is extremely reliable!), but to serve as a reminder (mostly to myself) that one should be immediately suspicious when someone says “never” or “always”.


Venturing in to the realm of Hackintosh-ing

Like you, I’ve been finding myself working from home more often than not. These days, I probably go to an office once a month. I have a 16 inch MacBook Pro, but using it in clamshell mode, all the time, connected to a 4K monitor was… not ideal. It would often thermally throttle way down (often, it would be really sluggish — I wondered, how fast is this running? 800MHz. 90C. Fans at 100%.)

The 16″ MacBook Pro was a great machine for when I’d go to an office 3-5 days a week, but it just doesn’t make sense when it’s essentially used as a desktop.

This is where I thought, why don’t I just get a desktop Mac, then? That gives me a few choices: iMac, iMac Pro, Mac Pro, Mac Mini.

iMac (Pro): Pretty good performance, but a little expensive for my target price. Also, I had just gotten a new 4K monitor, and I didn’t want to get rid of it so soon.

Mac Pro: Way too expensive.

Mac Mini: Probably pretty similar in performance and thermal characteristics to my MacBook Pro.

I didn’t want to spend too much on it — especially with the Apple Silicon Macs coming out, meaning Intel support is on its way out, limiting the longevity of whatever I’d be buying. I’ll probably get a 2nd or 3rd generation Apple Silicon Mac, so what I wanted is something high performance that makes sense to bridge the gap of a couple years.

This led me down the rabbit hole of building a Hackintosh. My target price was “less than $2,000” (the price also is due to tax reasons — I thought it would be more fun to build a PC than it would be to calculate depreciation over 4 years). r/hackintosh on Reddit was a huge help — it was an invaluable resource when picking parts. I would look through everyone’s success stories and pick similar (or identical) parts. Here’s my entry (from when it had Catalina — I have since upgraded it to Big Sur).

Using some parts from my previous PC, the new parts came out to about $1,500. Not bad for a 10-core i9-10900k with 32GB RAM and a 1TB SSD.

I’ve never done water cooling before, and it sounded pretty cool, so I got an all-in-one unit that was really easy to install. Building a custom water cooling loop sounds fun, but really time and tool intensive — not something I have a lot of here in Tokyo. Maybe next time.

All in all, doing this Hackintosh was a fun little project. There are a lot of excellent resources that hold your hand through the whole process. If you’re interested in trying it out for yourself, I recommend reading the OpenCore Install Guide and looking through the aforementioned Reddit community.

When I started out to write this blog post, I installed Catalina 10.15.7 with OpenCore 0.6.2. Since then, I’ve upgraded to Big Sur 11.0.1 with OpenCore 0.6.3. I’ve only done one major upgrade, but it was relatively smooth.

One word of warning, though: Hackintosh-ing is definitely not for the faint of heart, or someone who is not prepared to spend hours debugging some issue that involves reading a bunch of white text on a black background. I would not recommend this to anyone who doesn’t like fiddling with computers. I had a bunch of small, strange issues (Ethernet instability, hardware video decoding, to name a couple) that required multiple reboots and trial-and-error with the config.plist configuration file.

Thanks for reading! If you have any questions, don’t hesitate to leave a comment or send me a Tweet.

WordPress 日本語


このブログは、WordPress のカテゴリーを使って言語を分けています。



日本語の投稿の執筆時に、「日本語」というカテゴリーを選択します(厳密にいうと、カテゴリーのスラッグが japanese であることを使っています)。そうすると、その記事の設定が日本語設定になって、フォントの設定やテーマの翻訳が自動的に日本語になります。

コードは GitHub 上に公開しましたが、日本語しか対応していないためプラグインディレクトリに提出しておりません。使いたい方はご自身でダウンロードしてカスタマイズした上で使ってください。もし不明点や興味があればコメントや Twitter で声かけてくれると嬉しいです。

AWS WordPress 日本語

WordPress を AWS Lambda で運用する

以前 WordPress を AWS Lambda で 運用する記事を投稿 (英語) しましたが、EFS対応前に執筆しました。EFS を使える様になって、WordPress を AWS Lambda 内の実行環境が完全に変わったので新しい記事を書きました。

今回は、SAM より Terraform を選びました。理由はいくつかありますが、主には私が管理するインフラはほぼ Terraform で管理されているのため、既存環境と融合性が優れてる。

Terraform モジュールとして公開しています。ソースコードは GitHub で公開しています。


まあまあいいよ。このリンク先で稼働しています。めちゃくちゃ速いわけでもないけど、遅すぎというわけでもない。CloudFrontを利用して静的アセットをキャッシュし、 opcache をチューニングしたらだいぶ速くなった。

通常なら Lambda が同時並行で起動されるときは個々のインスタンスが独立されて実行されれますが、 EFS を使えば異なる Lambda のインスタンスを跨いでファイルシステムを同期させることができる。このため、通常通り WordPress の更新、テーマやプラグインインストール、アップロード等利用できる。


今回のチュートリアルでは、 Lambda のソースコードに入ってるのは PHP を実行する環境のみ。 WordPress のファイル等は、 EFS のボリュームにインストールするので、別途 EC2 のインスタンスを用意する必要があります。


  1. 有効なAWSアカウント
  2. インターネットにアクセスできるプライベートサブネット。 EFS を使うために VPC 内に Lambda を起動する制約がありますが、そうすると NATゲートウェイNATインスタンス (比較) を使わないとインターネットにアクセスできない。
  3. Terraform 0.12 以上
  4. MySQL データベース
  5. WordPress のファイルを初期インストールするための EC2 インスタンス

Terraform が作成するリソース一覧はこちらにあります


今回は独立した Terraform モジュールを使いますが、他の Terraform の環境に埋め込む場合は適宜修正してください。

このチュートリアルをそのまま使う場合、us-west-2 リージョンを使ってください。PHP は標準で提供されていないため、カスタムの Lambda レイヤーを使って実行します。私が公開したレイヤーが現在、 us-west-2 しか公開していない。他のリージョンでも公開することを努めていますが、その間は私がフォークした php-lambda-layer を直接作成することができます。

1. EC2インスタンスを起動する

今回、 t3a.nano を選びました。amazon-efs-utils のパッケージを予めインストールしておくのはおすすめです。

コンソールにいるついでに、データベースにアクセスできるセキュリティグループのIDと Lambda を起動するプライベートサブネットのIDをメモしておいてください。

2. Terraform 環境を作成する
$ git clone
$ cd ./terraform-aws-wordpress-on-lambda-efs


# ステップ1でメモしたセキュリティグループのIDを配列に入れます
security_group_ids = ["sg-XXX"]

# ステップ1でメモしたサブネットのIDを配列に入れます
subnet_ids = ["subnet-XXX", "subnet-XXX", "subnet-XXX"]

もし のデフォルトドメインより、カスタムドメインを使う場合は、 acm_certificate_namedomain_name 変数も指定します。

Apply すると、terraform がリソースを作ってくれます。

$ terraform apply

AWS 認証情報を求められる場合、中止 (Ctrl-C) した上、環境変数で認証情報を設定してください。私の場合、複数のAWS環境を管理しているので、AWS_PROFILE という環境変数をよく使います。

Terraform が実際にインフラのリソースを作成する前に、実際稼働しているインフラのリソースの差分を出します。今回は新しく作成しているはずなので、全て「追加」というように出ると思います。確認した上で、 yes を答えてください。

CloudFront distribution が入ってるため、apply に多少時間かかる(僕の場合は、全部で5分ぐらいかかりました)。完了したら、アウトプット変数がいくつか表示されます。この変数をまただす場合は、 terraform output コマンドを使ってください。


3. EC2 に EFS のファイルシステムをマウントする

EC2 が EFS にアクセスするためにセキュリティグループをアサインする必要があります。ステップ2の Terraform が EFS にアクセスできるためのセキュリティグループを作ってくれたので、それを使いました。 efs_security_group_id のアウトプット変数の値を EC2 インスタンスに貼り付けてください

次、EC2 にログインして、 EFS をマウントしましょう。下記のコマンドから、fs-XXXXXefs_file_system_id のアウトプット変数の値で置き換えてください。

$ sudo -s
# mkdir /mnt/efs
# mount -t efs fs-XXXXX:/ /mnt/efs


4. WordPress をインストールしましょう

ファイルシステムがマウントされて、やっと WordPress のファイルをインストールできるようになりました。 Terraform がランダムで新しいディレクトリを作った( /mnt/efs/roots/wp-lambda-$ランダム )ので、そこに cd しましょう。

最新の WordPress をダウンロードして、そこに解凍してください。

ここから、通常通りの WordPress のインストールを進められることができると思います。カスタムドメインを利用していない場合は cloudfront_distribution_domain_name でアクセスできます。カスタムドメインを利用している場合は、CloudFront のdistribution のドメインに CNAME を向けて、指定したドメインでアクセスできると思います。



  • アップロードファイルを EFS より S3 にアップロードする。私は Humanmade の S3 Uploads プラグインをよく使います。
  • src/php.ini に入ってる opcache の設定を調節する。
  • 静的アセット( JS / CSS 等)を軽量な nginx サーバーで返す。
  • handler.php を調節して Cache-Control を追加する。これによって、 CloudFront のキャッシュがもっと使えるようになります。


AWS のサービスの技術的な制約によって下記のようなリミットがあります。


  • FTP やログイン可能な SSH アクセスがありません。EC2 インスタンスを使ってファイルを管理しないといけない。
  • 無限に同時並行で実行できるプラットフォーム Lambda から接続数が有限な MySQL に接続します。もし接続で問題になるところがあれば Aurora Serverless を試してみる価値があると思います。


質問、難しかったところ、改善してほしいところ、コメント等は下記のコメントフォームや Twitter で連絡してください。

AWS English WordPress

WordPress on AWS Lambda (EFS Edition)

I previously wrote a post about running WordPress on AWS Lambda, but it was before EFS support was announced (EFS is a managed network file system AWS provides). Being able to use EFS completely changes the way WordPress works in Lambda (for the better!), so I felt it warranted a new blog post.

In addition, this time I’m using Terraform instead of SAM. This matches the existing infrastructure-as-code setup I use when I deploy infrastructure for clients. Here’s the Terraform module (source code).


It works. It’s OK. Check it out, it’s running here. It’s not the best, but it isn’t bad, either. The biggest performance bottleneck is the EFS filesystem, and there’s no getting around that. PHP is serving static assets bundled with WordPress as well, which adds to some latency (in this configuration, CloudFront is caching most of these files, however). Tuning opcache to cache files in memory longer helped a lot.

Because EFS is synchronized across all the instances of Lambda, online updates, installs, and uploads work as expected.

What You’ll Need

In this setup, Lambda is only used for running PHP — installing the initial WordPress files is done on an EC2 instance that has the EFS volume mounted. This is a list of what you’ll need.

  1. An AWS account.
  2. A VPC with Internet access through a NAT gateway or instance (comparison). This is important because EFS connectivity requires Lambda to be set up in a VPC, but it won’t have Internet access by default.
  3. Terraform (the module uses v0.12 syntax, so you’ll need to use v0.12.)
  4. A MySQL database (I’m using MySQL on RDS using the smallest instance available)
  5. An EC2 instance to perform the initial setup and install of WordPress.

For a list of the resources that Terraform will provision, take a look at the Resources page here.


These steps assume you’re running this Terraform module standalone — if you want to run it in the context of an existing Terraform setup, prepare to adjust accordingly.

If you’re following this step-by-step, be sure to choose the us-west-2 region. Lambda Layer that I’m using for this is only published in the us-west-2 region. I’m working on getting the layer published in other regions, but in the meantime, use my fork of the php-lambda-layer to create your own in the region of your choosing.

1. Start the EC2 instance.

(If it isn’t already running)

I’m using a t3a.nano instance. Install the amazon-efs-utils package to get ready for mounting the EFS volume.

Also, while you’re in the console, note down the ID of a Security Group that allows access to RDS and the IDs of the private subnets to launch Lambda in.

2. Get Terraform up and running.
$ git clone
$ cd ./terraform-aws-wordpress-on-lambda-efs

Create a file called, and put the following contents in to it:

# An array of the Security Group IDs you listed in step 1.
security_group_ids = ["sg-XXX"]

# An array of the Subnet IDs you listed in step 1.
subnet_ids = ["subnet-XXX", "subnet-XXX", "subnet-XXX"]

If you want to use a custom domain name (instead of the default randomly-generated CloudFront domain name), set the acm_certificate_arn and domain_name variables as well.

Now, you’re ready to create the resources.

$ terraform apply

If you’re asked for your AWS credentials, Ctrl-C and try setting the authentication information via environment variables. I manage a lot of AWS accounts, so I use the AWS_PROFILE environment variable.

Terraform will ask you if you want to go ahead with the apply or not — look over the changes (the initial apply should not have any modifications or deletions), then respond yes.

When the apply has finished, you should see some outputs. If you don’t (or you already closed the window), you can always run terraform output. Keep this window open, you’ll need it in the next step.

3. Mount EFS on the EC2 instance.

First, we need to give the EC2 instance access to the EFS filesystem. Terraform created a security group for us (it’s in the efs_security_group_id output), so attach that to your EC2 instance.

Log in to your EC2 server, then mount the EFS filesystem (replace fs-XXXXX with the value of the efs_file_system_id output):

$ sudo -s
# mkdir /mnt/efs
# mount -t efs fs-XXXXX:/ /mnt/efs

If you’re having trouble mounting the filesystem, double check the security groups and take a look at the User Guide.

4. Install WordPress.

Now that the filesystem is mounted, we can finally proceed to install WordPress. Terraform automatically created a directory in the EFS filesystem (/mnt/efs/roots/wp-lambda-$RANDOM_STRING), so cd to there first. Download the latest version of WordPress, then extract the files there.

Now, you can go ahead with the famous five-minute install like you would with any other WordPress site! If you didn’t set a custom domain name, your site should be accessible at the domain name outputted at cloudfront_distribution_domain_name. If you did set a custom domain, then set a CNAME or alias to the CloudFront distribution domain name, then you should be able to access the site there.

Where to go from here

Here are some ideas for performance improvements that I haven’t tried, but should have some potential.

  • Upload files to S3 instead of WordPress. I use this plugin by Human Made: humanmade/S3-Uploads.
  • Experiment with adjusting the opcache settings in src/php.ini.
  • Use a lightweight nginx server to serve static assets from EFS to CloudFront.
  • Experiment with setting Cache-Control headers in handler.php for static files.


There are a couple hard limits imposed by AWS due to the technical limitations of the infrastructure.

Here are some other limitations that you’ll have to keep in mind.

  • No FTP / SSH access — you’ll need to manage an EC2 instance if you need command line or direct file access.
  • All the considerations of accessing a connection-oriented database from Lambda. You can try using Aurora Serverless if you run in to connection problems. RDS Proxy may also be able to provide you with a solution.


Thanks for reading! If you have any questions or comments, please don’t hesitate to leave a comment or send me a tweet.


Habits I’ll be keeping after COVID-19

During the COVID-19 pandemic, schools and daycares have been closed, so my family decided to use this as an opportunity to make some habits to make sure we can get through this period with minimal interruptions to life and work. Here are some habits that have worked so well for us that we’re planning on keeping them, even after the kids go back to school / daycare.

Keeping a schedule

We use a schedule to make sure the time we eat, sleep, and do activities are at regular times every day. At first, I thought this would be a good tool to let the kids know when we were working and when we could play together, but it’s proven to be a useful tool to both adults and children. I’ve written a blog post in Japanese about this as well.

Cleaning the house every day

This is related to the schedule, but we clean up the house every day at 5 PM, start the Roomba, take a bath, then get ready for dinner. Before, we would clean up whenever we felt like it, and as you can imagine, the room got pretty messy. We’d probably vacuum once a week or so. Now, the house stays clean and it’s much less stressful.

Taking the trash out

No, this isn’t a euphemism. Before, we would let the trash (especially recyclables like glass bottles, cans, etc) stock up before bringing it to the trash room, but now that we’re a little more flexible on time, we can take it down to the trash room immediately.


Rails on AWS: Do you need nginx between Puma and ALB?

When I set up Rails on AWS, I usually use the following pattern:

(CloudFront) → ALB → Puma

I was wondering: Is it always necessary to put nginx between the ALB and Puma server?

My theory behind not using nginx is that because it has its own queue (while the Classic Load Balancer had a very limited “surge queue”, the ALB does not have such a queue), it will help in getting responses back to the user (trading for increased latency) while hindering metrics used for autoscaling and choosing what backend to route the request to (such as Rejected Connection Count).

I couldn’t find any in-depth articles about this, so I decided to prove my theory (in)correct by myself.

In this test, the application servers will be running using ECS on Fargate (platform version 1.4.0). It’s a very simple “hello world” app, but I’ll give it a bit of room to breathe with each instance having 1 vCPU and 2GB of RAM. I’ll be using Gatling on a single c5n.large instance (“up to 25 gigabits” should be enough for this test).

In this test, I wanted to try out a few configurations that mimic characteristics of applications I’ve worked on: short and long requests, usually IO-bound. A short request is defined as just rendering a simple HTML template. A long request is 300ms. The requests are ramped from 1 request/sec to 1000 requests/sec over 5 minutes.

Response Time Percentiles over Time (OK responses), simple render — 4 instances, 20 threads each, connected directly to the ALB.
Response Time Percentiles over Time (OK responses), simple render — 4 instances, 20 threads each, using Nginx.

As you can see, for the simple render scenario, Nginx and Puma were mostly the same. As load approached 1000 requests/sec, latency started to get worse, but all requests were completed with an OK status.

The 300ms scenario was a little more grim.

Number of responses per second (green OK, red error), 300ms response — 4 instances, 20 threads each, connected directly to the ALB.
Number of responses per second (green OK, red error), 300ms response — 4 instances, 20 threads each, using Nginx.

My theory that Puma will fail fast and give error status to the ALB when reaching capacity was right. The theoretical maximum throughput is 4 instances * 20 threads * (1000ms in 1 second / 300ms) = 266 requests/sec. Puma handles about 200 requests/sec before returning errors; Nginx starts returning error status at around 275 requests/sec, but at that point requests are already queueing and the response time is spiking.

Remember, these results are for this specific use case, and results for a test specific to your use case probably will be different, so it’s always important to do load testing tailored to your environment, especially for performance critical areas.



私は料理が好きです。特にパンを焼くのが好きで、この間初めて Cookpad に公開したのはフォカッチャのレシピでした。フリーランスでシステム開発をしているエンジニアで、ゼロスタートから運用まで持っていくことを得意分野としています。この2つの要素を組み合わせて新しい何かを作るのが趣味ですが、そのことを今まであまりブログに書いたことはありませんでした。今回のアプリを作ったことで、「アプリは家庭料理みたいでもいい(An app can be a home-cooked meal)」という記事のことを思い出して、同様の考え方を共有したいと思いました。







私たちは以前セールで買って、あまり使っていなかった Amazon Fire 7 タブレットを使用して常時表示させていますが、インターネットにアクセスし、画面がある機器なら何でも大丈夫です。古くなったiPadや携帯の再利用などに便利だと思います。ちなみに、Androidの場合は電源につなぎながらスリープしない設定は開発者モードじゃないと設定できないようです。iOSは設定→画面→オートロックで設定できます。






Code Snippets English

A quick shortcut to open a Ruby gem in VS Code

While working on a Ruby project, I often find myself referring to the code of various libraries when it’s easier than looking up the documentation. For this, I used to use code (bundle show GEM_NAME), but recently I’ve been getting this warning:

[DEPRECATED] use `bundle info $GEM_NAME` instead of `bundle show $GEM_NAME`

Okay, that’s fine, but bundle info returns a bunch of stuff that would confuse VS Code:

> bundle info devise
  * devise (4.7.1)
	Summary: Flexible authentication solution for Rails with Warden
	Path: /Users/keita/.asdf/installs/ruby/2.7.0/lib/ruby/gems/2.7.0/gems/devise-4.7.1

Luckily there’s bundle info $GEM_NAME --path. code (bundle info devise --path) is kind of long to type out every time, though, so I decided to make an alias.

I use the Fish shell, so the code here is written for that shell. Adapt it to your shell as required. You’ll also need the VS Code terminal integration installed for this to work.

function bundlecode
  if test -e ./Gemfile
    code (bundle info $argv[1] --path)
    set_color -o red
    echo "Couldn't find `Gemfile`. Try again in a directory with a `Gemfile`."
    set_color normal


> bundlecode devise
# VS Code opens!

How I use Git

I’ve been using Git at work for around 10 years now. I started using Git with a GUI (Tower — back when I was eligible for the student discount!), but now I use the CLI for everything except complicated diffs and merges, where I use Kaleidoscope.

A question I get asked by my coworkers often is: “how in the world do you manage using Git without a GUI?”. This blog post is supposed to answer this question.

First, I use the Fish shell. It fits with the way I think. A lot of you probably use bash or zsh, that’s fine, there is lots of documentation on how to integrate Git with those shells. This is the relevant part of .config/fish/

set __fish_git_prompt_show_informative_status 'yes'
set __fish_git_prompt_color_branch magenta
set __fish_git_prompt_color_cleanstate green
set __fish_git_prompt_color_stagedstate red
set __fish_git_prompt_color_invalidstate red
set __fish_git_prompt_color_untrackedfiles cyan
set __fish_git_prompt_color_dirtystate blue

function fish_prompt
  # ... 
  set_color normal
  printf ' %s' (prompt_pwd)
  printf '%s' (__fish_git_prompt)
  printf ' > '

I’ve omitted the irrelevant portions (status checking, # prompt when root, etc. If you want to see the full file, I’ve posted it as a gist.

On a clean working directory (that is, no changed files that haven’t been committed to the repository), this looks like this:

When updating some files, it will change to something like this:

This prompt doesn’t change in real time, so changes from other terminals won’t automatically change this prompt. I have a habit of tapping the “return” key to update the prompt.

To commit these changes:

The commands I use most often:

  • git add (if you only want to add a portion of a file, git add -p is your friend) / git commit
  • git push / git pull (git pull --rebase for feature branches being shared with other devs)
  • git diff @ — show all changes, staged or not, between the working directory and the latest commit of the branch you are on (@ is a alias for HEAD)
  • git diff --cached — show only changes that are being staged for the next commit
  • git status
  • git difftool / git mergetool (this will open Kaleidoscope)

This was obviously a very cursory, high-level look at how I use Git, but I hope it was useful. It’s been a long time since I’ve used a Git GUI full time, but whenever I do use one (for example, when helping a coworker), it feels clunky compared to using the CLI (that’s not saying I don’t have my complaints about the CLI — that’s another blog post 😇).

If you have any more questions, leave a comment or contact me on Twitter, and I’ll update this post with the answers.