Working with DynamoDB Global Tables

Just some stuff I’ve picked up while working with DynamoDB Global Tables. This was my first time using it; I used it to move a few tables from one region to another without downtime.

When deleting replica tables…

Note that this operation will delete the replica table and is non-reversible. This replica table cannot be re-added later to the global table.

This warning message is a little misleading — the replica table will be deleted, but it’s possible to re-create a new replica table in the region that was deleted.

Replica cannot be deleted because it has acted as a source region for new replica(s) being added to the table in the last 24 hours.

You have to wait for 24 hours before you can delete the source region.

Other stuff:

  • If you create a GSI in one region, it will automatically be created in all other regions as well.
  • If you delete a table from the list of tables (instead of the “Global Tables” tab), it will delete normally. All of the other tables in other regions will be unaffected.

For those times you don’t want to eval…

I wanted to make some advanced logic available, easily configurable via a database, in a couple apps that I’ve been working on recently.

Honestly, I could have just stored the code in the database and eval’d it — but no, I don’t want to take the risk of arbitrarily executing code. I could have done some gymnastics like running it in a network-less container with defined inputs and outputs. I could have made the configuration more capable. What I decided to do in the end, though, was to write an extremely compact domain-specific language (DSL).

To write this DSL, I chose a Lisp-style syntax due to its dead-simple parsing. The basic idea is to parse the string for tokens, generate an abstract syntax tree (AST), then just recurse through the AST and run whatever code is required.

In one example, I wanted to have an extendible SQL WHERE clause, with a very limited set of operators — AND, OR, LIKE, =.

(and (like (attr "") (str "%Keita%")) (= (attr " (str "Tokyo")))

This example will generate the SQL WHERE clause:

(( LIKE '%Keita%') AND ( = 'Tokyo'))

Here’s pseudocode for how I write the parser / interpreter for this:

  "and": (left, right) => { return f"(({left}) AND ({right}))" }
  "or":  (left, right) => { return f"(({left}) OR ({right}))" }
  "like":(left, right) => { return f"(({left}) LIKE ({right}))" }
  "=": (left, right) => { return f"(({left}) = ({right}))" }
  "str": (str) => { return escape_sql(str) }
  "attr": (str) => { return escape_sql_for_attr_name(str) }

def parse_ast(code):
  # parse the "code" string into nested arrays:
  # "(1 (2 3))" becomes ["1", ["2", "3"]]

def execute_node(ast):
  cmd = ast[0]
  argv = ast[1:]
  resolved_argv = [ execute_node(x) for x in argv ]
  return COMMANDS[cmd](*resolved_argv)

def execute(code):
  ast = parse_ast(code)

As you can see, this is a very simple example that takes the DSL and transforms it in to a SQL string. If you wanted to do parameterized queries, you might return a tuple with the string as the first element and a map of parameters for the second, for example.

The ability to map the language so closely to the AST, and being able to evaluate the AST just by recursion, makes this implementation simple and easy to write, easy to extend, and easy to embed in existing applications. While I probably won’t be switching to writing Common Lisp full time (for practical reasons), I definitely do get the appeal of the language itself.

This tool isn’t something I use all the time. It’s probably something that should be used very sparingly, and in specific circumstances. That said, it’s a good tool in my toolbox for those times for when I want to have on-the-fly customizable logic without the security concerns of using eval, or the complexity of creating a sandboxed environment for potentially unsafe code.

Last note: while this solution may be more secure than eval, it is definitely not 100% secure. In the simple example above, we do escape strings so SQL injection shouldn’t be a problem, but it doesn’t check if the column defined by the attr function is valid, or if the user is allowed to query information based on that column or not (although something like that would be possible). I would not use something like this to process completely untrusted input.

AWS English

My Brief Thoughts on the AWS Kinesis Outage

There have been multiple analyses about the recent (2020/11/25) outage of AWS Kinesis and its cascading failure mode, taking a chunk of AWS services with it — including seemingly unrelated Cognito — due to dependencies hidden to the user. If you haven’t read the official postmortem statement by AWS yet, go read it now.

There are an infinite amount of arguments that can made about cascading failure; I’m not here to talk about that today. I’m here to talk about a time a few years ago I was evaluating a few systems to do event logging. Naturally, Kinesis was in consideration, and our team interviewed an AWS Solution Architect about potential design patterns we could implement, what problems they would solve, what hiccups we may encounter on the way, et cetera.

At the time, I didn’t think much of it, but in hindsight it should have been a red flag.

ME: “So, what happens when Kinesis goes down? What kind of recovery processes do you think we need to have in place?”

SA: “Don’t worry about that, Kinesis doesn’t go down.”

The reason I didn’t think much of it at that time was that our workload would have been relatively trivial for Kinesis, and I mentally translated the reply to “don’t worry about Kinesis going down for this particular use case”.

We decided to not go with Kinesis for other reasons (I believe we went with Fluentd).

Maybe my mental translation was correct? Maybe this SA had this image of Kinesis as a system that was impervious to fault? Maybe it was representative of larger problem inside AWS of people who overestimated the reliability of Kinesis? I don’t know. This is just a single data point — it’s not even that strong. A vague memory from “a few years ago”? I’d immediately dismiss it if I heard it.

The point of this post is not to disparage this SA, nor to disparage the Kinesis system as a whole (it is extremely reliable!), but to serve as a reminder (mostly to myself) that one should be immediately suspicious when someone says “never” or “always”.


Venturing in to the realm of Hackintosh-ing

Like you, I’ve been finding myself working from home more often than not. These days, I probably go to an office once a month. I have a 16 inch MacBook Pro, but using it in clamshell mode, all the time, connected to a 4K monitor was… not ideal. It would often thermally throttle way down (often, it would be really sluggish — I wondered, how fast is this running? 800MHz. 90C. Fans at 100%.)

The 16″ MacBook Pro was a great machine for when I’d go to an office 3-5 days a week, but it just doesn’t make sense when it’s essentially used as a desktop.

This is where I thought, why don’t I just get a desktop Mac, then? That gives me a few choices: iMac, iMac Pro, Mac Pro, Mac Mini.

iMac (Pro): Pretty good performance, but a little expensive for my target price. Also, I had just gotten a new 4K monitor, and I didn’t want to get rid of it so soon.

Mac Pro: Way too expensive.

Mac Mini: Probably pretty similar in performance and thermal characteristics to my MacBook Pro.

I didn’t want to spend too much on it — especially with the Apple Silicon Macs coming out, meaning Intel support is on its way out, limiting the longevity of whatever I’d be buying. I’ll probably get a 2nd or 3rd generation Apple Silicon Mac, so what I wanted is something high performance that makes sense to bridge the gap of a couple years.

This led me down the rabbit hole of building a Hackintosh. My target price was “less than $2,000” (the price also is due to tax reasons — I thought it would be more fun to build a PC than it would be to calculate depreciation over 4 years). r/hackintosh on Reddit was a huge help — it was an invaluable resource when picking parts. I would look through everyone’s success stories and pick similar (or identical) parts. Here’s my entry (from when it had Catalina — I have since upgraded it to Big Sur).

Using some parts from my previous PC, the new parts came out to about $1,500. Not bad for a 10-core i9-10900k with 32GB RAM and a 1TB SSD.

I’ve never done water cooling before, and it sounded pretty cool, so I got an all-in-one unit that was really easy to install. Building a custom water cooling loop sounds fun, but really time and tool intensive — not something I have a lot of here in Tokyo. Maybe next time.

All in all, doing this Hackintosh was a fun little project. There are a lot of excellent resources that hold your hand through the whole process. If you’re interested in trying it out for yourself, I recommend reading the OpenCore Install Guide and looking through the aforementioned Reddit community.

When I started out to write this blog post, I installed Catalina 10.15.7 with OpenCore 0.6.2. Since then, I’ve upgraded to Big Sur 11.0.1 with OpenCore 0.6.3. I’ve only done one major upgrade, but it was relatively smooth.

One word of warning, though: Hackintosh-ing is definitely not for the faint of heart, or someone who is not prepared to spend hours debugging some issue that involves reading a bunch of white text on a black background. I would not recommend this to anyone who doesn’t like fiddling with computers. I had a bunch of small, strange issues (Ethernet instability, hardware video decoding, to name a couple) that required multiple reboots and trial-and-error with the config.plist configuration file.

Thanks for reading! If you have any questions, don’t hesitate to leave a comment or send me a Tweet.

WordPress 日本語


このブログは、WordPress のカテゴリーを使って言語を分けています。



日本語の投稿の執筆時に、「日本語」というカテゴリーを選択します(厳密にいうと、カテゴリーのスラッグが japanese であることを使っています)。そうすると、その記事の設定が日本語設定になって、フォントの設定やテーマの翻訳が自動的に日本語になります。

コードは GitHub 上に公開しましたが、日本語しか対応していないためプラグインディレクトリに提出しておりません。使いたい方はご自身でダウンロードしてカスタマイズした上で使ってください。もし不明点や興味があればコメントや Twitter で声かけてくれると嬉しいです。

AWS WordPress 日本語

WordPress を AWS Lambda で運用する

以前 WordPress を AWS Lambda で 運用する記事を投稿 (英語) しましたが、EFS対応前に執筆しました。EFS を使える様になって、WordPress を AWS Lambda 内の実行環境が完全に変わったので新しい記事を書きました。

今回は、SAM より Terraform を選びました。理由はいくつかありますが、主には私が管理するインフラはほぼ Terraform で管理されているのため、既存環境と融合性が優れてる。

Terraform モジュールとして公開しています。ソースコードは GitHub で公開しています。


まあまあいいよ。このリンク先で稼働しています。めちゃくちゃ速いわけでもないけど、遅すぎというわけでもない。CloudFrontを利用して静的アセットをキャッシュし、 opcache をチューニングしたらだいぶ速くなった。

通常なら Lambda が同時並行で起動されるときは個々のインスタンスが独立されて実行されれますが、 EFS を使えば異なる Lambda のインスタンスを跨いでファイルシステムを同期させることができる。このため、通常通り WordPress の更新、テーマやプラグインインストール、アップロード等利用できる。


今回のチュートリアルでは、 Lambda のソースコードに入ってるのは PHP を実行する環境のみ。 WordPress のファイル等は、 EFS のボリュームにインストールするので、別途 EC2 のインスタンスを用意する必要があります。


  1. 有効なAWSアカウント
  2. インターネットにアクセスできるプライベートサブネット。 EFS を使うために VPC 内に Lambda を起動する制約がありますが、そうすると NATゲートウェイNATインスタンス (比較) を使わないとインターネットにアクセスできない。
  3. Terraform 0.12 以上
  4. MySQL データベース
  5. WordPress のファイルを初期インストールするための EC2 インスタンス

Terraform が作成するリソース一覧はこちらにあります


今回は独立した Terraform モジュールを使いますが、他の Terraform の環境に埋め込む場合は適宜修正してください。

このチュートリアルをそのまま使う場合、us-west-2 リージョンを使ってください。PHP は標準で提供されていないため、カスタムの Lambda レイヤーを使って実行します。私が公開したレイヤーが現在、 us-west-2 しか公開していない。他のリージョンでも公開することを努めていますが、その間は私がフォークした php-lambda-layer を直接作成することができます。

1. EC2インスタンスを起動する

今回、 t3a.nano を選びました。amazon-efs-utils のパッケージを予めインストールしておくのはおすすめです。

コンソールにいるついでに、データベースにアクセスできるセキュリティグループのIDと Lambda を起動するプライベートサブネットのIDをメモしておいてください。

2. Terraform 環境を作成する
$ git clone
$ cd ./terraform-aws-wordpress-on-lambda-efs


# ステップ1でメモしたセキュリティグループのIDを配列に入れます
security_group_ids = ["sg-XXX"]

# ステップ1でメモしたサブネットのIDを配列に入れます
subnet_ids = ["subnet-XXX", "subnet-XXX", "subnet-XXX"]

もし のデフォルトドメインより、カスタムドメインを使う場合は、 acm_certificate_namedomain_name 変数も指定します。

Apply すると、terraform がリソースを作ってくれます。

$ terraform apply

AWS 認証情報を求められる場合、中止 (Ctrl-C) した上、環境変数で認証情報を設定してください。私の場合、複数のAWS環境を管理しているので、AWS_PROFILE という環境変数をよく使います。

Terraform が実際にインフラのリソースを作成する前に、実際稼働しているインフラのリソースの差分を出します。今回は新しく作成しているはずなので、全て「追加」というように出ると思います。確認した上で、 yes を答えてください。

CloudFront distribution が入ってるため、apply に多少時間かかる(僕の場合は、全部で5分ぐらいかかりました)。完了したら、アウトプット変数がいくつか表示されます。この変数をまただす場合は、 terraform output コマンドを使ってください。


3. EC2 に EFS のファイルシステムをマウントする

EC2 が EFS にアクセスするためにセキュリティグループをアサインする必要があります。ステップ2の Terraform が EFS にアクセスできるためのセキュリティグループを作ってくれたので、それを使いました。 efs_security_group_id のアウトプット変数の値を EC2 インスタンスに貼り付けてください

次、EC2 にログインして、 EFS をマウントしましょう。下記のコマンドから、fs-XXXXXefs_file_system_id のアウトプット変数の値で置き換えてください。

$ sudo -s
# mkdir /mnt/efs
# mount -t efs fs-XXXXX:/ /mnt/efs


4. WordPress をインストールしましょう

ファイルシステムがマウントされて、やっと WordPress のファイルをインストールできるようになりました。 Terraform がランダムで新しいディレクトリを作った( /mnt/efs/roots/wp-lambda-$ランダム )ので、そこに cd しましょう。

最新の WordPress をダウンロードして、そこに解凍してください。

ここから、通常通りの WordPress のインストールを進められることができると思います。カスタムドメインを利用していない場合は cloudfront_distribution_domain_name でアクセスできます。カスタムドメインを利用している場合は、CloudFront のdistribution のドメインに CNAME を向けて、指定したドメインでアクセスできると思います。



  • アップロードファイルを EFS より S3 にアップロードする。私は Humanmade の S3 Uploads プラグインをよく使います。
  • src/php.ini に入ってる opcache の設定を調節する。
  • 静的アセット( JS / CSS 等)を軽量な nginx サーバーで返す。
  • handler.php を調節して Cache-Control を追加する。これによって、 CloudFront のキャッシュがもっと使えるようになります。


AWS のサービスの技術的な制約によって下記のようなリミットがあります。


  • FTP やログイン可能な SSH アクセスがありません。EC2 インスタンスを使ってファイルを管理しないといけない。
  • 無限に同時並行で実行できるプラットフォーム Lambda から接続数が有限な MySQL に接続します。もし接続で問題になるところがあれば Aurora Serverless を試してみる価値があると思います。


質問、難しかったところ、改善してほしいところ、コメント等は下記のコメントフォームや Twitter で連絡してください。

AWS English WordPress

WordPress on AWS Lambda (EFS Edition)

I previously wrote a post about running WordPress on AWS Lambda, but it was before EFS support was announced (EFS is a managed network file system AWS provides). Being able to use EFS completely changes the way WordPress works in Lambda (for the better!), so I felt it warranted a new blog post.

In addition, this time I’m using Terraform instead of SAM. This matches the existing infrastructure-as-code setup I use when I deploy infrastructure for clients. Here’s the Terraform module (source code).


It works. It’s OK. Check it out, it’s running here. It’s not the best, but it isn’t bad, either. The biggest performance bottleneck is the EFS filesystem, and there’s no getting around that. PHP is serving static assets bundled with WordPress as well, which adds to some latency (in this configuration, CloudFront is caching most of these files, however). Tuning opcache to cache files in memory longer helped a lot.

Because EFS is synchronized across all the instances of Lambda, online updates, installs, and uploads work as expected.

What You’ll Need

In this setup, Lambda is only used for running PHP — installing the initial WordPress files is done on an EC2 instance that has the EFS volume mounted. This is a list of what you’ll need.

  1. An AWS account.
  2. A VPC with Internet access through a NAT gateway or instance (comparison). This is important because EFS connectivity requires Lambda to be set up in a VPC, but it won’t have Internet access by default.
  3. Terraform (the module uses v0.12 syntax, so you’ll need to use v0.12.)
  4. A MySQL database (I’m using MySQL on RDS using the smallest instance available)
  5. An EC2 instance to perform the initial setup and install of WordPress.

For a list of the resources that Terraform will provision, take a look at the Resources page here.


These steps assume you’re running this Terraform module standalone — if you want to run it in the context of an existing Terraform setup, prepare to adjust accordingly.

If you’re following this step-by-step, be sure to choose the us-west-2 region. Lambda Layer that I’m using for this is only published in the us-west-2 region. I’m working on getting the layer published in other regions, but in the meantime, use my fork of the php-lambda-layer to create your own in the region of your choosing.

1. Start the EC2 instance.

(If it isn’t already running)

I’m using a t3a.nano instance. Install the amazon-efs-utils package to get ready for mounting the EFS volume.

Also, while you’re in the console, note down the ID of a Security Group that allows access to RDS and the IDs of the private subnets to launch Lambda in.

2. Get Terraform up and running.
$ git clone
$ cd ./terraform-aws-wordpress-on-lambda-efs

Create a file called, and put the following contents in to it:

# An array of the Security Group IDs you listed in step 1.
security_group_ids = ["sg-XXX"]

# An array of the Subnet IDs you listed in step 1.
subnet_ids = ["subnet-XXX", "subnet-XXX", "subnet-XXX"]

If you want to use a custom domain name (instead of the default randomly-generated CloudFront domain name), set the acm_certificate_arn and domain_name variables as well.

Now, you’re ready to create the resources.

$ terraform apply

If you’re asked for your AWS credentials, Ctrl-C and try setting the authentication information via environment variables. I manage a lot of AWS accounts, so I use the AWS_PROFILE environment variable.

Terraform will ask you if you want to go ahead with the apply or not — look over the changes (the initial apply should not have any modifications or deletions), then respond yes.

When the apply has finished, you should see some outputs. If you don’t (or you already closed the window), you can always run terraform output. Keep this window open, you’ll need it in the next step.

3. Mount EFS on the EC2 instance.

First, we need to give the EC2 instance access to the EFS filesystem. Terraform created a security group for us (it’s in the efs_security_group_id output), so attach that to your EC2 instance.

Log in to your EC2 server, then mount the EFS filesystem (replace fs-XXXXX with the value of the efs_file_system_id output):

$ sudo -s
# mkdir /mnt/efs
# mount -t efs fs-XXXXX:/ /mnt/efs

If you’re having trouble mounting the filesystem, double check the security groups and take a look at the User Guide.

4. Install WordPress.

Now that the filesystem is mounted, we can finally proceed to install WordPress. Terraform automatically created a directory in the EFS filesystem (/mnt/efs/roots/wp-lambda-$RANDOM_STRING), so cd to there first. Download the latest version of WordPress, then extract the files there.

Now, you can go ahead with the famous five-minute install like you would with any other WordPress site! If you didn’t set a custom domain name, your site should be accessible at the domain name outputted at cloudfront_distribution_domain_name. If you did set a custom domain, then set a CNAME or alias to the CloudFront distribution domain name, then you should be able to access the site there.

Where to go from here

Here are some ideas for performance improvements that I haven’t tried, but should have some potential.

  • Upload files to S3 instead of WordPress. I use this plugin by Human Made: humanmade/S3-Uploads.
  • Experiment with adjusting the opcache settings in src/php.ini.
  • Use a lightweight nginx server to serve static assets from EFS to CloudFront.
  • Experiment with setting Cache-Control headers in handler.php for static files.


There are a couple hard limits imposed by AWS due to the technical limitations of the infrastructure.

Here are some other limitations that you’ll have to keep in mind.

  • No FTP / SSH access — you’ll need to manage an EC2 instance if you need command line or direct file access.
  • All the considerations of accessing a connection-oriented database from Lambda. You can try using Aurora Serverless if you run in to connection problems. RDS Proxy may also be able to provide you with a solution.


Thanks for reading! If you have any questions or comments, please don’t hesitate to leave a comment or send me a tweet.


Habits I’ll be keeping after COVID-19

During the COVID-19 pandemic, schools and daycares have been closed, so my family decided to use this as an opportunity to make some habits to make sure we can get through this period with minimal interruptions to life and work. Here are some habits that have worked so well for us that we’re planning on keeping them, even after the kids go back to school / daycare.

Keeping a schedule

We use a schedule to make sure the time we eat, sleep, and do activities are at regular times every day. At first, I thought this would be a good tool to let the kids know when we were working and when we could play together, but it’s proven to be a useful tool to both adults and children. I’ve written a blog post in Japanese about this as well.

Cleaning the house every day

This is related to the schedule, but we clean up the house every day at 5 PM, start the Roomba, take a bath, then get ready for dinner. Before, we would clean up whenever we felt like it, and as you can imagine, the room got pretty messy. We’d probably vacuum once a week or so. Now, the house stays clean and it’s much less stressful.

Taking the trash out

No, this isn’t a euphemism. Before, we would let the trash (especially recyclables like glass bottles, cans, etc) stock up before bringing it to the trash room, but now that we’re a little more flexible on time, we can take it down to the trash room immediately.


Rails on AWS: Do you need nginx between Puma and ALB?

When I set up Rails on AWS, I usually use the following pattern:

(CloudFront) → ALB → Puma

I was wondering: Is it always necessary to put nginx between the ALB and Puma server?

My theory behind not using nginx is that because it has its own queue (while the Classic Load Balancer had a very limited “surge queue”, the ALB does not have such a queue), it will help in getting responses back to the user (trading for increased latency) while hindering metrics used for autoscaling and choosing what backend to route the request to (such as Rejected Connection Count).

I couldn’t find any in-depth articles about this, so I decided to prove my theory (in)correct by myself.

In this test, the application servers will be running using ECS on Fargate (platform version 1.4.0). It’s a very simple “hello world” app, but I’ll give it a bit of room to breathe with each instance having 1 vCPU and 2GB of RAM. I’ll be using Gatling on a single c5n.large instance (“up to 25 gigabits” should be enough for this test).

In this test, I wanted to try out a few configurations that mimic characteristics of applications I’ve worked on: short and long requests, usually IO-bound. A short request is defined as just rendering a simple HTML template. A long request is 300ms. The requests are ramped from 1 request/sec to 1000 requests/sec over 5 minutes.

Response Time Percentiles over Time (OK responses), simple render — 4 instances, 20 threads each, connected directly to the ALB.
Response Time Percentiles over Time (OK responses), simple render — 4 instances, 20 threads each, using Nginx.

As you can see, for the simple render scenario, Nginx and Puma were mostly the same. As load approached 1000 requests/sec, latency started to get worse, but all requests were completed with an OK status.

The 300ms scenario was a little more grim.

Number of responses per second (green OK, red error), 300ms response — 4 instances, 20 threads each, connected directly to the ALB.
Number of responses per second (green OK, red error), 300ms response — 4 instances, 20 threads each, using Nginx.

My theory that Puma will fail fast and give error status to the ALB when reaching capacity was right. The theoretical maximum throughput is 4 instances * 20 threads * (1000ms in 1 second / 300ms) = 266 requests/sec. Puma handles about 200 requests/sec before returning errors; Nginx starts returning error status at around 275 requests/sec, but at that point requests are already queueing and the response time is spiking.

Remember, these results are for this specific use case, and results for a test specific to your use case probably will be different, so it’s always important to do load testing tailored to your environment, especially for performance critical areas.



私は料理が好きです。特にパンを焼くのが好きで、この間初めて Cookpad に公開したのはフォカッチャのレシピでした。フリーランスでシステム開発をしているエンジニアで、ゼロスタートから運用まで持っていくことを得意分野としています。この2つの要素を組み合わせて新しい何かを作るのが趣味ですが、そのことを今まであまりブログに書いたことはありませんでした。今回のアプリを作ったことで、「アプリは家庭料理みたいでもいい(An app can be a home-cooked meal)」という記事のことを思い出して、同様の考え方を共有したいと思いました。







私たちは以前セールで買って、あまり使っていなかった Amazon Fire 7 タブレットを使用して常時表示させていますが、インターネットにアクセスし、画面がある機器なら何でも大丈夫です。古くなったiPadや携帯の再利用などに便利だと思います。ちなみに、Androidの場合は電源につなぎながらスリープしない設定は開発者モードじゃないと設定できないようです。iOSは設定→画面→オートロックで設定できます。