Categories
Elixir English

Elixir’s StringIO may not be what you think it is

In Ruby, there is a very handy class called StringIO. Basically, it allows you to treat a string like you would an IO object, such as an open file, etc. Very useful for in-memory “files” that you may not want to write to a temporary file.

In Elixir, there is a module called StringIO in the standard library. At first glance, these seem pretty similar:

Ruby:

Pseudo I/O on String object.

Elixir:

This module provides an IO device that wraps a string.

However, there are some subtle differences. For example, you can’t rewind the position of an Elixir StringIO:

iex(1)> {:ok, io} = StringIO.open("foo")
{:ok, #PID<0.59.0>}
iex(2)> IO.read(io, :all)
"foo"
iex(3)> :file.position(io, :bof)
===> hang!

(For more information about the :file.position/2 function, check the docs out. :bof stands for “beginning of file”)

Let’s see why this is happening. StringIO has a function to show its current buffers:

iex(1)> {:ok, io} = StringIO.open("foo")
{:ok, #PID<0.59.0>}
iex(2)> StringIO.contents(io)
{"foo", ""}
iex(3)> IO.read(io, :all)
"foo"
iex(4)> StringIO.contents(io)
{"", ""}

Basically, after you read data from an Elixir StringIO, it’s gone. So, we look to Erlang. Erlang’s file:open/2 function accepts a ram option that we might be interested in:

Returns an fd() which lets the file module operate on the data in-memory as if it is a file.

Let’s try it out.

iex(1)> {:ok, io} = :file.open("foo", [:ram, :binary])
{:ok, {:file_descriptor, :ram_file, #Port<0.1471>}}
iex(2)> IO.binread(io, :all)
"foo"
iex(3)> IO.binread(io, :all)
""
iex(4)> :file.position(io, :bof)
{:ok, 0}
iex(5)> IO.binread(io, :all)
"foo"

Rewinding works now.

Note that because of differences between Elixir’s IO / File modules and Erlang’s file module (probably related to how Elixir works with character encodings), you have to use the binary option to :file.open/2 and the bin-prefixed functions in Elixir land.

Categories
English Tools

Web Development Tools: Mailtrap

This is the inaugural article of my “Web Development Tools” series I plan on continuing for at least a few more posts, sharing some of the essential tools I use for web development every day. When I have the chance to work with new people, we always exchange useful information about the tools and libraries we use. This series is an attempt to organize this information.

Today’s pick: Mailtrap

Once upon a time, I was working on a 2.0 for a client. Major overhaul. The database schema was completely different. So, I wrote a data migration script. Runs fine locally. Then, it became time to import test data to the staging environment.

I had completely forgotten that when creating a new user, a “Welcome to our service, please confirm your account!” e-mail was sent out.

You can imagine what happened next. Thousands of e-mails were being sent while the migration script was running, sending “Welcome!” e-mails to unsuspecting customers — with a link to the staging environment.

This continues to be one of the most embarrassing moments of my career to date. Following this event, I sought out to find something that would make sure this never happened again. I found Mailtrap.

Mailtrap is an SMTP server — an SMTP server that doesn’t forward messages to users. Instead, they’re saved to the Mailtrap mailbox, accessible via a nice web GUI.

Incoming Mailbox
Incoming Mailbox

From there, Mailtrap gives you some nice tools — inspecting the text content if you have a multipart e-mail, viewing HTML source, seeing the raw e-mail.

Raw E-mail View -- headers and all.
Raw E-mail View — headers and all.

When I started using Mailtrap, it just had the most basic feature — catching e-mail. Now, it has a lot of very useful features — shared mailboxes so you can share test e-mails with your team, forwarding so you can forward test e-mails to a real client to see how they look — the list goes on.

Today, I use Mailtrap by default in all of my projects that send any kind of e-mail. It’s very simple to set up — they have instructions for the popular platforms and frameworks (including Sendmail, heh!) — and if your framework isn’t in there, they have the plain old SMTP settings for you to plug in.

Mailtrap is free for one mailbox with up to 50 messages, and plans start from $9.99/monthly. Once the 50 message limit is reached, older messages will be deleted to make room for the new messages. For personal projects, the free tier has been more than enough for my needs.

Disclosure: I am not affiliated with Mailtrap, nor am I receiving any compensation (financial or otherwise) from Mailtrap for writing this article.

Mailtrap is a trademark of Railsware Products, Inc..

Categories
English

Brutal Simplicity

My favorite pizza is the pizza Margherita. Any pizzeria I go to, I will order the Margherita first.

Why? It’s brutally simple. Four ingredients: dough, tomato sauce, mozzarella cheese, basil.

I wish more websites and applications would be like a good Margherita.

Simple.

Delicious.

????

Categories
English

Homebrew and PostgreSQL 9.5 (or 9.6)

Edit Sept. 30 2016: PostgreSQL 9.6 was released today, and these instructions should work — just replace 9.4 with 9.5 and 9.5 with 9.6. I also have a guide using pg_upgradecluster on Ubuntu.

PostgreSQL 9.5 was released on Jan. 7, with lots of exciting new features.

I wrote a post about upgrading from 9.3 to 9.4 in the past, and many people found it useful, so I decided to update it a bit for the 9.4 to 9.5 upgrade.

  1. Turn PostgreSQL off first:
    $ launchctl unload ~/Library/LaunchAgents/homebrew.mxcl.postgresql.plist
    # or, if you're running a current version of Homebrew
    $ brew services stop postgresql
    
  2. Update PostgreSQL itself:
    $ brew update && brew upgrade postgresql
    
  3. Make a new, pristine 9.5 database:
    $ initdb /usr/local/var/postgres9.5 -E utf8
    
  4. Migrate the data to the new 9.5 database. Note that I have 9.4.5_2 in here, it could be that you aren’t on the latest version. Replace 9.4.5_2 with the most current version of postgres in that directory.
    $ pg_upgrade \
      -d /usr/local/var/postgres \
      -D /usr/local/var/postgres9.5 \
      -b /usr/local/Cellar/postgresql/9.4.5_2/bin/ \
      -B /usr/local/Cellar/postgresql/9.5.0/bin/ \
      -v
    
  5. Move 9.5 data directory back to where PostgreSQL expects it to be:
    $ mv /usr/local/var/postgres /usr/local/var/postgres9.4
    $ mv /usr/local/var/postgres9.5 /usr/local/var/postgres
    
  6. Start PostgreSQL back up!
    $ launchctl load ~/Library/LaunchAgents/homebrew.mxcl.postgresql.plist
    # or, if you're running a current version of Homebrew
    $ brew services start postgresql
    

Note: If you’re using the pg gem for Rails, you should recompile:

$ gem uninstall pg
$ gem install pg

Note 2: If you’ve already uninstalled a previous version of PostgreSQL, there is a good post on StackOverflow with instructions to install previous versions.

Categories
Elixir English

Elixir Pattern Matching in Anonymous Funs

filter_zed_by = "1"
list = [
  %{"a" => "1", "b" => "2"},
  %{"a" => "1", "b" => "5"},
  %{"a" => "2", "b" => "5"},
  %{"z" => "1", "x" => "2"}
]
Enum.filter list, fn
  %{"z" => ^filter_zed_by} -> true
  _ -> false
end

# => [%{"z" => "1", "x" => "2"}]
  • case in a fun is usually redundant
  • if is even worse
  • keep it simple
Categories
AWS English

Hosting a Single-Page App on S3, with proper URLs

Note (2019/07/05): I’ve posted a follow-up to this post about limitations about the technique used here, especially when hosting an API on the same domain.

Amazon S3 is a great place to store static files. You might want to even serve a single-page application (SPA) written in JavaScript there.

When you’re writing a single-page app, there are a couple ways to handle URLs:

A) http://example.com/#!/path/of/resource
B) http://example.com/path/of/resource

A is easy to serve from S3. The server only sees the http://example.com/ part, and so it serves that file to everyone.

B, however, is a little tricky. Single-page apps usually use pushState or replaceState to change the current URL without reloading, but once you reload (or give the URL to someone else) — BAM! You’ll get presented with a 404 Not Found error.

So why don’t we just use A? There are quite a few advantages to using B, over just being more elegant than putting that pesky #! in there. In my opinion, the biggest advantage of using B is that you’ll be able to make backend changes in the future without having to redirect URLs. For example, as your app gets bigger, you want to render some (or all) components server-side (see Isomorphic or Universal JavaScript).

To implement the B strategy, we need to serve the same index.html file to any URL requested by the client. As I mentioned earlier, we can’t do this with S3 itself, so we’ll enlist the help of CloudFront.

First, create a CloudFront distribution for the S3 bucket. Since CloudFront caches items for quite a long time, you might want to either set Cache-Control headers on your S3 files, or set the default TTL to something short, like a few seconds, in the CloudFront distribution settings. Once everything is set up (and you can access index.html by itself), click the “Error Pages” tab.

Screen Shot 2015-11-24 at 9.28.46 AM

Click the big blue button, “Create Custom Error Response”:

Screen Shot 2015-11-24 at 9.28.55 AM

Now, I think you can tell what I’m up to now. Enabling “Customize Error Response” allows you to change a 404 from the backend (in this case, S3) in to a 200! Note that S3 will return a 403 response if you use the “S3 Origin” option instead of the S3-hosted origin. If you’re getting a 403 error from S3, customize the 403 error as well.

Screen Shot 2015-11-24 at 9.29.23 AM

You can try out this setup below:

https://d3qxx6yxxvp94v.cloudfront.net/https://d3qxx6yxxvp94v.cloudfront.net/testhttps://d3qxx6yxxvp94v.cloudfront.net/l87v3

These all serve the same index.html. If you inspect the headers, the first link should be X-Cache: RefreshHit from cloudfront or Miss from cloudfront. However, if you look at the other requests, it will be X-Cache: Error from cloudfront. The status returned, however, is 200 — just as we wanted it.

Any questions? Contact me or leave a comment in the box below.

Categories
English Uncategorized

Podcasts I’m Listening To (November 2015 Edition)

My wife Naoko wrote a reply to this post. It was fun comparing how different the podcasts we listen to are. 🙂

First, I’d like to plug a podcast that I’m a semi-regular guest on, techsTalking(5417), a podcast where technology people just talk about whatever is on our mind.

Here are some other podcasts that I’m currently subscribed to:

  • The Incomparable — a podcast about anything geeky. Star Wars? Check. Star Trek? Check. Silly drafts? Check. Crazy movies? Check.
  • The Incomparable Game Show — born from The Incomparable proper, regular panelists play crazy games for your entertainment. On the podcast.
  • Incomparable Radio Theater — The Incomparable podcast, once upon a time, liked to do funny things on April Fools. Like, say: release a full-length episode in the format of old-time radio drama. Including equally funny sponsors (some fake, some real). Now, they’ve spun it off in to a separate podcast.
  • Random Trek — Incomparable regular Scott McNulty hosts a podcast with non-random guests talking about random episodes of Star Trek.
  • Robot or Not? — Is it a robot? Or not?
  • Astronomy Cast — A weekly “facts-based journey through the cosmos”.
  • Reconcilable Differences — Two of my favorite podcasters, John Siracusa and Merlin Mann, get together on one podcast.

A few other podcasts I listen to occasionally:

And assorted programming-specific podcasts.

Categories
English Open Source Projects

Runroller UI

I recently released a simple API to un-shorten URLs. A few people wanted a super-simple interface to this, so I whipped one up: https://keita.blog/unroll/. Enjoy!

Some notes about the tools I’ve used:

  • React — I’ve used React in portions of sites before, but this is the first, albeit simple, full-page React app I’ve made.
  • Brunch — used by default in Phoenix apps, it’s just what I’m used to these days.

Just like the service that runs the API, the UI is also open-source. Hack away!

Categories
Elixir English Open Source Projects

Link Unroller Service

As a small side project, I recently launched a “link unroller” service. This is a very simple service. You give it a URI, and it follows any redirect chain for you. Then it spits out the final URI via a friendly JSON API.

Give it a spin:

https://unroll.kbys.me/unroll?uri=http://bit.ly/1QZ6acT

Basically, all you do is send a GET request to:

https://unroll.kbys.me/unroll?uri=<URI to unroll>

Done. If there are no problems, you will get a JSON response:

{
  "uri":"http://bit.ly/1QZ6acT",
  "unrolled_uri":"https://keita.blog/",
  "redirect_path":[
    "http://bit.ly/1QZ6acT",
    "http://keita.blog/"
  ],
  "error":false
}

The unrolled_uri parameter is the final link in the chain, and the redirect_path is an array of the links that were traversed.

If you’d like to take a look at the code, make some contributions, or submit some bugs, please head over to the GitHub page.

Technical details:

  • The server is in Tokyo.
  • Written in Elixir.
  • Backend responses are ~ 600 microseconds on a cache hit.

Policy details:

  • Up to 7 redirects will be followed.
  • The request will time out after 20 seconds and return an error.
  • 301 redirects are cached forever, regardless of Cache-Control or Expires headers present in the response.
  • 302 redirects will honor caching headers, with a minimum TTL of 1 minute (this is for DoS protection on my side)
  • 200 responses are cached for 1 hour.
Categories
English

bundler gotcha

So, this is a thing:

bundle install --without development:test
...
...
Bundle complete! XX Gemfile dependencies, XX gems now installed.
Gems in the groups development and test were not installed.

Now,

bundle install
...
...
Bundle complete! XX Gemfile dependencies, XX gems now installed.
Gems in the groups development and test were not installed.

Basically — you run bundle install --without <group> once, and that’s saved in .bundle/config. So next time you run bundle install without any arguments, it won’t install gems in the groups you specify.

It looks like it is fixed in Bundler 2.0, though.