Using Elixir/Phoenix to poll BART arrival times

I started writing this post nearly 5 years ago, as I was starting to play around with Elixir and Phoenix. I’m publishing this to push me to finish writing the rest of the series.

I’ve recently been exploring Elixir and Phoenix. As an exercise I decided to write an app to poll BART train arrival times. I’ve done this in a couple of languages now, so it was interesting to see how Phoenix, Elixir, and the underlying Erlang VM simplified a number of pieces (e.g. API polling, websockets).

Note: This tutorial assumes a basic familiarity with Elixir and Phoenix and a working environment for the same. I’ll be using Elixir v1.12.2 and Phoenix v1.5.9. If you need help getting your environment set up, the Phoenix Installation Guide is a good place to start.

Part 1: Station Data #

Create a new project #

Let’s go ahead and create a new Phoenix project using the default Postgres database adapter and with support for Phoenix.LiveView (more on that later):

$ mix phx.new bart --live
* creating bart/config/config.exs
* creating bart/config/dev.exs
* creating bart/config/prod.exs
* creating bart/config/prod.secret.exs
[...]
* creating bart/assets/static/images/phoenix.png
* creating bart/assets/static/robots.txt

Fetch and install dependencies? [Yn] Y
* running mix deps.get
* running mix deps.compile
* running cd assets && npm install && node node_modules/webpack/bin/webpack.js --mode development

We are almost there! The following steps are missing:

    $ cd bart

Then configure your database in config/dev.exs and run:

    $ mix ecto.create

Start your Phoenix app with:

    $ mix phx.server

You can also run your app inside IEx (Interactive Elixir) as:

    $ iex -S mix phx.server

Database configuration #

Before we can run the mix ecto.create task mentioned above, we need to create the database user:

$ createuser --createdb --pwprompt bart
[follow password prompts]

We then need to update our database configuration in config/dev.exs with the credentials:

# Configure your database
config :bart, Bart.Repo,
  username: "bart",
  password: "bart",  # or whatever you entered at the prompt above
  database: "bart_dev",
  hostname: "localhost",
  show_sensitive_data_on_connection_error: true,
  pool_size: 10

And now we can actually create our database: $ mix ecto.create

Station schema #

Now that our database has been created, let’s create the schema to represent a BART station.

Here’s an excerpt of the data returned from BART’s Station API:

<?xml version="1.0" encoding="utf-8" ?>
<root>
<uri><![CDATA[ http://api.bart.gov/api/stn.aspx?cmd=stns ]]></uri>
  <stations>
    <station>
      <name>12th St. Oakland City Center</name>
      <abbr>12TH</abbr>
      <gtfs_latitude>37.803664</gtfs_latitude>
      <gtfs_longitude>-122.271604</gtfs_longitude>
      <address>1245 Broadway</address>
      <city>Oakland</city>
      <county>alameda</county>
      <state>CA</state>
      <zipcode>94612</zipcode>
    </station>
    ...

This seems fairly reasonable, so our schema will essentially mirror those same properties:

$ mix phx.gen.context Stations Station stations name:string abbreviation:string latitude:float longitude:float address:string city:string county:string state:string zip_code:string
* creating lib/bart/stations/station.ex
* creating priv/repo/migrations/20210818045243_create_stations.exs
* creating lib/bart/stations.ex
* injecting lib/bart/stations.ex
* creating test/bart/stations_test.exs
* injecting test/bart/stations_test.exs

Remember to update your repository by running migrations:

    $ mix ecto.migrate

One of the things I like about Phoenix is that it doesn’t try to be too smart. Unlike Rails, it doesn’t know how to pluralize words, so you provide both the schema name as well as the table name explicitly (via the second and third arguments to mix phx.gen.context, respectively). This means you completely avoid the mess of ActiveSupport inflection.

We’ll be using the station abbreviation as a unique identifier for each station, so we’ll need to tweak the migration (at priv/repo/migrations/{YYYYMMDDHHMMSS}_create_stations.exs) to ensure that it is, indeed, unique. Open the migration and mark name and abbreviation as non-null and add a unique constraint to abbreviation:

...
def change do
  create table(:stations) do
    add :name, :string, null: false
    add :abbreviation, :string, null: false
    ...
  end

  create unique_index(:stations, [:abbreviation])
end
...

Then run the migration: $ mix ecto.migrate

Let’s also go update lib/bart/stations/station.ex to only require the name and abbreviation (by default, all the properties will be marked as required):

def changeset(station, attrs) do
  station
  |> cast(attrs, [:name, :abbreviation, :latitude, :longitude, :address, :city, :county, :state, :zip_code])
  |> validate_required([:name, :abbreviation])
end

Upserting stations #

Now that we have our schema created, let’s add support for fetching station data from BART and populating our database table.

API Client #

Create a new file at lib/bart/client.ex, and let’s define our BART API client:

defmodule Bart.Client do
end

You’ll notice that the module name reflects its location in the project directory structure — this is common practice but isn’t required. We could have called the module Bart.ApiClient or put it in lib/super_secret.ex — Elixir doesn’t really care. Never again will you have to deal with autoload hell.

The first thing we want to do with this client is fetch the list of stations. The API docs show we can do this via an HTTP GET request to http://api.bart.gov/api/stn.aspx?cmd=stns&json=y&key=MW9S-E7SL-26DU-VV8V.

Note: The API key on the end of the URL is a shared key. You don’t have to register your own, but it is a good idea. Otherwise, your application may be subject to a tragedy of the commons if somebody else abuses that same key.

You’ll notice that one of the query parameters on the URL is json=y. The BART Station Information API can return responses in either XML or JSON format. It’s significantly more straightforward to use the latter, so that’s what we’ll do.

To make the HTTP request we’re going to leverage a third-party library called Mojito. To add this dependency to your app, append it to the list of dependencies in your mix.exs file:

defp deps do
  [
    {:phoenix, "~> 1.5.9"},
    {:phoenix_ecto, "~> 4.1"},
    {:ecto_sql, "~> 3.4"},
    {:postgrex, ">= 0.0.0"},
    ...
    {:jason, "~> 1.0"},
    {:plug_cowboy, "~> 2.0"},
    {:mojito, "~> 0.7.9"}
  ]
end

Run $ mix deps.get to download the new dependency.

Now let’s go back to our client module and add logic to make the HTTP request (the comments are there for clarity):

defmodule Bart.Client do
  # Define a function `fetch_stations` which takes a single parameter, the BART API key
  def fetch_stations(api_key) do
    # Build the full URL
    url = "http://api.bart.gov/api/stn.aspx?cmd=stns&json=y&key=" <> api_key

    # Make the HTTP request
    case Mojito.request(:get, url) do
      {:ok, %Mojito.Response{body: body, status_code: 200}} ->
        # Success!
        IO.puts "We got a successful response: #{body}"
      _ -> 
        # Failure =(
        IO.puts "There was an error retrieving the list of stations."
    end
  end
end

The above snippet illustrates one of the other things I love about Elixir: powerful pattern matching. We switch on the return value of Mojito.request — if it’s a tuple of :ok and an Mojito.Response whose status_code is 200, extract the body into a local variable body for handling. If it’s anything else, we log an error message. You will see this pattern everywhere in Elixir.

Note: In a real application you probably want to match for other possibilities (e.g. other status codes) or at least extract the error for logging/alerting.

To check that everything is working, let’s fire up the Elixir console via $ iex -S mix.

iex(1)> Bart.Client.fetch_stations("MW9S-E7SL-26DU-VV8V")

If everything is working properly, you should see a log message with the JSON response. Congrats!

You’ll notice that we didn’t have to create an instance of Bart.Client to call fetch_stations. As a reminder, Elixir is a functional language; there aren’t classes or instances, just functions.

Parsing JSON #

Now that we have a JSON response we need to parse it into something more useful. To do that we’re going to use the Jason library.

Update Bart.Client.fetch_stations/1 to decode the response body as JSON:

defmodule Bart.Client do
  def fetch_stations(api_key) do
    url = "http://api.bart.gov/api/stn.aspx?cmd=stns&json=y&key=" <> api_key

    case Mojito.request(:get, url) do
      {:ok, %Mojito.Response{body: body, status_code: 200}} ->
        parse_stations(body)
      _ ->
        {:error, "There was an error retrieving the list of stations."}
    end
  end

  defp parse_stations(json) do
    {:ok, stations} = Jason.decode(json)
    {:ok, stations["root"]["stations"]["station"]}
  end
end

Now, when you call that function from IEx, you’ll get the parsed result:

iex(1)> Bart.Client.fetch_stations("MW9S-E7SL-26DU-VV8V")
{:ok,
 [
   %{
     "abbr" => "12TH",
     "address" => "1245 Broadway",
     "city" => "Oakland",
     "county" => "alameda",
     "gtfs_latitude" => "37.803768",
     "gtfs_longitude" => "-122.271450",
     "name" => "12th St. Oakland City Center",
     "state" => "CA",
     "zipcode" => "94612"
   },
   ...

That’s all for now. In the next installment, we’ll save the stations to our database so we can use them later.

 
4
Kudos
 
4
Kudos

Now read this

Refactoring Uber’s Rider app

There was a lot of discussion at the end of 2020 about Uber’s mobile apps, largely due to a Twitter thread by McLaren Stanley. Many wondered aloud why we didn’t just refactor the app instead of rewriting it. I thought I’d add some... Continue →