Rails database timeout on first connection

I'm running rails 3.2, passenger 4.0.53, nginx 4.8.2 and postgres 9.3.5. Postgres and the Rails app are running on different servers.

I'm having an issue where the very first time I try and connect it is very very slow, to the point where nginx times out. A refresh of the page however results in a normal response. If I let the app sit for a while with no connections, this problem repeats itself.

There are no errors in the logs however, as nothing really fails since nginx is just simply timing the connection out. Rails continues to wait for the database response.

I'm not sure if this is a rails, postgres, nginx or generic network issue. How would I go about debugging this?

My database config looks like this (tried to add the keepalive settings to resolve it but it doesn't seem to have any effect):

production:
  adapter: postgresql
  encoding: unicode
  pool: 5
  host: IP
  port: 5432
  database: DB
  username: USER
  variables:
        tcp_keepalives_idle: 60
        tcp_keepalives_interval: 60
        tcp_keepalives_count: 100

Thanks for your help.