Ruby on Rails: Left Outer Join with Sort

I have a RoR application for sending emails, I have the models Contacts, Recipients, and Users, whereby a user can have many contacts and a contact can have many recipients. The data looks like this:

 User
 ID     Name
 1      Ben
 2      Emily
 3      Brian

 Contact
 ID       Name     User_ID
 1        Jack     1
 2        Joe      2
 3        Chloe    2
 4        Jacob    1
 5        Trevor   3

 Recipient
 ID     Contact_ID     Email_ID (not relevant for this question)
 1      1              2
 2      1              1
 3      4              5
 4      1              6
 5      2              8

What I want to do is on the index page for contacts, display all the contacts belonging to a user allow the user to choose what to order them by, such as name and the amount of emails received.

The problem I have is that I can display all the contacts belonging to a user no problem, but I am struggling to allow a user to sort the contacts by the amount of emails they have received. The amount of emails they have received can be calculated by counting the amount of recipients with the same contact id. The reason I have this problem is because I want to display all the contacts belonging to a user irrespective of whether they have received emails (a contact may have received 0 emails). The code I have tried is:

@contacts = Contact.where(user_id: session[:user_id]).all.order(params.fetch(:sort, 'id asc'))

But this doesn't allow a user to sort by the amount of emails the contacts have received.

I have also tried this:

  @contacts = Contact.where(user_id: session[:user_id]).joins(:recipients).group('contacts.id').all.order(params.fetch(:sort, 'id asc'))

But this only displays the contacts who have received emails.

And I have also tried this:

@contacts = Contact.where(user_id: session[:user_id]).joins("LEFT OUTER JOIN contacts ON contacts.id = recipients.contact_id").all.order('max(recipients.contact_id) DESC')

This displays all contacts who have and have not received emails and allows sorting, but it shows duplicate data. I.e. using the above data, if Ben were to look at his contacts (user_id 1) he could see Jack and Jacob, but because Jack has received 3 emails he would see the following:

First Name:
Jack
Jacob
Jack
Jack
Jack

Is there a way to solve this? Please note that I am currently using the following code on index.html.erb to allow a user to select what they want to order the records by:

<form>
    Order By:
    <select name="sort">
        <option value="firstname ASC">First Name (A - Z)</option>
        <option value="firstname DESC">First Name (Z - A)</option>
        <option value="surname ASC">Surname (A - Z)</option>
        <option value="surname DESC">Surname (Z - A)</option>
        <option value="email ASC">Email (A - Z)</option>
        <option value="email DESC">Email (Z - A)</option>
        <option value="max(recipients.contact_id) ASC">Emails Recieved (Low - High)</option>
        <option value="max(recipients.contact_id) DESC">Emails Recieved (High - Low)</option>

        <option value="updated_at ASC">Date Updated (Oldest First)</option>
        <option value="updated_at DESC">Date Updated (Newest First)</option>
    </select>
    <button>
        Go
    </button>
</form>