PostgreSQL ActiveRecord Query Optimization

I'm new to RoR and I need a query that returns the same instances as the following definition of queue :

@all = User.all
@membersOfActivity = User.where(id: User.joins(:memberships).where("activity_id = ?", session[:current_activity_id])) 
@usersIAlreadyLiked = User.where(id: User.joins(:likes).where("activity_id = ? AND activity_likes_user IS NOT NULL", session[:current_activity_id])) 
@notWanted = @membersOfActivity + @usersIAlreadyLiked 
@queue = @all - @notWanted

However, I understand how inefficient the query I just wrote is for it downloads all the Users first to then select the complement. Do you have any idea on how to make it more efficient and directly select the complement of @notWanted? I tried several queries but none of those worked. If you think you have a better solutions please let me know! Thanks!