Multiple has_many relationships with same model through two different models, Rails

I have a model in my rails app:

class Image < ActiveRecord::Base
  has_many :users, through: :sites
  has_many :users, through: :suppliers
end

Obviously this code won't work as the second line will just override the first, but I'm illustrating what I'm trying to achieve.

The other classes:

class User < ActiveRecord::Base
  has_and_belongs_to_many :sites, -> { uniq }
end

class Site < ActiveRecord::Base
  has_and_belongs_to_many :users
end

class Supplier < ActiveRecord::Base
  has_and_belongs_to_many :users
  has_and_belongs_to_many :images
end

A user should own images that they've uploaded through sites and suppliers that they own.

Is there another way of writing this or do I need to reconfigure the existing set up. Any help appreciated, let me know if you need more info.