ActiveRecord - List all parents DO NOT have one specific child

I have a many-to-many relationship between users and badges.

class Badge < ActiveRecord::Base
    has_and_belongs_to_many :users
end

class User < ActiveRecord::Base
    has_and_belongs_to_many :badges
end

A user can have many badges.

How can I efficiently list all the users who DO NOT have a specific badge id using ActiveRecord?

I have a lot of users and badges.