Ruby HATBM - Negative Filtering behaving strange

Lets imaging looking at a ruby schema with two resources (user & books) which have an HABTM association.

When I am filtering based on this association, whether a user has a book or not, ruby behaves a bit strange:

2.5.3 :007 > Contact.includes(:books).count
(4.7ms)  SELECT COUNT(*) FROM "contacts"
=> 36880 

Which is fine, because it returns the total value of contacts with any book association.

2.5.3 :010 > Contact.includes(:books).where(books: {"id":[1,2] }).count
   (34.0ms)  SELECT COUNT(DISTINCT "contacts"."id") FROM "contacts" LEFT OUTER JOIN "contacts_books" ON "contacts_books"."contact_id" = "contacts"."id" LEFT OUTER JOIN "contacts" ON "contacts"."id" = "contacts_books"."book_id" WHERE "books"."id" IN (?, ?)  [["id", 1], ["id", 2]]
 => 24864 

Which is also fine, since it returns all contacts which have book with id 1 or 2.

2.5.3 :011 > Contact.includes(:books).where.not(books: {"id":[1,2] }).count
   (13.0ms)  SELECT COUNT(DISTINCT "contacts"."id") FROM "contacts" LEFT OUTER JOIN "contacts_books" ON "contacts_books"."firm_id" = "contacts"."id" LEFT OUTER JOIN "contacts" ON "books"."id" = "contacts_books"."book_id" WHERE "books"."id" NOT IN (?, ?)  [["id", 1], ["id", 2]]
 => 0 

Now this part doesnt make sense, because it should be total - #associations, so 36880 - 24864 leaving me with 12016 records and not 0.

Or what am I missing?