ActiveRecord::InvalidForeignKey in Admin::UsersController#destroy

In ActiveAdmin, when I want to delete a user I have the following error :

ActiveRecord::InvalidForeignKey in Admin::UsersController#destroy 
PG::ForeignKeyViolation: ERROR: update or delete on table "users" violates foreign key constraint "fk_rails_b080fb4855" on table "notifications" DETAIL: Key (id)=(15) is still referenced from table "notifications". : DELETE FROM "users" WHERE "users"."id" = $1

I already have has_many :notifications, dependent: :destroy in my User model so I don't understand the error.

In my Notification model I have :

 belongs_to :user
 belongs_to :contact, :class_name => "User", :foreign_key => "contact_id" 

Maybe the has_many :notifications, dependent: :destroy only affect the user and not the contact? If so, how do i fix it?