Find comments with no reply either by vote or comment by current_user

Overview

I have a list of comments where the current_user can either reply with a new comment or vote for the comment. Now i'm trying to create a view to force the user to reply with a new comment or just vote for the comment.

Task

Find comments with no reply (vote or comment) by current_user.

Models

Comment Model

class Comment < ApplicationRecord
  belongs_to :question
  belongs_to :user
  belongs_to :parent,  class_name: 'Comment', optional: true
  has_many :replies, class_name: 'Comment', foreign_key: :parent_id, dependent: :destroy
  has_many :votes, dependent: :destroy
end

Vote Model

class Vote < ApplicationRecord
  belongs_to :user
  belongs_to :comment
end

Question

Is this possible with a decent performance to create an activerecord query for that? Or would it be easier to create a record every time a user creates a comment in a new table like pending_replies or something like that and then delete on creation of reply?

Any hints and starting points would be amazing!