superclass must be a Class in rspec after ruby upgrade

I have a class that looks like this:

module API
  module MyNamespace
    class Base < ActiveRecord::Base
    end
  end
end

module API
  module MyNamespace
    class MyClass < Base
    end
  end
end

When trying to upgrade from ruby 1.9.3 to 2.1.5, I started getting an error when executing the entire spec suite.

superclass must be a Class (Module given)

This is strange, because API::MyNamespace::Base is a class. Also if I run the tests as part of a smaller run (individual spec, or everything in the API namespace), then everything is fine.

I'm not really sure what to try here.