Upgrading Rails 3 to Rails 4 rake tasks "Don't know how to build task"

I'm upgrading a Rails 3 application to Rails 4 and had existing plugins using the old vendor/plugins directory. Where possible I have switched to gems, in other cases I have moved these to lib and created initializers in config/initializers.

Some of these plugins add rake tasks that now are failing with the error "Don't know how to build task 'X' where X is any task that was migrated to the lib directory. Is there some other steps that need to be taken to migrate rake tasks when upgrading to Rails 4?