How can I make sure actionmailer & mail gems are only loaded when needed?

I'm trying to reduce my rails 3.2.22 app's memory usage and read that the mail gem eats memory like it's a buffet (http://ift.tt/1ATwCQT). This page says to upgrade mail gem to 2.6.3, but actionmailer 3.2.22 seems to be incompatible with that version of mail gem.

I only send emails from a few spots in the app and I'm fine moving the actionmailer and mail gems to a specific Gemfile group, but that didn't see to work. I'd prefer not to upgrade to rails 4 yet since that's a much larger task.

What options do I have?

Thanks!