My controllers tend to come in two flavors:
Type 1 is providing an external api to a database and is interacting directly with an Active Record object. Type 2 will be making calls to a Plain Old Ruby Object or maybe my own interface defined in a model. In either case, there is absolutely no testable logic left in a controller, therefore it is not under unit test.
I arrived at this solution not because I was drawing colored rectangles on a whiteboard and reveling in Architecture. No, it's because testing controllers is a pain in my ass and I would rather not do it. Extracting every last drop of logic out of them just happens to be the path of least pain.