I would have to know, how far
I would have to know, how far your daily drive is verses what is the heaviest object you tow more than four times a year. Medium towing could mean 3000.lbs up to 5000.lbs. But, that being said, your torque loss would not be noticeable especially when you use your Tow/Haul mode whilst towing said object. It will be more noticeable when just towing in "Drive", especially with the six speed. Your Mpg Will be negligible as say compared to a 3.55 when towing in Tow/Haul since your going to running at a higher Rpm. Better overall Mpg when not towing of course.
So in conclusion I think the setup that your looking will be fine for your needs.