I have them pretty much all removed.
Over the years, as different work was done, I've just removed them when handy. The big ones above the exhaust were actually removed by a Subaru specialist shop when they replaced the driveshaft since they were somewhat in the way (they had been mangled somehow). They said it was no big deal, maybe just a little more heat in the cabin.
Which is true, but it's only noticeable if you actually reach down and touch the carpet. So far so good as far as it affecting the transmission or anything, it's been about 3 problem-free years.
I agree with Lee though, I wouldn't go out of my way to remove those otherwise.