New rotors/calipers, but now there's this sound....

[SubaSteevo]

It sounds like metal grinding....because it is metal grinding. The lower part of the calipers in the rear seam to be grinding the rotors right where the clips for the pads are, but it is definately the caliper piece and not the clips that is doing the grinding. If I push on the rotors and turn the axle, there's no sound, but if it is slightly off it starts grinding. Just torquing the wheels does not seem to hold the rotors steady enough to avoid the grinding. Is there a trick to it? Should I try to widen the gap with a dremel so the rotor has more room to travel?