You might lose some power. It has been shown at least on some racing engines that light weight pulley can interfere with the valve train in fact a heavy damper can sometimes make more power. Take the mass out of the flywheel since it is damped by the clutch or lighten the internals.