It's not so dumb a question Marcus.
Quite a few in the US have thought of this, but there are obstacles, mostly of the electronic kind.
First you need to remember and realise that the highest numbers of SVXes sold in all markets were sold during the first year or two years of coming on sale. For most markets this means 92 and 93 cars. All these cars had the early unmodified gearboxes and are most likely to give problems. They can cause problems whether they are the US type ACT4 or the Japanese type VTD gearbox.
So effectively the most commonly available Japanese gearbox could actually be the early unmodified VTD from scrapped early SVXes, and the US owners could be buying a better type of gearbox granted, but which was still prone to the early overheating related failures.
You are lucky in that your S4 is the last of the line and your gearbox would probably be as reliable as it gets. For an SVX.
The obstacles I mentioned above would be wiring changes. The Japanese gearbox fitted would need a Japanese TCU of the same year as the gearbox fitted to the car to control the VTD. The US type TCU would not do the job. In addition to this there is another complication in that Japanese and US type cars have one of the two speed sensors in the front diff and it's a different type sensor than the Aus and UK sensor which is inside the gearbox. The TCU used to control the transmission would need the same type of speed sensor [SS2] fitted so that it can process the relative speed differences between front and rear axles so it can apportion torque and avoid slippage. And of course read speed for the speedo in the same units [Kilometres Japan, Miles USA] as the market where the VTD gearbox came from.
So the answer to your question would be that like everything in life, it's possible.
It's not exactly a DIY job though.
Joe