Paul,
You are not over a barrel unless you gave the shop instructions which they can be confirmed by a witness, to overhaul the transmission regardless of cost.
At this point they can only claim the cost for removal and whatever strip down they have done. They can only put a lien on the car to the extent of this figure and which you are entitled to dispute.
