leaking oil???

I just found out that the person I bought my car from had the oil changed before I bought it and whoever changed it put way too much oil in and now their is oil pooring out the seals. If I let some oil out to the normal level will it stop leaking or at least slow down, or is it too late to reverse the damage? I dont have a grand to spend on getting all the seals replaced and so the car will end up sitting for a year or two (my neon is my main focus)