Need Advice On Hard Starting Problem

I notice that my car has a problem starting up when it is really hot. It does not always happen, kind of a hit and miss situation. I do know that it doesn't start when I hear the fans kick in. I am thinking it may be because it is drawing power from the battery. Due to this problem, I am having a problem trusting my car. It is no longer reliable and am leary about taking it out.

I am wondering if the problem stems from the starter or the battery. The battery is atleast four years old, but reads as good, I dont think it is the battery, but I dont want to remove the starter until I know for sure. The car seems fine 90% of the time, but cant trust it any longer. Would like to know how to test for the problem. Does anyone have any advice or has ever suffered from similar problems? I would appreciate any kind of help that I can get.