As long as you have the underbelly cover in place it would be a very very slight chance that something could fly in there and get caught in between the timing belt and crank pulley and make it jump a tooth. Probably never going to happen. It could I guess, but chances seem slim something could get in there.
Other than the obvious risk of sand/fine particles being more likely to get in to the front main seal I doubt you'll have a problem.
I wouldn't do it
but you could probably get away with it.