I agree that app is of no use to you after you ordered your free case. As long as you got your confirmation e-mail, that's all you need.
If you don't want to delete it but would like it not to show on your phone, go to iTunes and uncheck that app. That way you can always restore it.