Yes it is the default app. I have notifications on and push selected, I have mine buzz when I get mail but you can set any audio alert and set visible too.
You can, just tap and hold the app until it starts wiggling and the X appears in the upper left corner. Tap the X and confirm the deletion in the pop up that appears.