Yeah it definitely is a pain but unfortunately thats the only way you can clear apps out of the background if youre not jailbroken.
Which since we're on this topic, your headline for the topic made me think of a cool idea. If jailbroken, there are a couple tweaks you can download as a sort of "one action" or "clear all app icon" or theres even a way to disable it all together. But what i would like to see is a tweak where they add a ui button in the task switcher next to the rotation lock and ipod controls to clear the apps. That would be cool to see a button there that does that and looks like its part of ios4. Im not advanced enough to create something like that so if somebody stole this idea then it would be okay with me. Sorry for the rambling.