I found this quote by Elder Ballard, and it really struck me:
"What will you say to the Lord when He asks you what you have done with the precious gift of life and time? Surely you will not feel comfortable telling Him that you were able to pass the 100,000-point level in a challenging video game." (http://www.lds.org/ldsorg/v/index.jsp?hideNav=1&locale=0&sourceId=fda866dfc311c010VgnVCM1000004d82620a____&vgnextoid=2354fccf2b7db010VgnVCM1000004d82620aRCRD )
(Actually, that whole article is well worth reading).
I have wondered if there could be a parallel question the Lord might ask, like "What have you done with other gifts I've given you, like your talents, money and resources?" I think I'd be ashamed if I were to say something like, "Oh, I got a nose job" or "Three houses. I bought three houses and seven cars." I just don't think I could look the Savior of the World in the face and say that.
So this has been a mild change of view for me. Not that I was doing really horrible, time-wasting things. But now I find myself asking when I start an activity, "Would I be ashamed to tell the Lord I spent my time/talents/resources on this?" I am relieved to find that often, the answer is no.
Obviously, this has to be approached with reason. Does the Lord expect us never to sing and dance? No. Does he expect us never to watch TV, never read novels, never touch a computer game? No.
Read all of Elder Ballard's talk and you'll see that isn't the case. We're supposed to enjoy our lives. We just aren't supposed to WASTE them. Relaxation is not a waste. Having fun is not a waste. Even the prophet enjoys recreational reading and watching shows (musicals are one thing he specifically mentioned).
There just needs to be balance.
But it is very worth it to stop ocassionally and ask, "Would I be ashamed to report I'd done this?"