Someone may have already mentioned this, but if you arrive around May 10th the Curry Village rafting may not be open. I think it usually depends on the depth and/or speed of the current. Maybe the water temperature too. I have been there several times in May when I wasn't able to take my family rafting and it was a huge disappointment. But maybe you have your own rafts.
As far as nice towns, I've always enjoyed staying in Oakhurst. The people are very friendly there. They have the Von's and Raley's grocery stores in town. More expensive than Fresno, but not by a lot. Because of availabilty, I've probably stayed at most of the hotels there. The Shiloh Inn is good. The Best Western Yosemite Gateway lodge is good. There is the Days Inn / Holiday Inn Express as well. I gave it the slash mark because over the years I think it has been both at least twice. I really miss the Schezuan Chinese restaurant there. Theresa was the owner and they always took good care of our groups - even after a long, hot day of hiking. They had the best Hot n Sour soup on the planet. I begged her for the recipe more than once, but I never did get it. I don't know why they lost the lease (or gave it up), but it was my favorite restaurant in Oakhurst. I've heard the Jade Inn has pretty good food too. El Cid always seems to be crowded, but I've never been there.
There's a pretty decent deli in Yosemite Village. The "Lost Arrow" is a very good and hearty turkey sandwich that fills you up after a long hike.
As far as NOT going to Glacier Point? That would be like going to Paris and skipping the Eiffel Tower, or seeing Athens but not the Acropolis! I'd never do that. It's only 16 miles up Glacier Point road from the cutoff from CA-41. The view of Half Dome from Glacier Point is worth the drive all by itself. Plus, you will get a few of Yosemite Falls which should be wonderful this time of year. Take the time to go to Glacier Point. I wish I could be there on May 10th.
Just saying....!
Have a great time. In Yosemite it's hard not to have a great time.