Yup, same place.
In the past there was only one YARTS bus in each direction per day.
The bus to Mammoth arrived the Mammoth Mountain Inn around 9 pm in the evening.
The bus to Yosemite left the Mammoth Mountain Inn at 8 am in the morning.
Looks like the same this year, but
the bus has added a couple of runs where the principle remains the same,
arrives Mammoth in the evening,
leaves Mammoth in the morning.
So the obvious solution is to take the evening bus from Yosemite at the beginning of your trip, arrive in Mammoth, and stay at the Mammoth Mountain Inn.
Or arrive in Mammoth in the evening at the end of the hike, stay at Mammoth Mountain Inn and take the morning bus to Yosemite.
That was my plan when I last hiked out to Agnew Meadows in 2008. Meeting a few other backpackers on the bus we all decided we much rather would go in to Mammoth to hang out, eat and drink for the evening. We discovered that there is a bus that shuttles mountain bike riders up to Mammoth Mountain Inn, and returns them to Mammoth. The return ride to Mammoth was free even if you didn't pay to go up.
Then when we arrived in the town of Mammoth there was everything a backpacker wants after a hike in terms of food, drink and entertainment. We enjoyed. Additionally there was a free bus shuttle that transported us around town. At the end of this revelry we all ended up spending the night at the campground, also accessible by the free shuttle bus. The morning after we boarded the morning 8:30 am YARTS bus outside the Shilo Inn.
YMMV as to the details but as far as public transportation goes, Mammoth rocks!