Hipster hiker here, finally decided to hit the touristy Grays and Torreys at the tail end of the season. Maybe just as bad during other parts of the season, as it was Labor Day weekend. Pro-tip: continue South on Stevens Gulch road, past the trailhead footbridge for more parking in the trees.
Missed the hot left to go up Grays first. I think it worked out better doing Torreys first; the North slope of Grays was icy in places where the sun seems to miss. I was able to jog down the easy East slope switch-backs of Grays protected from the wind. That wind on the saddle is brutal ~9am.