For all its reputation as a forward-looking, optimistic series about the future of humanity, why is it that Star Trek is consistently best when it goes into truly dark places?
In all seriousness, this seems to be an issue: the best of the original Trek movies is generally agreed to have been Star Trek: The Wrath of Khan, which was a bloody revenge story, a recapitulation of Moby Dick in space.
The consistently-chosen fan favorite episode of the Original Series is “The City On The Edge Of Forever,” which affirms that sometimes war is the only way to solve a problem, that addiction to peace at any price is dangerous, and that doing the right thing may involve accepting the death of what you love.
This doesn’t change in the Next Generation, either: “Yesterday’s Enterprise” is the fan favorite there, a trip into an alternate timeline in which the Klingon Empire is slowly destroying the Federation, and the Enterprise-D is destroyed saving the Enterprise-C and the original timeline.*
ETA: An Alert Reader pointed out that I had these backward. And no, I don’t, but I was very unclear. So, to explicate and thus restore my lost nerd-cred: The Enterprise-D, in the War Timeline, sacrifices itself to allow the Enterprise-C to return through the temporal discontinuity so that it (the Enterprise-C) can sacrifice ITSELF to save the original timeline.
Gosh, I wonder why Star Trek didn’t do more time-travel episodes.
None of these stories are without hope, of course, but they are consistently darker than Roddenberry’s vision, and certainly in opposition to his (much-derided) dream for first-season Next Generation of a future in which human interpersonal conflicts had pretty much been transcended.
My own feeling is that Roddenberry’s vision simply took too little account for what people demand in a good story, and far from inspiring people, ended up looking rather insipid, while what the fans wanted were stories in which our heroes laid it all on the line, sacrificing all that they were or wanted in order to save what really mattered. In the end, you cannot “transcend” these things. They are themselves transcendence.
G. Scott Huggins 
I think this is exactly why DS-9 resonated with a lot of people.
Here’s what you’re right about: Gene Roddenberry was a secular humanist who didn’t really understand what people wanted out of a good story. But for each of the episodes/movies you’ve referenced, I feel like you’ve left out something pretty important, and it’s a key motive they all have in common. Let’s talk about Kierkegaard (kinda)!
In all three examples, characters (Kirk, Spock, and in TNG, the captain of the Enterprise-C Rachel Garret (and alt-Yar, I guess) face and almost-teleological moment, where they have to make not so much a leap of faith, but a leap toward faith. Kirk loves Edith Keeler, but has to choose to let her die because he believes that’s the only way to save The Universe. In Yesterday’s Enterprise, Picard has this choice as well, albeit indirectly, whereas Garret (and Yar, I guess) face it directly: going back means a 100% chance of death and the possibility of saving The Universe. In both cases, they KNOW they will endure a great loss (of love or life, respectively) and do not know for certain that it’s the right course; only their faith guides them. Maybe not so much as Abraham (after all, neither Kirk nor Picard has ever been shown praying!), but especially for Kirk in City On the Edge of Forever, remember when Joan Crawford got iced by the truck? Kirk’s “fear and trembling” was literal: he was needing the real contact with his friend to ground him in his choice as much as he was needing to hold Bones back from saving her. Say what you want about Shatnerian Acting: he nails that scene for heart-breaking, gut-wrenching effect. And this was no green-skinnned space floozy, but a real, wonderful woman who was a match for our captain in every way. As for Yesterday’s Enterprise, Garrett admits she’s scared as hell about going back, but we don’t see her literal “fear and trembling” before a few Klingon warships show up and some exploding bits of the bridge lodge themselves in her grill, but still, both she and Picard are seasoned space captains in a pitched space battle: one wouldn’t expect to see too much, if any, “fear or trembling” from them in that situation.
(Before I continue, Yar: I can’t quite go the same way, because I was never sure if she was going back for duty or because she fell in love (or something) with Lt. Handsome Squarejaw or because she thought she was supposed to because Guinan told her she should… and it’s not really germane to the point, and also, it let them creat an intriguing villain with Cmdr. Sela who they then seemed to forget about for the rest of Trek, save the first Worf two-parter and the Spock two-parter, and here’s you B5 reference: I’d have to check, but is it possible that Andreas Katsulas as Tomalak got more episodes and screen time than Crosby did as Sela? Anyway…)
As to WoK: sure, it’s about revenge, but that’s a only a side-dish (best served cold, according to the old Klingon proverb) for this feast. The main themes are so much more about age and youth and renewal and family and sacrifice… and love. By the climactic ending, Kirk has been (or will be) renewed by the rekindling of his relationship with Carol Marcus, the discovery of a son he can be proud of, and his pride in the skill and dedication of a crew of trainees (all giving him the satisfaction of a life well-spent, my family will go on, my work will go on), and his age and wisdom (and guile) have beaten Khan. Not just beaten him, but broken his plans, his followers, and his body. But Khan’s rage is great, and in his defeat, his rage would see the world burn if he thought he could catch Kirk in the flames. Kirk is on the bridge, and while the tension is palpable, Kirk is stoic in his resolve: death is inevitable, but he’s still asking for time and distance readouts as the Enterprise limps away.
And so, Spock’s teleological moment: he has no time to discuss the danger with McCoy, so nerve-pinch, “Remember,” gloves on and in we go, into the reactor room and certain death. He knows he’s going to die, but he believes that by doing so, he’ll save his friends, and by extension, The Universe. A strong argument could be made that logical Spock was simply applying his favorite axiom, and since inaction meant certain death for all, there was no teleological stake. However, it’s been pointed out many times that Spock was the most conflicted, and in Kirk’s own words, most human character on the ship. Khan’s rage could only be negated by love (narratively-speaking), and who better to show that human love than Spock?
Does Trek go dark? Only to show us the light within ourselves.
Well, that was indeed the stirring reply, and well said. I couldn’t agree more. Trek’s darkness is not the darkness of despair, but the darkness that is accepted willingly for the cause of the light. But the consequences of that darkness are real, not to be shrugged off as the death of random redshirts. WoK almost parodizes this with one of Kirk’s best lines on the series: “No, I cheated death. Tricked my way out of death, and then patted myself on the back for my own cleverness. I know nothing.” He has to turn to his son to show him that his words about dealing with death are the ones he now must really believe in to show him the light in the darkness. And the light is real, but it does not, as Kierkegaard and all believers must know, fully ease the pain we find in that darkness.
At the risk of taking it too far, does this mark Kirk’s transition to a Knight of Faith vs. a Knight of Infinite? If you don’t count the odd-numbered movies (and we don’t!), does that mean his journey back to save the whales (and Earth, consequently) in 4 and his willingness to risk incarceration, execution, and destruction to save the peace conference (and The Universe, I suppose) in 6 are the steps he’s taking toward accepting who he is, resigning his prejudices and preconceptions to his better nature and embracing his destiny?
I mean, Shatner’s good. He’s great. But he’s not THAT good…
Also, the Kirk-quote from WoK is “…patted myself on the back for my own ingenuity.” #nerdcred