I don't have an exact answer for you off the top of my head but i'm leaning toward your turn signal switch to some extent. I don't know if they go bad or not, maybe someone an chime in on that, but here's what I'm thinking...
You said your stop lamp/haz fuse is good so starting from there, there's an orange wire that runs from the fuse to feed the trunk lamp, if equipped and yours is, and the dome light (which you say also works). From there, it runs to the brake light switch and feeds 12v to that. That's the orange wire you tested at the brake light switch. Now from there, when your brakes are applied, the brake lamp switch sends 12v up to the turn signal switch. The reason being, your brake lights need to be "overridden" by the turn signal since the rear signals and brake lamps share the same bulb.
I can't find a wiring diagram of the turn signal switch in the column yet but with no turn signals on, that switch would have to allow 12v through it to feed the brake lamps. Then, say you flip on your left signal, it would have to interrupt the 12v feed from the brake switch and replace it with the intermittent 12 volts from the turn signal. So my guess is maybe your switch is bad. If I find a wiring diagram for the signal switch I'll have a look but it seems like it would be possible for the switch to successfully feed power to your front signals but not the rear.