The fact that you have to wait for it to restart almost sounds like it's flooding. When the choke is on, the throttle plates are open more, which leans out the mixture which is why it runs fine at that time. When at your curb idle, the throttle plates are near closed providing the most ideal conditions for flooding. Then again, the fact that it dies immediately after it comes off the choke may mean that something else is going on. It may be lean-stumbling, too, which means you just need to enrichen the idle mixture.
How long do you let it stay on the choke before blipping the throttle to take it off the fast idle?
Holleys are appealing to a lot of people because they're calibrated very rich to run on just about any engine. You can slap a Holley onto a weed whacker and get it to run. QJets get a bad rap because they're calibrated very lean and are application-specific, right down to the motor and options of the car they're made for. For instance, the original QJet that came on your Olds 403 is very different than the one that came on my Pontiac 400.
You don't always need black smoke to be running rich. Have you pulled the plugs to see if they're carbon-fouled?
The suggestions regarding ignition hold water, too. Luckily it's an easy and a cheap thing to rule out. Ignition components are so easy and cheap to replace that I just throw parts at the ignition system until something catches.
Also take a look at the ignition control module. You mentioned it has no boxes, so I'm guessing it's an HEI style dizzy? Couldn't tell in the video. Your symptoms point to an ICM that is overheating and then cools down allowing you to restart the car.
Often times, the worst carburetor problems are ignition problems.
And I know this sounds silly, but do a thorough once-over for vacuum leaks. Like, massive, blatantly-obvious ones. When I pulled apart my carb once, I forgot to reconnect the PCV line when I re-installed the carb and it did exactly what your car is doing. It would run while giving it throttle (which is essentially all the choke mechanism is), but would die when returning to idle. Small vacuum leaks will cause your idle to either drop or speed up depending on your mixture, but big enough ones will just stall out your motor completely.