Check your base ignition timing. If it's too far advanced, you'll have hot start issues that are EXACTLY as you describe because it kicks back the starter.
I had the same issue when I first got my '79 w/ a 400. When hot, it sounded like the battery was dying when I cranked it, but it usually did start regardless. Checked my timing, turned out I was at 35 degrees initial at idle without vacuum advance, which is absolutely absurd. Heck, the vacuum advance wasn't even hooked up. Dropped it down to 18 BTDC initial per factory spec, hooked the vac advance back up; and the car now starts at the turn of a key when hot.
If your timing is way up there, prepare to turn your idle speed screw in a turn or so to bring the idle back up, since lowering your ignition timing drops your RPMs.