I'm not a Pontiac engine guy so this will be general info only, and probably obvious. Do you have an assembly manual? Or the 76-81 Parts? Or even service manual? A few diagrams in there showing the order of assembly for brackets etc.
Sounds like something off, a bracket added in wrong order, or a spacer or washer where it shouldn't be. If you have to add any washers to line things up you already know something isn't right. I'm not sure if Pontiac used different length water pumps or swapped pulley types to suit. Did you have all of it together before? Apart from the brackets coming off, new parts that are affected are water pump only?
Many rebuilt alternators simply use an existing case, doesn't mean that's what it currently suits, just began life as decades earlier.
I have #10009669 as a 301 power steering pulley. It may or may not be the 'same' as used on 400 with updated number, but will add that in case it matters.
I have #481038 as standard Pontiac 400 family (and 301) pulley with AC. Should be ok. Same also for #480509 pulley.
I don't have a listing for #481040 water pump pulley, will look elsewhere.
Alternator bracket looks to be correct.
Is this your general pulley and belt arrangement?
20240626_170308 by
Ben, on Flickr