I'm not sure about this because I cant get to my 360 for a few hours, but I think I know what's going on with the diagonal glitches. I'm pretty sure it's not the move being varied, but both moves activating with seperate effects.
The logic of 4A+B~?A+B's coding is probably written like this:
*If 1_4_7 is held, do 4A+B.
*If 3_6_9 is held, do 6A+B.
*If 2_5_8 is held, do A+B (animation).
*If 4 or 6 is NOT held, do A+B (actual move). {<--the error}
I'm also thinking that A+B starts a few frames before 4A+B or 6A+B do - this explains the variance you mentioned in 4A+B~3_9A+B. A tip stab will ring out because the enemy is no longer in 4A+B's hold, but 4A+B~A+B's toss, which many moves can ring out from. This also explains the damage reduction, since another move is being added. Logically, 4A+B~1_7A+B is a bad option since 4A+B~4A+B can only pull an enemy towards you if they're still held by 4A+B - instead they're in 4A+B~A+B, and it's a wasted move.
And about the failed 4A+B~A+B, it could be replicated in Recorded Command? Maybe it ties into my theory above - somehow activating the A+B animation, but not the move. If I'm right, holding 4 and 8 at the same time would do it, but of course that's impossible, lol. Not sure what the real cause is, maybe it's accidentally tied into force cancelling...
EDIT: I just realized, maybe 4A+B~6A+B is just activating a standard move (3B+K?) that has special properties (the AT) only in 4A+B's hold. All this makes perfect sense so far, I wonder if it's any help in explaining the original glitch in question...