Please Help: Game enemy movement and interaction

I’m relatively new to creating games, however I’ve been a flash website developer (and designer) for over 7 years, and am used to OOP and AS2 Event-based programming. It was a bit tough to swallow the one-global function runs everything method but it all seems to be going fine now. The only problem I’m having right now is my math, and character movement and interaction. Also figuring out hitTests when the player punches, seeing as how you might have multiple hitTests be true, yet I want only those in the front row to receive damage.

Here’s what I’m working on: http://www.mode7interactive.com/projects/cockpuncher/

What I would like it to do, is for the enemies to swarm, but not walk “through” each other. Because the player only has his fists to defend himself (stipulation of the game unfortunately), they need to start far away, but then come up to the player’s range and then mosy about in front of the player, jabbing him from time to time. The only thing, is that currently they look like paper dolls, because they move through each other, and the movement isn’t very good either. This will allow the player to punch away their health until theirs none left, starting the new wave.
So there it is, my first big problem is the enemy movement and interaction with each other in a simulated 3d environment. The second would be how to get the player to deal damage to them, how can you tell that he punched the guy in the front row, etc.
If someone can either walk me through the math and principles, or guide me to where I can research it, I would be very greatfull.
-James