EventListener problem

In my application there are four different interfaces/layouts: start, recording, playback and save data. In all layouts you can go forward and backward. Before loading a new layout (all done by ActionScript) it removes all display objects and after that all listeners associated (I´m using a weak listener reference). The problem is when you go back and forward a couple of times buttons, timers etc. don´t work anymore. I can´t see the “light in the tunnel” yet because It removes all listeners and display objects before creating new so it always start from the beginning. Any suggestions why this happends?

I´m using a state design pattern for play/stop/pause/record/append state wich gives a good structure, but when it comes to create and remove interface/layouts it feels very messy. Anyone who can share some ideas of how to structure up the layout-events-thing?