BitmapData.draw on vectorized sprite

I got a bitmapData that is drawing some content from an array, over and over again.
This array contains a lot of sprites, less detailed and more detailed vectorized images.

When the bitmapData draw the less detailed sprite it takes around 4ms, which is very good!
However, when it draws the more detailed image, it can be up to 250ms!

I’ve tried to set the sprites to cacheAsBitmap before the bitmapData draws it,
but that made no difference.

Some more details:
The sprites are about 500x500
I’m using Starling, which means I can’t add vectorized images as they are.
(I have to use bitmapData and create a texture from that)

Very grateful for help!
~Tompa