If your compost is outdoors, worms might end up there naturally, either from worm cocoons stuck in soil stuck on weed roots ...