Do I need to match the dryer ball color with my laundry?

No, the color of the wool dryer balls will not transfer. If you have garments that bleed (like dark denim or red shirts) and are concerned about the fabric dyes transferring between loads, you can color-coordinate the balls with your loads: use a darker color dryer ball for darker loads, and reserve a lighter color dryer ball for your white sheets and towels.