Think about it this way: with colored fabric, you're essentially filtering
white light, and by doing so, wasting power and producing a far less spectrally pure light source. You will also not be able to get away from seeing the "white" in the light of the illumination source at the base. The better way is to use a light that produces the color you want in question and if necessary use white fabric to reflect that light. The process is akin to using colored LED bulb covers over white LEDs, vs. pure colored LED lights. The latter will be so much better quality, both in light and color.
Depending on the amount of power you have to work with, you can make some extremely bright colored backdrops using high powered LED emitters. At the lower level you can use 1-3W RGB emitters such as these
. They're cheap and easy to set up and control with the drivers at that site. You could also buy some of the M16 RGB LED bulbs and some M16 track lighting if you want something easy to set up.
At the high end, you can also use some high power emitters, like 10-30W LEDs
. These are spectrally pure LEDs and will look gorgeous. Colors like Red and Green will appear brighter than blue, and have higher luminous flux because of better energy conversion and sensitivity of the eye.
If you want to use black lights, don't get LEDs, get fluorescent black lights, as they produce far more black light than UV LEDs, and cost far less.