Hi. The Giver is one of my favorite books too. When I read the book I thought the colors have always been there it's just that the people are not taught the concept of "colors" so they never thought about it. They never distinguish or prefer something because the color is different. It's like they never see any person as beautiful or ugly or different from the other. That's how I interpret it.
Do you remember the special ability that the Giver has? I only remember that Jonas can see red. My book is as of right now not here with me so I can't look up.