Wow, so many comments and great ideas. Here's my two sense worth.
In my mind the best way to handle it is to keep it simple. If it's too complicated, it turns me right off.
Personally I don't like the rating system for critiquing. Here's why:
A) I think it would be complicated
B) I think it would be very time consuming for the panel to control and could prevent the panel members to have the time to post their own photos and critique too!
C) I think it could encourage people to critique without thought to get it done quickly in order to upload more of their own photos.
D) I also think it would be stifling for those who prefer to learn from others by reading critiques and not critique until they feel more comfortable.
I believe encouragement for those people is a must and understanding them too.
I think a permanent section on the site on how to critique would be a good idea.
And, I think it's important for us all to remember that no matter what grade of photographers we are, newby's to advanced, we all want to learn more and improve on what we do. That puts us all in the same boat and therefore we can all critique from what we know or like and from our knowledge and personal likes and dislikes.
Here's another idea:
Why not put a special "critique" gallery together so we can all peruse it easily. I think a lot of our photos that aren't getting critiqued is because people can't easily find them. In the critique gallery there would be all new photos specifically ready to be critiqued by their photographer and after a predetermined number of critiques have been made on that photo, say 3, that photo is automatically removed from that gallery and put in the general gallery.
That gives the photographer the choice of whether or not he/she wants their photos critiqued or simply shown. And those that are in the critique gallery will welcome being critiqued and therefore no one should be shy about critiquing them.
Hope I've helped.
Lots of varied opinions here, I'm just another one.