Shouldn't it be 12/52 * 11/51 * 10/50?
The answer will only come after you seriously think about all the decisions you need to make to get three of a kind.
12/52 * 11/51 * 10/50--you are picking your 1st card that has probability of 12/52 and then you are picking your 2nd card that has probability of 11/52 and then you are picking your 3rd card that has probability of 10/52. The problem is that you at best only picked three cards.
You need to understand how when giving a deck of cards how to go about picking 5 cards that has 3 of one denomination and 2 other cards of different denomination. Random guessing will not work.
Here are the obvious steps to get 3 of a kind.
Pick a denomination (for the 3 of a kind)
Pick 3 cards of that denomination.
Now pick 2 other denominations (for the two '1 of kind')
Pick 1 card of the one of two these denominations.
Pick 1 card from the 2nd denomination.
Now that you know what the steps are you need to find out the number of ways each step can happen.