Gem blocks explode when you cause damage to them with matches or birds. Matching with items of the same color as the gem block causes the gem block to explode right away. 

If you make matches with items that are not the same color as the gem block, you will need three matches for the gem block to explode.