This phrase is very disturbing to me. Mostly the part about "kid gloves". To me, this can only have a certain number of possible meanings, none of which are good.
Kid Gloves- Gloves made out of the skin of children
Kid Gloves- Gloves made out of the skin of baby goats
Kid Gloves- Gloves specifically used to punch children
Kid Gloves- Gloves specifically used to punch baby goats
I suppose "kid gloves" could also mean to jokingly tease a pair of gloves, but frankly, that's just too far fetched to even consider. So, before you use the expression "take off the kid gloves", take a minute to consider that the phrase has extremely barbaric origins and should not be used lightly.
Personally, the least disturbing meaning is a pair of gloves specifically used to punch baby goats. But as to why anyone would have enough cause to punch baby goats so much that they would create gloves specifically for that purpose is beyond me.