1)Какая высота этого холма? -how high is this hill?
2)Какой глубины это ущелье? - how deep is that canyon?
why was the genitive case (этого холма) used in 1) but nominative case (это ущелье) was used in 2)?
Both are actually correct. And you are right, the first one is a little more usual.
I am afraid I cannot give you a complete explanation (we need a linguist here), but maybe we can try literal translations as analogies:
Какая высота этого холма? — What is the height of this hill?
(“of this hill” turns into genitive case in Russian)
Какой глубины это ущелье? — This canyon is of what depth? ( I know it is very bad English but I hope the phrase is possible to understand. “of what depth” turns into genitive case in Russian.
I see what you mean! So какой in 2) also genetive case like глубины, correct??? Thank you for your help!! I was really dying to know 
Yes exactly, both words in the phrase “какой глубины” are feminine singular genetive case (nominative would be “какая глубина”).
Got you!! Thank you so much 