How does Harry Potter find his invisibility cloak when he takes it off?

Wouldn't it go invisible? In the first movie, the inside of the cloak is visible when he's wearing it and looking through it. So would he have to turn the cloak inside out in order to find it again later?