Yeah, it's an entrance that looks like a big head with its mouth open, on a tiny island on the lake. But I would advise that you at least pay a passing visit to Sheogorath's Shrine (somewhere north of Leyawin along the west boundary), just to get a flavor of what you are in for.
Also, when you are stout enough, there's an unspoken miscellaneous quest up in the great north that you shouldn't pass up, especially if you're a fan of
Beowulf (the epic, not the atrocious movie).
There is no quest trigger or anything, but if you head up northeast to places only higher level characters can reach, you'll come across Lookout Rock, which is the most spectacular view in the game. On that rock, there is a small campsite with a journal nearby. That should get you started.
Oh, and on the subject of leveling your character, you can only level up when you sleep. And since you
choose to sleep, you can maintain a lower level for a long while, which is helpful in getting yourself situated without much fear.
HINT: When fighting in caves, ruins, etc., if you get injured badly, try backtracking to a place where no enemies are nearby and then "Wait" for an hour (see controls for how to "wait"). This will replenish all your health, magic, stamina, etc... and you won't even have to use a potion or spell.
Man, I need to get off work so I can go home and fire up
Oblivion. There's got to be one or two quests I haven't done yet.