From what I've seen and heard, you don't have to worry much about them unless you get too close to City Hall. The Organization has their base there. I guess they use the heartless as security.

Though, I have to ask. If you're fine with talking animals, then what's so far-fetched about this being a castle?