It depends on what you're building for.  If you're building a replica of a particular subject, you might strive to get as many details accurate to references as you can.  If you feel like letting your imagination run wild, then the only limit is really how well you can translate the picture in your head into the object.  It's up to you.  I'd say the same thing as Scott-it's a hobby, so whatever makes you happy doing it.

There was a modeler who wrote back in the day, and when things went online, he used to write articles for the online magazine, Internet Modeler.  His motto, which he included as a signature, was "Build what you like, the way you like!"

(Internet Modeler morphed into Cybermodeler Online, by the way.)