"But here's the key: for internal use, not just for showing other people our photos."
"We need something that combines 'we'd like to show people some photos' with 'we have a lot of photos we just store and annotate.'"
These phrases are exactly what I asked myself a few years ago. After some heavy digging around, I settled upon Photo Organizer. I started customizing it to my needs, and ended up contributing so much that I eventually was handed maintainership of the project.
http://po.shaftnet.org is the project's current home.
It's not all that you asked for, but it wants to be. It's database-driven (PostgreSQL), fundamentally multi-user, and is intended to be a photographer's primary image repository. It has decent access controls (beyond "public" and "private", that is) and pretty good tagging abilities. It has good export capabilities, is GPL'ed, and runs on your own server, so there's no danger of lock-in.
As an added bonus you can group together multiple versions of photos. Its filesystem layout makes it fairly easy to back up, but there's no automatic mechanism for doing so. (I use a nightly cron job that does a database dump and an rsync)
My personal installation runs off a server at home, and currently manages just shy of 30K photos taking up some 113 gigs.
Photo Organizer sounds like it is a fairly close match for your stated needs, and is improving continually as new itches come up. :)