Came across this pretty cool tool from VKI Studios in Vancouver. It’s a quiz about the usability of your site. Questions range from details about your site layout and design to strength of your copy and keywords.
It isn’t necessarily comprehensive nor is it designed to solve all your problems, but if your boss has said to you "We need to improve the usability of the site" (or more likely "we need more web sales") then this tool is a place to start. At the very least it gives you a framework to start thinking about web usability.