This subject is very topical in my workplace. Essentially if you afford someone access to your premises, you have a duty to consider their egress. In my opinion, the assessor is being too specific and is verging onto territory that is the preserve of the building managers. However, if there are people above the ground floor that have impaired mobility, and you have facilitated their access, you should also take reasonable measures to ensure that they are safe in the event of a fire or other emergency that would require a full evacuation. Just my observations, but I hope you reach a successful conclusion.