Perhaps now I have a guess as to the problem. You should have only one contact for a given doctor - even if he has multiple offices.
The visit page does NOT (currently) capture what office (where) you had your visit - just who you visited.
In your contacts section - for a given contact - you can list multiple locations (offices). But when you link a visit to a provider - just link the provider - not a composite entity of provider and office.
I hope this makes sense...
Records For Living Support Team.