Need title-Swedish film - Wife n Mother discuss affair of husband/son

I'm looking for the title of what I believe is a Swedish film - possibly one of Bergman's. The film takes place in summer in the well-off estate of a successful man. During the course of the film, it is revealed that the male protagonist is having an affair with a servant. Later, the co-workers of the servant approach the wife and mother of the man to let them know that he promised the servant she'd be given a shop. Later in the movie the wife and mother are sitting at a table and begin to discuss the situation. The man is walking behind them and protests that he does want them talking about his affair, so they stop for a time but then later resume their discussion.

I've checked the plots of the prominent Bergman films I thought it might be but none seem to fit the bill. Does anyone know what film this takes place in?

Thanks for your help!

I should have added another detail - at another point in the film, another character observes that the wife is extremely tolerant to put up with her husband's infidelities. The response to this is that the husband is so good to her (the wife) that she doesn't need to mind.

I think I may have gotten it wrong that it was the man's well-off estate - I believe it is the mother's and the man and his wife are there on a visit or for the summer.

Smiles of a Summer Night

Don't think so - In "Smiles of a Summer Night" Petra is the maid and she eventually winds up going off with another servant. The male protagonist is infatuated with his young wife, for whom the marriage is unconsummated - I checked that earlier so, it's got to be another film.

My first guess is Fanny and Alexander (1982) (Fanny och Alexander)
My second guess is Zorn (1994)