I don't smoke but don't mind others doing it. If anyone visits me and wants to smoke they go in the garden or if bad weather a small room that has a door to the garden, with door open.A bit of 'passive' smoke out of doors is not going to hurt anyone, you'd have to be in a smoky atmosphere for a long time to be harmed and even then probably wouldn't be though the smell would linger on clothes. In the 1980s I remember people smoking at their desks at work, nobody said anything, but that all changed later in the decade.
SteveH has said he smokes pipe & not heard of any problems with pipe smoking except rarely on the lip. Steven most your pipe around, don't always park it in the same place in your mouth. I expect where you work you're not allowed to smoke so it's only in the evening or weekends. I quite like smell of pipe smoking.