Whenever I have been swimming I have found that people really aren't that interested in looking at me/my body... I think because you're not sunbathing around the pool on a tropical island (well maybe you are haha but I'm not), most people are there just to exercise and focussed on getting whatever their goal is in the pool done. Some pools dont allow you to wear 'non swimwear material clothing' in the pool (like cotton tshirts/shirts) because they can stretch in the water and drag you down/tangle up a bit (safety risk), thought it's worth knowing in case you want to check with the pool if they have any restrictions. I used to wear 'rash shirts', which are like boarding/uv tshirts for surfing etc, but i found it just meant i had one more soggy piece of clothing to take home, and nobody was looking at me really, so i decided id just swim in my swimsuit (once youre in the water its hard to see what's happening on anyones skin anyway).
Swimming is great, it's worth the sometimes uncomfy dash into the pool haha
hope you manage to get yourself there once your arm has healed some more!