I “secured” my server with fail2ban, a tool to ban ips when a ssh login failed (in my case anyway)…
So… i have friends who have accounts on the server as well.
You can extrapolate for yourself, but the essence is that i have to unban ips on a regular basis.
So here is the line to unban an ip:
sudo fail2ban-client set sshd unbanip $ipoffriend
sshd is my jailname and personalised. The ipoffriend has to be the ip of the friend.