macOS machines can normally be connected to with the machine setting "Enable NetBIOS" set to ON or OFF. It's worth trying both settings, though, in case a firewall is blocking one of the TCP ports that FileBrowser uses.
The most likely cause of this error is a firewall blocking FileBrowser's network requests. We recommend switching off the firewall temporarily to prove whether this is the cause of the problem. If you can connect when the firewall is disabled, please click the link below to show you how to configure the firewall to permit 'File Sharing'.