Turns out it's not that difficult.

  1. Pick or create an image with your preferred logo or whatever
    If you search for an image to be used on the internet, be wary of copyrights!

  2. Create an .ico-file that is 16 x 16 pixels from the image in §1.

  3. Upload the resulting favicon to your site's root folder. Note that the file must be called favicon.ico. It should also be owned by the apache user and group with octal permission 644.

  4. Check your homepage, refresh as necessary, and check for the icon. 

  5. Done.

 

Source:
https://www.ostraining.com/blog/joomla/how-to-add-a-custom-favicon-in-joomla/.