I just don't want garbage file names when sharing a file system between systems that don't agree on the encoding. I was thinking maybe some mount option. I can use ISO-88591 and skip UTF-8. I haven't found a mount option for ext4 or xfs yet.
There isn't one. The names in ext4 and xfs are opaque binary with some simple limitations (like null bytes). Encodings simply don't exist at the fs layer.
You could probably write some filter using fusefs, but in practice... I think you should configure the servers / clients to agree on encoding instead. Better supported and shouldn't be that much work.
Linux filesystems are not encoding aware. Paths are just treated as opaque byte strings. However, there is ongoing work add configurable safe filenames to Linux: https://lwn.net/Articles/686789/
But it won't allow you to force ISO-8859-1 in this form. However you could filter out non-ASCII characters.
