Here's how to delete from iCloud Music Library: https://support.apple.com/en-us/HT201323
Here's how to delete iTunes Store content: https://support.apple.com/kb/HT204343
For the latter, it says that deleting something you haven't downloaded will hide it from your purchase history as well.