That would completely depend on your definition of 'annoying': ticks spread more diseases than just Lyme. For example, I'm really not sure whether Human Anaplasmosis [1] is only as annoying as a leech.
Lyme disease is a couple times more common than other diseases, and it can have permanent, sometimes debilitating effects. In many people it can be hard to recognize or diagnose before permanent effects set in. The other things you can get just make you sick.
But true, its very hard to get sick from a leech. A mosquito or black fly would be a better comparison. Without the vaccine, ticks are second only to like... bot flies and tarantula wasps.
[1] https://www.cdc.gov/anaplasmosis/symptoms/index.html