My first guess was that for most fonts the glyph "B" has splines, while "#" does not (only segments) and in line of principle rendering splines is more expensive than rendering segments. But probably this is nowhere that relevant.
That was my first thought too, but I expect glyphs will probably be cached for a while after being rendered, which should make any difference in rendering speed negligible.