Indeed, accessibility labels are the only way to reference views. Granted, your tests are running within your process, so they sky is pretty much the limit.
We're still considering the best way to implement gestures. There's some rudimentary support, but it's incomplete right now. Your input is welcome!
Also, keep in mind that many views just "do the right thing" as far as accessibility goes. Buttons for example will use their title as the default accessibility label if you don't set one explicitly.
Are accessibility labels the only way to reference views?