Of all the problems you can have with this setting, brightness is not one of them.
I have been using 4k 40" tvs and monitors with OSX and FreeBSD/linux for over 2 years now and overall I am quite happy with the choice.
I wouldn't go much above 40", as it would involve too much head movement (while sitting at least, I find myself comfortably using only 2/3 of the height).
One annoying problem especially with older or low end TVs was definitely lag. Even after disabling all "picture enhancements" and setting the equivalent of gaming mode, many TVs in 2014-2015 had high lag, from the 120ms that even Samsung/LG were showing, up to the 300+ms of Seiki and Haier and other no-name TVs. That makes the monitor unusable even for text editing. From personal experience, 2016 Samsung at least are much better, but better try them before buying.
Next comes viewing angle and color -- cheap panels have a limited vertical viewing angle, and sometimes even limited color depth.
Then you have to fight with OS and applications. For some combinations of OSX and TVs, I had the OS applying non-removable overscan after identifying the screen as a TV. OSX also does antialiasing which results in blurred characters with small fonts in terminals. I haven't found a good bitmapped font to use.
Apps also have problems, e.g. chrome on OSX does not refresh well the bottom bar when you move windows between the 4k and the retina screen. Or, some video players fail to go full screen on 4k.
