The consensus algorithm of both is very unlike the other.
IOTA has something like "Monte Carlo Random Walk for Tip Selection by most Proof-of-Work" while Byteball has 12 trusted witnesses to stamp and order transactions as they see them, stamp only when satisfying consensus rules, a Main Chain can then be built from all possible transaction histories (branches). No proof-of-work aside from signatures.
Today all witnesses are run by main developer. A hub on the Byteball network can push witnesses, if users trust a particular hub they can check a box to pull the witness list, this is just user-friendliness.
The witnesses are supposed to be selected by humans by talking, like a popularity contest. There is no automatic protocol-selection process aside from described below.
Each user/transaction is allowed 1 mutation in the 12 list, so technically there can be more witnesses than 12 in the network, but only 12 selected in each transaction.
Then, when enough transactions have replaced same witness with a new one, the stability point, ie best parent, can be advanced and next witness can be replaced. See the whitepaper for more details on this advancing of "best parent" in the Main Chain. Like a snowball.
The process is user-driven, the costs to replace it by malicious intent is too high due to transaction fees, the limit on 1 mutation and hence slow time to actually replace all of them or a majority is also to ensure human herd-behaviour doesnt make bad decisions.
IOTA has something like "Monte Carlo Random Walk for Tip Selection by most Proof-of-Work" while Byteball has 12 trusted witnesses to stamp and order transactions as they see them, stamp only when satisfying consensus rules, a Main Chain can then be built from all possible transaction histories (branches). No proof-of-work aside from signatures.