On: GPT4 is 8 x 220B params = 1.7T params

[deely3]

Heh. I understand all these words separately. Btw, to which of the question of parent comment this is an answer?

[seattleeng → deely3]

Ancestor model = pre-trained model that used a large diverse corpus

Descendant models = models fine tuned from an ancestor on one particular domain, e.g. by partitioning your training data by subject or source

Democracy = some weighted mix of the descendant models is used to find the next token