No, it's a collection of vectors (which may be considered ordered or unordered depending on context). The 'vectors' part is fine, but the 'collection' part is the bit that causes the confusion. You can't differentiate a collection (even of vectors), any more you can differentiate a bag of coloured balls. What's the derivative of 'blue'?
But if that collection is plugged into an expression, so as to denote a collection of expressions when result when the individual element vectors are plugged into the expression, and the resulting expressions are differentiable, and you then differentiate each of those expressions and then collect the result back in the same order and represent it as a matrix, then that's fine.
This is what happens when we talk about "matrix differentiation", we mean "differentiation with matrices". But the matrix itself is not a derivable object. Only the expressions upon which it confers its notational semantics have a derivative.
No. It is a mathematical fact that matrices fulfill the conditions of forming a vector space (subject to reasonable conditions like having elements drawn from a field). Matrices are just as differentiable as (other) vectors!
I think you're arguing that a matrix with constant real entries isn't differentiable with the standard derivative. This is true of course, but hasn't anything to do with vectors.
