Octave and Matlab focus on numerical computing, while Mathematica focusses on symbolic computing. The first means working with arrays (and higher dimensional arrays) of (mostly) floating point numbers, while the later means working with abstract syntax trees of mathematical expressions.
Specifically, the one "equivalent" to Mathematica is Maxima [0], I remember using it for symbolic computation on my linux machine. Especialy coupled with Wx GUI [1].
Both Maxima and GNU Octave are pretty old (mature?) projects with probably outdated-looking (feeling) GUIs. Just to mention some younger and more "fresh" projects: In the SciPy ecosystem, there is SymPy https://www.sympy.org which has the obvious advantage of being Python, gaining from a big (also scientific) ecosystem, such as numpy, pandas or the JupyterLab.
A large OSS player is nowadays also SageMath https://www.sagemath.org/ which is a kind-of-meta CAS which can actually proxy evaluations to Mathematica, Maxima, Octave and many others, in a more or less uniform pythonic notation.
Probably needless to say, when it comes to evaluation power (for instance for integral solving), Mathematica is unrivalled from OSS projects. However, in certain fields (such as number theory for SageMath or the overall web notebook infrastructure by the IPython and Jupyter projects), the concurrency is catching up or may already have overtaken Wolfram's products.
Wow, sympy looks amazing. Seems to solve exactly the kind of things I'd use Mathematica/Wolfram for, especially back in my university days. Will play around with it when I have some time to repeat my math knowledge, been meaning to refresh some things for a while now.
