In essence there are cases outside the well developed uses (CNN, LSTM etc.) such as Neural ODEs where you need to mix different tools (ODE solvers and neural networks) and the ability to do Differentiable Programming is helpful otherwise it is harder the get gradients.
The way I can see it being useful is that it helps speed up development work so we can explore more architectures, again Neural ODEs being a great example.
Differential programming is about building software that is differentiable end-to-end, so that optimal solutions can be calculated with gradient descent.
Probabilistic programming (which is a bit more vague) is about specifying probabilistic models in an elegant and consistent way (which than then be used for training and inference.)
So, you can build some kinds of probabilistic programs with differential programming languages, but not vice versa.
In essence there are cases outside the well developed uses (CNN, LSTM etc.) such as Neural ODEs where you need to mix different tools (ODE solvers and neural networks) and the ability to do Differentiable Programming is helpful otherwise it is harder the get gradients.
The way I can see it being useful is that it helps speed up development work so we can explore more architectures, again Neural ODEs being a great example.