I think you'd have to start by talking with a linguist rather than a mathematician, because the first step will be to define "most complex". A language consists of many parts; it might have a very simple phonology but complex grammar, for example. (And what I consider simple, you might call complex!) Furthermore, even within grammar, say, there might be no way to compare two entirely different kinds of grammar.