# Gimme five...

When the imminent Mathematica release 5.0 was announced, at the beginning of June, it was the first full-digit upgrade for four years. Even incremental updates had only come twice within that time; that's unusual these days. Admittedly, Wolfram Research has been putting a lot of its attention into a broadened market base with collateral product releases but, nevertheless, a major version caused ears to prick up. The final beta appeared shortly thereafter (the full release appearing in early July) so I've had it out on the road for about two months now, at the time of writing.

Wolfram tags it 'The Advanced Algorithm Release' (complete with capital initials), and heavily emphasises speed. Overall, this is a release in line with the general thrust of Wolfram's activity profile: it aims to break out of the market segment within which Mathematica grew up, and establish itself across a wider area. There is a growth in numerical aspects, in particular, and in the ability to deploy symbolic pre-processing in support of them. A number of applied math aspects are also expanded, reworked or repositioned closer to the core. Functions have also been added or enhanced, and very usefully so - but speed and numeric prowess get the limelight.

There is, indeed, a dramatic increase in execution speed across many areas. Large matrix operations, for instance, show an order of magnitude performance increase, or better, over the previous best. Wolfram invites direct comparisons with dedicated numerical systems in this respect, and these do seem to show that Mathematica can, intelligently applied, broadly compete on equal terms. My own practical test results, on problems to hand, don't quite match Wolfram's but, to be fair, I have devoted considerably less time and expertise to their optimisation. Mathematica offers its own functions (Timing, which looks at net kernel time spent in the CPU; and AbsoluteTiming, which concerns itself with gross execution time from command issue to result display) for exploring speed issues. Having a nasty suspicious mind, I tried other means of comparison, but had to agree that, on the whole, it lives up to its claims. Some of this comes from submitting the numerics engine to thoroughgoing revision; some from the intelligent application of symbolic methods to prepare problems for maximum efficiency in numeric handling.

Internal speed or efficiency of the kernel (or other components) is not the only issue; those components have to talk to one another in order to deliver results. Because Mathematica uses TCP/IP for intercomponent communications, adoption of a new protocol that allows those communications to operate at the underlying platform speed, pays off in total system response. The resulting dividend is obviously dependent on the particular platform in use. There are other mechanisms at work, too; all explained in the literature but increasingly fuzzy in my mind. Suffice to say that, on the evidence of even short-term results, they seem to work well in concert. With statistical aspects weighing heavily in my work, developments in that area always attract my early attention; and this is, of course, tied to the numerical emphasis anyway. The most immediately obvious change was the absorption of common statistical descriptors into the kernel, which is nice. It always niggled me having to load a separate package for such everyday details. The speed demon raises its head here, too, with faster import and export of data stored in tabular arrays. Wolfram quotes improvements of between one and two orders for such data, depending on its size and structure. Even at gross measures, I can vouch for a very useful sixfold jump, which makes a real difference when repeatedly handling very large bodies of information.

A new function, Total, also proved its worth. Total is equivalent to Apply[plus, list] and so is analogous to the Sum() function in Excel-type spreadsheets, but subtler. It takes the usual range of supplementary arguments covering detailed list handling. List items can be, for instance, complex quantities; and the function can be applied to sparse array elements (of which more later). Despite its almost throwaway announcement, this one function makes a very real difference to some activities.

Another addition for which I found good statistical use, curious at first sight but growing on me as its possibilities revealed themselves, is the programming construct Sow and Reap pair. As their names suggest, Sow scatters things behind it according to specified rules and Reap follows along picking them up in specified ways. There's no room here for a more exhaustive treatment than that, but I have used them for revealing simulation studies and they are of much wider application to operational problems, amongst others.

Moving along in the same vein, there is a new standard package, StatisticsPlots, which provides a variety of basic exploratory representations. Box and whisker plots, I was pleasantly surprised to discover, offer a free choice of the quantile range enclosed by the central box rather than just the first and third quartiles often provided. A pairwise matrix plot (small 2D scatter plots presenting every variable pairing within a list of n variables as an nxn contingency grid), a quantile/quantile plot to explore commonalty of distribution, and a Pareto plot which can take either raw or frequency grouped input, are in the same spirit. None of this is earth shattering in itself, but from the user viewpoint it greatly reduces the initial drudgery in large projects. Therefore, from a Wolfram perspective, it reduces the initial pain barrier for statisticians plucking up courage to use the more heavyweight tools that a mathematical package can offer.

Having referred to one new package (statistical plots), it makes sense, before passing on, to mention the other. Algebraic numbers, bundled as 'root objects' containing a minimal polynomial and an integer signifying the root represented, can now be handled easily and submitted as-is to functions for processing. As with many developments, they package and simplify operations that hitherto had to be purpose built from other constituent parts.

My mathematical life revolves largely around communication. Whether with clients, colleagues or students, mathematical work usually culminates in output that must be packaged for conceptual transmission. For that reason, the next area that drew my own particular interest was a significant extension of Mathematica's authoring tools. These are not new in version 5 (they were introduced in 4.2), but they have certainly grown up. Wolfram has been working on separate application development in this area, and some fruits of that work have found their way into the flagship product. There is an now a new authoring palette for slide shows, the environment has improved significantly, style sheets benefit, and so on. Also new on the scene, promising a future of developing collaboration support, is a wordprocessor-like facility for difference tracking.

The publicity emphasis on numerics should not overshadow the symbolic development. Symbolic manipulation, after all, is not only where Mathematica comes from, and its core strength, but also one of the means by which numerics have been pushed so far and with such success. Solution to polynomial systems is now complete over real or complex numbers; rational ordinary differential equation systems are fully solved; solution sets are represented for discrete and continuous algebraic and transcendental cases; differential-algebraic equations are supported, as are nonlinear, partial and difference systems. And so it goes on through a whole clutch of new or modified features.

For conversion of polynomial systems to tensors, the function CoefficientArrays extracts variable coefficients from a system of p equations, yielding a list length p+1 of sparse array objects. Sparse arrays are one of the headline items in 5.0; the position p of value v in an nD array of dimensions d1,d2...dn is given for those items with value other than the default value (assumed to be zero if not specified). Given my concern with simulations, this is a real winner; those whose work involves optimisation will be equally pleased. A more generally useful way to generate these sparse arrays, however, is to apply SparseArray to an incoming dense array data set; conversion then happens automatically. Conversion back to sparse can be achieved in the same way, using either Normal and Matrixform; I've found that these back-and-forth conversions offer a very convenient way to explore different base states in a model - just change the default element value, and process the resulting set. Along with the sparse array structure comes a stable of specialised sparse algorithms for conducting operations on them; this makes many extremely large matrices manageable for the first time but also, conversely, adds a layer of dramatic speed increase - even for smaller ones.

There are a raft of additions and improvements in numerical linear algebra, which gains arbitrary precision support. It benefits greatly from sparse handling, of course, but there are significant improvements in the optimisation of dense forms too. Improved efficiency in the repeated solution of matrix equations, a FindFit function which seeks parameters for an expression to best fit a data set, p norms, and so on, all increase the fetch of numerical work - which seems to be a priority area for expansion in Wolfram's current strategy. For some operations, where the OS supports it, multiple processors are accessed. The recurrence equation solver RSolve has been moved to the kernel and given a lot more clout. A feature which seemed a bit 'something and nothing' when I first looked at it, a function and associated structures for assuming variable values, turned out to be extensively useful once I'd gotten my head around the possibilities and spent some time running the examples. It ties in particularly well with the newly polished-up and revamped integration facilities, for instance.

There are a number of areas for which I have no personal use, but which are important and which also interest me in a spirit of pure enquiry. These I have of course not explored in as much depth as the ones that directly affect me. Greatly improved PDE solvers are one case in point. Valuable for those whose work it affects, and of morbid academic fascination to me as one who was always bottom of the class in this sort of thing, is the new ability to prove rigorously many geometric identities and inequalities. A noticeable speed increase in large number arithmetic is another example - Wolfram says up to three times faster where there are less than a thousand digits, and better than that as the size increases into the millions, though on limited experiment I can personally vouch only for a clearly noticeable improvement.

The flip side of progress is an inevitable concern over backward compatibility. The documentation lists four functions that have become obsolete, and some cases of possible conflict. I did find one case where a new function name clashed with a label used in one of my past programming efforts, easily solved by a slight modification of my label. When I briefly revisited some of the add-on packages which I've reviewed against version 4.2, however, I found no problems - just an increase in speed.

I can't provide more than a fistful of snapshots in a review space like this, nor explore every avenue in the time available. It's clear, though, that Wolfram had not given this upgrade its full 'point oh' designation lightly. There is thoroughgoing change throughout the whole structure, and a clear but prudent shift in direction to engage industrial and commercial markets head on; numeric and speed issues dominate, though not at the expense of respectable advance on the symbolic front.