Astronomers deal with enormous quantities of data. One night's telescope scan can keep a university astronomy team going for months. Reserving a time slot on a telescope and, more importantly, being able to use the slot, whether it be in the visible, infrared, X-ray or radio region, is a preoccupation of astronomers around the world. The data that they collect is normally the intellectual property of the individual researcher or group for a proprietary period - generally a year. After this time, the complex digital images are available to the wider astronomical community.
But how do astronomers access this wealth of data easily, at the minimum so that they can avoid duplicating work that has already been done? A data management software system now being installed in the UK will allow this to happen and, coincidentally, will allow astronomers to check out telescopes and associated instruments for faults, before the all-important time-window for observation opens up and the opportunity lost.
The Institute of Astronomy (IoA), at the University of Cambridge in the UK, is using data management software from K-PAR Archiving Software to store and index data on optical disks. The Cambridge Astronomy Survey Unit (CASU), a group within the IoA, now acts as the archival centre for ground-based optical and infrared astronomical data.
Stored images in the archive also contain engineering data concerning telescope and instrument operation that can help diagnose hardware and software faults.
Jim Lewis, senior research associate at CASU, said: 'One example is a camera shutter only opening halfway - one of those 'it happens once in an unpredictable while' sort of problems which this system will detect. The header of the image has lots of engineering data, which can diagnose a problem like this - it may be that the shutter may not open properly if the camera is allowed to get too warm.
'There may also be instances where an upgrade to the code that reads out detectors has caused an artefact to appear on the image or perhaps a noisy preamp is causing too much readout noise. These are all sorts of problems that can be diagnosed with the help of images that are accessible from a fault database.'
These data are now being put online and astronomers around the world can contact the IoA (generally via the Web) to view and download images.
'Telescope datasets represent unique snapshots of astronomical objects,' says Lewis. 'Once the data are taken, they can be analysed in many different ways and hence long-term storage is absolutely vital. Traditionally tape media was the only viable method for long-term storage of large amounts of digital data. However, tape tended to degrade very quickly with time and usage, and we often lost data as a result. Astronomers were sometimes put off requesting archival data because of the long lead-time in having their requests fulfilled (as information retrieval from tape is inefficient). The inability to view data in real time and assess their usefulness for a particular study, also meant that sometimes data would be requested but would turn out to be inappropriate for the astronomer's needs.'
A typical use of the system is a more detailed study of an object that might be in the archive. The section of the sky could have a main object in the centre of the frame and the object of interest fainter to one side. From this image, the astronomer can tell exactly where to point the telescope and, depending on its brightness, how long it needs to be trained on the object.
The IoA is now storing a large amount of its archived data on DVDs held in a Pioneer DRM 7000 DVD-R jukebox, which is controlled through K-PAR Archiving Software's Archimedia data management software.
Archimedia is a general-purpose, automated library management and archiving product aimed at the Windows NT/2000 and Solaris operating systems. It is feature-rich and includes file-based caching (which is essential for fast access of large files, or when many users are accessing files simultaneously); graphical management tools (which simplify installation and configuration); and offline media management (which notifies users if the disk they require has been removed from the library).
A good deal of the UK's ground-based astronomical data is taken at the Isaac Newton Group of Telescopes (ING) on La Palma in the Canary Islands. Here the data are taken by any of three telescopes and are automatically archived using Archimedia on one of two Pioneer optical disk jukeboxes.
A copy of the original data is passed onto the principal observer and a copy is held in the jukeboxes on site. Having the data on a write-once medium such as a DVD is an insurance policy against accidental deletion or corruption, which can often happen on magnetic disk media. Once the jukeboxes reach capacity, the disks are shipped to the IoA in Cambridge and ingested into the archive.
One area where Archimedia has proven its worth at the ING is in the tracking of faults at the telescopes. The instruments used on telescopes can be very complex, and if they are not working properly, it is possible that the data quality can be seriously compromised.
With hundreds observations done every clear night, it is vital that data can be viewed and assessed quickly in order to ensure that any faults can be spotted and rectified before the next night's observing begins.
Jim Lewis explains: 'We needed a solution which would give us greater control over our data, right from the telescope and on to its final destination in the archive. Fast information retrieval and reliable data integrity were essential.
'Optical disks, such as DVDs, offered us data security through write-once technology, and Archimedia data management software greatly simplified and accelerated the whole data retrieval process. Because Archimedia is non-proprietary, we were able to achieve a greater understanding of the software and how it works.
'This made us feel confident that the software could meet our requirements before we invested in it. Also, Archimedia is a more cost effective solution compared to competitive products - like all research projects, finances are tight, so it is vital that money is spent wisely. Finally (and probably most importantly for us), we liked the fact that K-PAR has its own support team based in the UK. They have proven to be helpful and responsive to our queries and requests whenever we have needed them.'
The implementation of the archival solution was supported by systems integrator, Active Network Systems, which specialises in providing storage, backup and archival solutions specifically for educational establishments and research institutions.
Andy Ibbotson, managing director at Active Network Systems, said: 'The IoA knew it wanted to use DVDs as its archival storage medium. While other jukebox manufacturers were hesitant about developing a DVD product, Pioneer recognised the market potential and launched the first DVD compatible jukebox. K-PAR has always kept on top of industry developments, and as such was the first data management software manufacturer to support DVD jukebox technology.'
Jim Lewis concludes: 'At the end of the day, it's the quality of the data and the ease of access that makes an archive useful. Astronomers need to access data quickly, as rapid assessment is the only way to know whether the data they are about to request will, in fact, fulfil their requirements. Fast access at the telescope also helps enhance the quality of the data, as detection and diagnosis of instrumental failures can be speeded up greatly.'
First light for Europe's virtual observatory
Rapid progress is being made with Europe's Astrophysical Virtual Observatory (AVO). January saw a live demonstration of the prototype system, and a newly launched AVO website which explains the progress being made with this European Commission-funded project.
Like its US cousin, the National Virtual Observatory (NVO), the AVO will enable astronomers to gain instant access to data from both ground- and space-based telescopes that are observing across the entire wavelength range - from gamma rays, through visible light, to radio waves - and then to combine these data seamlessly, using a common interface from any machine, thereby enabling remote mining of multi-wavelength data archives. (For more in-depth coverage, see Scientific Computing World, May/June 2002.)
The partner organisations in the programme are the European Southern Observatory (ESO) in Munich, Germany; the European Space Agency (ESA); AstroGrid (funded by the Particle Physics and Astronomy Research Council as part of the UK's E-Science programme); the CNRS-supported Centre de Données Astronomiques de Strasbourg (CDS); the University Louis Pasteur in Strasbourg, France; the CNRS-supported TERAPIX astronomical data centre at the Institut d'Astrophysique in Paris; and the Jodrell Bank Observatory of the Victoria University of Manchester, UK.
The AVO consortium quickly formed a close alliance with the NVO and both teams have representatives on the other's respective committees. Both are also in communication with other similar initiatives elsewhere in India, Australia and Japan. The ultimate aim is a truly global virtual observatory, under the auspices of the recently formed International Virtual Observatory Alliance.
A key element of the programme is Grid computing and reusable Grid middleware - the glue that sticks the system together. AstroGrid is a UK consortium of six university groups and a government lab, and is also one of the six AVO partners.
It is dedicated to creating a distributed computing network, based on next-generation Grid technology, that will manage the data processing, storage and delivery tasks associated with accessing and mining the vast astronomical archives.
Jim Lewis, of the Institute of Astronomy at the University of Cambridge, commented: 'The AVO is not quite the same thing as the K-PAR Archimedia system. What they are doing is essentially more high-level, in trying to link various astronomical database and catalogues with all sorts of different database managements systems and middleware into a single structure. There may be some K-PAR stuff in there, but it will be at the level of the individual contributing sites, rather than with AVO per se.'
The AVO partners will now join with all astronomical data centres in Europe to put forward an FP6 IST (Sixth Framework Programme - Information Society Technologies Programme) integrated project proposal to make a European virtual observatory fully operational by the end of 2007.