We have had to take the database offline to repair our server. Multiple of the disks in the raid system storing all the data (mostly the images) have started to fail. In order to ensure all the data are save while we are ordering replacement hard disks, the server has been shut down. We hope to get the new disks as soon as possible and bring the database back online quickly after that. In the meantime, as usual, please keep observing all the targets and upload the data when we are back online.
Update 2023-01-31: It seems all admin people that need to approve purchases have now done so and the new drives should be ordered soon.
Update 2023-02-02: The new hard disks are being ordered.
Update 2023-02-10: Contrary to the above assurance, I have just found out that the new disks have only been ordered today!
Update 2023-02-14: The replacement hard disks have arrived today.
Update 2023-02-16: The 1st replacement disk has been added to the raid and is rebuilding.
Update 2023-02-21: There is more damage than anticipated to the raid hard disks. We will have to do some more investigations before hopefully restoring the system. If that fails we have to re-install the entire server from scratch and re-instate the database from the backup which we have.
Update 2023-02-24: We have a system drive version from 2018 that is readable. The database is backed up on the day we shut the database down. We have a complete copy of the code from June 2022. All that is still missing is the raw data which is on the damaged disk.
Update 2023-03-03: The disk will need to go for recovery. IT is looking to get a quote for the cost. The hope is that we can recover the live database file from the drive to not have to recover and re-index the database from the backup. Failing this, we will aim to include the database upgrades (x-match to Gaia and split of the large photometry table) in the recovery from the backup.
Update 2023-03-08: We have a quote for the disk recovery. While we are looking into how to fund it and waiting for the recovery attempt to happen, we plan to set up the server system and recover all the tables in the database bar the large photometry table. This will allow us to get back to data gathering quickly. We will the restore all e data in the background, upgrade the database structure and when this is done transfer the newly gathered data into the full database. The only think not working is access to all the data via the light curve plotting.
Update 2023-03-16: The server operating system is back running (2018 version). data search working. Need to test data submission and then re-install latest backup (without photometry) and and 2022 control software.
Update 2023-03-23: We restored a fully working version of the database and dropped the other 2018 version. Only the photometry table is blank. We now need to clean the new version from all images that had not yet been fully processed – these are 2405 images. Those users will be emailed later to please resubmit those images, we will send everyone a list of the filenames. Once all the tables have been purged of these data, we will test the image submission and processing steps to ensure they are all working and then we can go bakc life for data collection. Hopefully next week.
Update 2023-03-24: We tested the data upload and processing. It turns out that the reference catalogues for the photometry were not part of the backup. We have restored all but 5 of these. All for Gaia19eyy are missing and the U-band for IC348. These targets and filters are only observed by two people so no immediate problem. We can restore the U-band from the original image (we requested it) and remake the Gaia19eyy ones from other data. The last thing to solve before going live again is the back-up of the database every night.
Thank you very much!