This weeks featured image shows the size of our daily database backups. As you can see, since Monday, despite people adding data, the database has slightly decreased in size. So is there something wrong again and we are loosing data?
No. We have a script that checks in any extracted light curve, if there are duplicate data entries. This happens at times if the ‘submit data to database’ command is send more than once before the data is fully written. In the past we have run the script occasionally on some individual light curves and then reset and re-processed the offending image.
As part of our next large research paper, we have identified all potential members of the young clusters in ALL HOYS fields and extracted their data. Thus, we now have run the duplicate data check over these 10546 light curves. It turns out that indeed there are ~250 images for which the data has been duplicated (sometimes more than twice). This is a small fraction of all images (~71000), and we have started to reset and re-process those. Thus, the slightly decreased database size.
There are a few observations though. While the occurrences of the duplications are widely distributed amongst all the users, hence they cannot be completely avoided, there are a few users with a significantly enhanced number of duplicate images. Thus, when you are processing the images, please remember: The buttons on the website are to be clicked once. If you double click, the associated commands will be send/executed twice and thus cause errors or duplication. Furthermore, especially when your internet connection is slow, please be patient. Do not press the ‘submit’ button again, even if it takes a while for the new page to load. Also: Do not press the ‘return’ button on your web browser, or <ALT>+<LEFT> to return to the previous page and submit the data again. All of these, can potentially cause these problems, and there is no simple automatic way to correct them, other than resetting the problem images and re-processing them, which is very time consuming.