HPE: Patch accidentally destroys 77 TBytes of Kyoto University research data

Actually, a patch from Hewlett Packard Enterprise (HPE) was only supposed to make a backup script readable, but the consequence of the update turned into a disaster. A whopping 77 terabytes of research data and its backups at Kyoto University were destroyed by the bug and are gone.

Serious bugs caused by HPE patch at Uni Kyoto

For Kyoto University, the year 2022 starts with a disaster. Because at the end of the year2021, the university has irretrievably lost a whole 77 TBytes of research data due to a faulty patch on the part of HPE. Around 34 million files from 14 different research groups were affected.

The incident already occurred between December 14 and 16. About a third of the groups will probably never see their data again, according to the release from Kyoto University. The reason lies in a simple script update for the servers that accidentally deleted all backups due to an error.

Hewlett Packard Enterprise, respectively the Japanese offshoot Hewlett Packard Japan GK operates the affected supercomputer of the university. According to the report, an update was distributed for a backup script that was supposed to automatically delete log files after ten days.

However, this was only supposed to improve its visibility and traceability by renaming variables. Instead, the script had overwritten another, already active bash script. As a result, the variables were undefined, which mistakenly led to a subsequent deletion of the real research files, instead of just the log entries.

Data storage on other systems

HP Japan says it takes full responsibility for the incident, as confirmed in a letter to the university. Kyoto University plans to prevent a reoccurrence of the bug and initiate additional measures, such as mirroring data. Additional, regular backups should also ensure more efficient and less error-prone data storage.

The researchers themselves were also encouraged to additionally store their data on other systems. The measures would not prevent data loss due to hardware failure or force majeure, the statement added.

Known primarily for the fields of materials science, immunotherapies and chemistry, Kyoto University is one of Japan’s most prestigious institutions.