HyperSQL Database Files and Recovery

$Revision: 5925 $

2022-10-20

Table of Contents

Database Files
States
Procedures
Clean Shutdown
Startup
Restore

Database Files

Database catalogs opened with the file: protocol are stored as a set of files. This document describes the contents of these files and how they are stored.

A database named 'test' is used in this description. The database files will be as follows.

Database Files

test.properties

Contains the entry 'modified'. If the entry 'modified' is set to 'yes' then the database is either running or was not closed correctly. When the database is properly shutdown, 'modified' is set to 'no'.

test.script

This file contains the SQL statements that makes up the database up to the last checkpoint - it is in sync with the contents of test.backup.

test.data

This file contains the binary data records for CACHED tables only.

test.backup

Depending on the backup mode ( SET FILES BACKUP INCREMENT {TRUE | FALSE} ), this file contains either a backup of the parts of the test.data that have been modified since the last checkpoint (the default setting, TRUE) or the complete compressed backup of the test.data file at the time of last checkpoint (when FALSE).

test.log

This file contains the extra SQL statements that have modified the database since the last checkpoint. It is used as a redo log.

test.lobs

This file contains the lobs. If a database has no BLOB or CLOB object, this file will not be present. This file contains all the lobs that are currently in the database, as well as those that belong to rows that have been deleted since the last checkpoint. The space for deleted lobs is always reused after a CHECKPOINT.

A CHECKPOINT is an operations that saves all the changed data and removes the test.log followed by the creation of an empty log. A SHUTDOWN is equivalent to a CHECKPOINT followed by closing the database.

States

Database is closed correctly

State after running the SHUTDOWN statement

  • The test.data file is fully updated.

  • When BACKUP INCREMENT TRUE is used, there is no test.backup at all. Otherwise the test.backup contains the full compressed test.data file.

  • The test.script contains all the metadata and CREATE TABLE and other DDL statements. It also contains the data for MEMORY tables.

  • The test.properties contains the entry 'modified' set to 'no'.

  • There is no test.log file.

Database is closed correctly with SHUTDOWN SCRIPT

State after running the SHUTDOWN SCRIPT statement

  • The test.data file does not exist; all CACHED table data is in the test.script file

  • The test.backup does not exist.

  • The test.script contains all the metadata and DDL statements, followed by the data for MEMORY, CACHED and TEXT tables.

  • The test.properties contains the entry 'modified' set to 'no'.

  • There is no test.log file.

Database is aborted

If the database process was terminated with a SHUTDOWN, or the SHUTDOWN IMMEDIATELY was used, the database is in aborted state.

Aborted database state

  • The test.properties contains 'modified=yes'.

  • The test.script contains a snapshot of the database at the last checkpoint.

  • The test.data file is not necessarily consistent.

  • The test.backup file contains just sections of the original test.data file, or a full snapshot of test.data that corresponds to test.script at the time of the last checkpoint.

  • The test.log file contain all data change statements executed since the checkpoint. As a result of abnormal termination, the end of file may be incomplete.

Procedures

The database engine performs the following procedures internally in different circumstances.

Clean Shutdown

Procedure B.1. Clean HyperSQL database shutdown

  1. The test.data file is written completely (all the modified cached table rows are written out) and closed.

  2. If backup mode is not INCREMENT, the test.backup.new is created which contains the compressed test.data file.

  3. The file test.script.new is created using the current state of the database.

  4. The entry 'modified' in the properties file is set to 'yes-new-files' (Note: after this step, the test.data.new and test.script.new files constitute the database)

  5. The file test.log is deleted

  6. The file test.script is deleted

  7. The file test.script.new is renamed to test.script

  8. The file test.backup is deleted

  9. If the file test.backup.new exists, it is renamed to test.backup

  10. The entry 'modified' in the properties file is set to 'no'

Startup

Procedure B.2. Opening the Database

  1. Check if the database files are in use by checking a special test.lck file.

  2. See if the test.properties file exists, otherwise create it.

  3. If the test.script did not exist, then this is a new database.

  4. If it is an existing database, check in the test.properties file if 'modified=yes'. In this case the RESTORE operation is performed before the database is opened normally.

  5. Otherwise, if in the test.properties file 'modified=yes-new-files', then the (old) test.backup and test.script files are deleted and the new test.script.new file is renamed to test.script.

  6. Open the test.script file and create the database objects.

  7. Create the empty test.log to append any data change statements.

Restore

The current test.data file is not necessarily consistent. The database engine takes these steps:

Procedure B.3. Restore a Database

  1. Restore the old test.data file from the backup. Depending on the backup mode, decompress the test.backup and overwrite test.data, or copy the original sections from the test.backup file.

  2. Execute all the statements in the test.script file.

  3. Execute all statements in the test.log file. If due to incomplete statements in this file an exception is thrown, the rest of the lines in the test.log file are ignored. This can be overridden with the database connection property hsqldb.full_log_replay=true which results in the startup process to fail and allows the user to examine and edit the test.log file.

  4. Close the database files, before opening the restored database.


$Revision: 6621 $