Appendix B. HyperSQL Database Files and Recovery

Database Files

Database catalogs opened with the file: protocol are stored as a set of files. This document describes the contents of these files and how they are stored.

A database named 'test' is used in this description. The database files will be as follows.

Database Files

test.properties	Contains the entry 'modified'. If the entry 'modified' is set to 'yes' then the database is either running or was not closed correctly. When the database is properly shutdown, 'modified' is set to 'no'. You can add the optional entry readonly=true to make the database files read only.
test.script	This file contains the SQL statements that makes up the database up to the last checkpoint. All the data for MEMORY tables is also stored in this file.
test.data	This file contains the binary data records for CACHED tables only.
test.backup	This file contains a backup of the parts of the `test.data` that have been modified since the last checkpoint. The file is deleted at each checkpoint or shutdown. The size of this file can be as large as the `test.data` file.
test.log	This file contains the extra SQL statements that have modified the database since the last checkpoint. It is used as a redo log.
test.lobs	This file contains the lobs. If a database has no BLOB or CLOB object, this file will not be present. This file contains all the lobs that are currently in the database, as well as those that belong to rows that have been deleted since the last checkpoint. The space for deleted lobs is always reused after a CHECKPOINT.

A CHECKPOINT is an operations that saves all the changed data and removes the test.log followed by the creation of an empty log. It also deletes the test.backup file. A SHUTDOWN is equivalent to a CHECKPOINT followed by closing the database.

The disk space needed for the database depends on the amount of data. After the database is opened more space, up to slightly greater than the size of the test.script and test.data files is needed during operation.

States

Database is closed correctly

State after running the SHUTDOWN statement

The test.data file is fully updated.
There is no test.backup file.
The test.script contains all the metadata and CREATE TABLE and other DDL statements. It also contains the data for MEMORY tables.
The test.properties contains the entry 'modified' set to 'no'.
There is no test.log file.

Database is closed correctly with SHUTDOWN SCRIPT

State after running the SHUTDOWN SCRIPT statement

The test.data file does not exist; all CACHED table data is now in the test.script file
There is no test.backup file.
The test.script contains all the metadata and DDL statements, followed by the data for MEMORY, CACHED and TEXT tables.
The test.properties contains the entry 'modified' set to 'no'.
There is no test.log file.

Database is aborted

If the database process was terminated without a SHUTDOWN, or the SHUTDOWN IMMEDIATELY command was used, the database is in aborted state.

Aborted database state

The test.properties contains 'modified=yes'.
The test.script contains a snapshot of the database at the last checkpoint.
The test.data file is not necessarily consistent.
The test.backup file contains just sections of the original test.data file that correspond to test.script at the time of the last checkpoint.
The test.log file contain all data change statements executed since the last checkpoint. As a result of abnormal termination, the end of file may be incomplete.

Procedures

The database engine performs the following procedures internally in different circumstances.

Clean Shutdown

Procedure B.1. Clean HyperSQL database shutdown

The test.data file is written completely (all the modified cached table rows are written out) and closed.
The file test.script.new is created using the current state of the database.
The entry 'modified' in the properties file is set to 'yes-new-files' (Note: after this step, the test.data and test.script.new files constitute the database)
The file test.log is deleted
The file test.script is deleted
The file test.script.new is renamed to test.script
The file test.backup is deleted
The entry 'modified' in the properties file is set to 'no'

Startup

Procedure B.2. Opening the Database

Check if the database files are in use by checking a special test.lck file.
See if the test.properties file exists, otherwise create it.
If the test.script did not exist, then this is a new database.
If it is an existing database, check in the test.properties file if 'modified=yes'. In this case the RESTORE operation is performed before the database is opened normally.
Otherwise, if in the test.properties file 'modified=yes-new-files', then the (old) test.backup and test.script files are deleted if they exist and the new test.script.new file is renamed to test.script.
Open the test.script file and create the database objects.
Create the empty test.log to append any data change statements.

Restore

The current test.data file is not necessarily consistent. The database engine takes these steps:

Procedure B.3. Restore a Database

Restore the old test.data file from the backup by copying the original sections from the test.backup file.
Execute all the statements in the test.script file.
Execute all statements in the test.log file. If due to incomplete statements in this file an exception is thrown, the rest of the lines in the test.log file are ignored. This can be overridden with the database connection property hsqldb.full_log_replay=true which results in the startup process to fail and allows the user to examine and edit the test.log file.
Close the database files.
Open the restored database.

	Appendix B. HyperSQL Database Files and Recovery
Appendix A. Lists of Keywords		Appendix C. Building HSQLDB Jars