Chapter 7. Text Tables

Text Tables are defined in a similar way to conventional tables with the added TEXT keyword.

 CREATE TEXT TABLE <tablename> (<column definition> [<constraint definition>])

The table is at first empty and cannot be written to. An additional SET command specifies the file and the separator character that the Text table uses. It assigns the file to the table.

 SET TABLE <tablename> SOURCE <quoted_filename_and_options> [DESC]

The default field separator is a comma (,). A different field separator can be specified within the SET TABLE SOURCE statement. For example, to change the field separator for the table mytable to a vertical bar, place the following in the SET TABLE SOURCE statement, for example:

 SET TABLE mytable SOURCE "myfile;fs=|"

Since HSQLDB treats CHAR and VARCHAR strings the same, the ability to assign a different separator to the latter is provided. When a different separator is assigned to a VARCHAR, it will terminate any CSV field of that type. For example, if the first field is CHAR, and the second field VARCHAR, and the separator fs has been defined as the pipe (|) and vs as the period (.) then the data in the CSV file for a row will look like:

 First field data|Second field data.Third field data

This facility in effect offers an extra, special separator which can be used in addition to the global separator. The following example shows how to change the default separator to the pipe (|), VARCHAR separator to the period (.) within a SET TABLE SOURCE statement:

 SET TABLE mytable SOURCE "myfile;fs=|;vs=."

HSQLDB also recognises the following special indicators for separators:

Furthermore, HSQLDB provides csv file support with three additional boolean options: ignore_first, quoted and all_quoted. The ignore_first option (default false) tells HSQLDB to ignore the first line in a file. This option is used when the first line of the file contains column headings or other title information. The first line consists of the characters before the first end-of-line symbol (line feed, carriage return, etc). It is simply set aside and not processed. The all_quoted option (default false) tells the program that it should use quotes around all character fields when writing to the source file. The quoted option (default true) uses quotes only when necessary to distinguish a field that contains the separator character. It can be set to false to prevent the use of quoting altogether and treat quote characters as normal characters. All these options may be specified within the SET TABLE SOURCE statement:

 SET TABLE mytable SOURCE "myfile;ignore_first=true;all_quoted=true"

When the default options all_quoted= false and quoted=true are in force, fields that are written to a line of the csv file will be quoted only if they contain the separator or the quote character. The quote character inside the field is doubled when written out to the file. When all_quoted=false and quoted=false the quote character is not doubled. With this option, it is not possible to insert any string containing the separator into the table, as it would become impossible to distinguish from a separator. While reading an existing data source file, the program treats each individual field separately. It determines that a field is quoted only if the first character is the quote character. It interprets the rest of the field on this basis.

The setting, null_def, can be used to simplify importing text files containing empty fields. These fields are interpreted as null but the user may want an empty string or another default value instead of null. With null_def=true defined in the text source string, and a table column that is defined as DEFAULT <val> NOT NULL with a constant value for the default, the default value will be used instead of any empty or NULL field.

The character encoding for the source file is ASCII by default, which corresponds to the 8-bit ANSI character set. To support UNICODE or source files prepared with different encodings this can be changed to UTF-8 or any other encoding. The default is encoding=ASCII and the option encoding=UTF-8 or other supported encodings can be used. From version 2.3.4, the two-byte-per-character encodings of UTF-16 are also supported. The encoding=UTF-16BE is big-endian, while encoding=UTF-16LE is little-endian. The encoding=UTF-16 is big-endian by default. This encoding reads a special Unicode character called BOM if it is present at the beginning of an existing file and if this character indicates little-endian, the file is treated as such. Note HSQLDB does not write a BOM character to the files it creates from scratch.

Finally, HSQLDB provides the ability to read a text file as READ ONLY, by placing the keyword "DESC" at the end of the SET TABLE SOURCE statement:

 SET TABLE mytable SOURCE "myfile" DESC

Text table source files are cached in memory. The maximum number of rows of data that are in memory at any time is controlled by the cache_rows property. The default value for cache_rows is 1000 and can be changed by setting the default database property. The cache_size property sets the maximum amount of memory used for each text table. The default is 100 KB. The properties can be set for individual text tables. These properties do not control the maximum size of each text table, which can be much larger. An example is given below:

 SET TABLE mytable SOURCE "myfile;ignore_first=true;all_quoted=true;cache_rows=10000;cache_size=1000"

The properties used in earlier versions, namely the textdb.cache_scale and the textdb.cache_size_scale can still be used for backward compatibility, but the new properties are preferred.

Text tables may be disconnected from their underlying data source, i.e. the text file.

You can explicitly disconnect a text table from its file by issuing the following statement:

 SET TABLE mytable SOURCE OFF

Subsequently, mytable will be empty and read-only. However, the data source description will be preserved, and the table can be re-connected to it with

 SET TABLE mytable SOURCE ON

When a database is opened, if the source file for an existing text table is missing, the table remains disconnected from its data source but the source description is preserved. This allows the missing source file to be added to the directory and the table re-connected to it with the above command.

Disconnecting text tables from their source has several uses. While disconnected, the text source can be edited outside HSQLDB, provided data integrity is respected. When large text sources are used, and several constraints or indexes need to be created on the table, it is possible to disconnect the source during the creation of constraints and indexes and reduce the time it takes to perform the operation.