High-cooperative ISAM files

High-cooperative (HC) files are a sub-type of ISAM files that are designed to be safely and efficiently shared by multiple programs or processes at the same time, with built-in support for coordinating file access. The name refers to their emphasis on cooperative access rather than exclusive ownership. Key benefits include the following:

The additional metadata required by an HC file to support online structural changes and concurrent access is stored in its header. You can create a new ISAM file as high-cooperative using ISAMC, or you can patch an existing file with isutl -p to turn it into a high-cooperative file.

To modify HC files without exclusive access, use the hcutl utility. Hcutl enables you to extend record lengths or add and remove keys while other processes continue to read from and write to the file. As a result, downtime is limited to the brief period needed to replace executables, rather than the potentially lengthy process of modifying and replacing large data files.

You must update your applications to use the HC file changes. Even after the file header is updated, applications must be updated to recognize and use new keys or record layouts. In particular, applications should call %ISINFO or ISSTS to check for the length of a record and the number of keys in the record to react to planned hcutl operations. Existing applications should be able to read and write normally even with concurrent hcutl operations, but if a record is extended or a key is added or deleted while a file is open in an application, the application needs to make the appropriate %ISINFO or ISSTS call to get the latest file information and react to the change.

Tip

If an application gets its structure definitions from Repository and you're adding a new key to match an existing field, you may want to modify your repository to include the new key to stay consistent, but you don't need to rebuild the application. However, if you want to extend a record and add a new field and key into the extended portion, we recommend that you update your repository to reflect the expected updated record, update the application to use this repository definition but keep using the old definition until %ISINFO says the new key is ready, and then use hcutl to perform the desired operations.

Compatibility considerations for high-cooperative files

High-cooperative files have the following features and restrictions:

When to use (or not use) high-cooperative ISAM

Using high-cooperative ISAM files is most advantageous when your system must be available 24/7. If downtime is unacceptable, HC files are a good fit, because they support online structural changes (like adding or removing keys) and online patching, and recovery-friendly update behavior.

You may prefer regular ISAM files if

Creating a high-cooperative file

There are three ways to create a new HC file:

Extending a record's length in a high-cooperative file

Use hcutl -e to extend a record in an HC file. See hcutl for more information.

Working with keys

Adding a key

Use hcutl -ka, and specify a key using the same syntax you use when calling ISAMC.

Deleting a key

High-cooperative files allow you to delete existing, alternate keys without requiring exclusive access to the file. To delete a key, use hcutl -kd.

You can’t delete or replace the primary key in an HC file.

Deleting a key that uses qualified duplicates doesn't change the way existing records are stored. However, if you want to add a new key to replace the old one, you won't be able to specify qualified duplicates; you'll have to choose between no duplicates (if allowed by the existing data) or non-qualified duplicates, just as with any other new key.

If you delete the last key in the file, it will be completely removed. As soon as a key is deleted, it becomes unusable for any file I/O operations.

Note

As a convenience, you can also use isutl -kd to delete the last key in a file without converting the entire file to HC, but there are restrictions. See isutl -k for more information.

Non-qualified duplicate keys

By default, duplicate ISAM keys are stored with a qualifier that allows SDMS to distinguish between different records and keep the duplicates in a first-in-first-out or last-in-first-out order (depending on how DUPLICATE_ORDER is set). High-cooperative files, however, can’t use qualifiers, because they could cause conflicts with existing stored data. Instead, they use a different kind of key called a non-qualified duplicate, which allows a duplicate key to be added to a file without changing how the existing records are stored, with duplicate order undefined.

Although non-qualified duplicates were designed to support HC files, they’re also available for general use by specifying "DUPS=noqual" in a key specification when calling ISAMC or by including "DUPLICATES yes" and "DUPLICATE_ORDER none" in an XDL file.

Non-qualified duplicates require a minimum ISAM compatibility level of 6.5. This means that a file will be created with a compatibility level of 6.5 if a dups=noqual key is specified during file creation.

Virtual keys

When an HC file is created or an existing file is patched to HC, the system allocates a certain number of "virtual keys" to reserve space for the maximum number of keys that can be added to the file using hcutl. (You can also delete an existing key to make room for a new one.) By default, enough virtual keys will be allocated to fill a header block. Depending on factors like the number of actual key definitions, this will probably be about 50 keys, which should be more than enough for normal usage.

If you don't want to use the default, you can alternatively specify the maximum number of virtual keys manually when creating or patching an HC file with ISAMC, isutl -p, a .par file, or an .xdl file:

Since this value sets the maximum number of keys that can ever be added without recreating the file, make sure you choose a number that is high enough. Virtual keys can only be specified when creating a new HC file; files that are patched to HC later must use the default value.

Example using high-cooperative files

To perform high-cooperative operations on a file that's actively being used by an application, you may need to make some changes to the application. In particular, expanding a record's length could cause runtime errors if you attempt to store or write a record using a length that doesn't match the current state of the file. You can avoid these errors by updating the application ahead of time to use the new record length, but call %ISINFO before writing to the file to determine whether to use the old or new length.

If the application writes to the file in many locations, it might be tedious to update all of them. One way to simplify this process is to use global I/O hooks to hook into the existing file I/O operations. The following example suggests such an approach. This example assumes an application that performs I/O on a file named customers.ism, which needs to be expanded to make room for a new key. The application uses a repository structure definition for the customer record. Before expanding the record length, the structure definition in the repository is updated to reflect the new length, and the application is rebuilt and linked against a source module compiled from the following new code:

import Synergex.SynergyDE.IOExtensions

subroutine syn_globalhooks_open
    in channel, n
    in modes, OPModes
    in filename, a
global common
    global_hooks, i1
    global_chan, d3

proc
    if ((int)(modes & OPModes.OPEN_INDEXED) && filename == "customers.ism")
    begin
        (void)new global_hooks(channel,filename)
    end
    xreturn
end

namespace hooks_namespace
class global_hooks extends IOHooks
.include "customer" REPOSITORY, PUBLIC RECORD = 'customer'

private record
    filechan		,i4
    filename		,a256
    savedbuf		,a256
    savedrfa		,a6

public method global_hooks
    in channel, n
    in name, a
    parent(channel)

proc
    filechan = channel
    filename = name
endmethod

method ~global_hooks
proc
endmethod

public override method store_pre_operation_hook, void
    inout buffer, a
    in flags, IOFlags
record
    reclen, i4
    numkeys, i4
    bufflen, i4

proc
    reclen = %isinfo(filechan, "SIZE")
    numkeys = %isinfo(filechan, "NUMKEYS")
    bufflen = ^size(buffer)
    if (bufflen > reclen)
    begin
        ; Attempting to write a record larger than the on-disk record
        ; size is going to cause an error, but there's nothing we can
        ; do the prevent the error at this point. Instead, we'll save
        ; off the record and rfa so we can write it out correctly
        ; after the error occurs.
        savedbuf = %atrim(buffer)
    end

    mreturn
endmethod

public override method store_post_operation_hook, void
    inout buffer, a
    in optional stored_rfa, a
    in flags, IOFlags
    inout error, in
record
    reclen, i4
    numkeys, i4
    bufflen, i4
    localfilechan, i4

proc
    reclen = %isinfo(filechan, "SIZE")
    numkeys = %isinfo(filechan, "NUMKEYS")
    bufflen = ^size(buffer)
    if (error == 316) ;%DBR-E-IRCSIZ, Invalid record size
    begin
        ; If the initial write operation failed due to invalid record size,
        ; try again using a new channel and saved RFA (if applicable).
        if (savedbuf .eqs. %atrim(buffer))
        begin ;If the current buffer matches the saved buffer, the saved RFA should be current too
            open(localfilechan=0,u:i,filename)
            store(localfilechan,buffer(1:reclen)) [err=store_err]
            error = 0 ;Clear error, now that we've successfully stored
store_err,
            close localfilechan
        end
    end

    mreturn
endmethod

endclass
endnamespace

If a store operation occurs before the record length has been extended, the initial operation will fail with an "Invalid record size" error. In that case, this code will respond to the error by retrying the store on a new channel using the current record length in the file. After the record length has been extended, the initial store operation should succeed, and the retry should be unnecessary.

If the program performs more I/O operations on the file besides store, additional override methods may be needed. See Synergy I/O Hooks API for more information.