Updating an enterprise application can be tricky. If an application is central to business operations, it needs to have as much uptime as possible to avoid interrupting work. At the same time, applications often need improvement to keep up to date with the business’s current needs. These two requirements can come into conflict. For Synergy applications, one of the biggest points of contention can be in modifying ISAM files.

As time passes and software requirements are re-evaluated, you may find an application would work better if its files had longer records to store more data. You may also find additional index keys are needed to better organize and look up certain data, or that existing keys were defined incorrectly or aren’t very useful. In such cases, you’d want to update not only the application itself but also the data files it uses. But this can become challenging for very large applications. While executables can be replaced quickly during a brief period of downtime, the files can take much longer to update.

The normal way to make these types of changes to an ISAM file is to use fconvert, reading in the existing file and writing the records out to a new file with the new specification, e.g.,

fconvert -ii oldfile.ism -oi newfile.ism -d newfile.xdl

However, this requires exclusive access to the file, meaning that the application must be stopped (or at least prevented from accessing the file) while the conversion is taking place. If multiple large files need to be converted, this can add up to a lot of downtime. Businesses may be understandably wary of taking that much time to update an application and may forgo useful improvements as a result.

To alleviate such problems, we’re pleased to introduce high-cooperative (HC) files, a new subtype of ISAM files that can be modified using a command-line utility without requiring exclusive access. By using an HC ISAM file, you can extend the length of a record or add or remove keys and still allow other processes to read and write to the file while the conversion is underway. Then the downtime can be limited to the time it takes to replace executables, without worrying about the time it takes to modify and replace large data files.

New concepts

The new high-cooperative ISAM features introduce a new file type and some additional new concepts required to support them. While I’ll cover these concepts in more detail later, I’d like to introduce them here for easy reference.

HC files

High-cooperative files are a subset of ISAM files designed to support high-cooperative operations. They’re mostly the same as regular ISAM files, but they store some additional information in their headers. They’re compatible with most other ISAM features (e.g., ERASE_ON_DELETE, NETWORK_ENCRYPT, TEXT, (FULL)RESILIENT, STORED_GRFA, and PORT_INT), but not all.

You can create a new ISAM file as HC, or you can patch an existing file to make it an HC file. See “How to get started” below.

The hcutl utility

Once you have an HC file, you can run hcutl to perform HC operations, such as initial conversion, extending a record, adding a key, or deleting a key. (See hcutl in the 12.4 Synergy/DE documentation.) Other programs can still access the file normally while these operations are executing.

You can use the hcutl -i command to quickly access high-cooperative information about a file, such as whether it’s already an HC file or can be converted to HC and what HC operations might be available. By using a wildcard, you can check multiple files at once. For example:

hcutl -i *.ism

[changetracking.ism]
changetracking.ism is not an HC file
It cannot be directly converted to HC, as it has change tracking enabled.
Consider transferring contents to a new file.

[intermedhc.ism]
intermedhc.ism is an intermediate HC file
To finish converting, run:
        hcutl -c intermedhc.ism

[normalfile.ism]
normalfile.ism is not an HC file
To patch to HC, run:
        isutl -p -qfile=hc normalfile.ism

[segkey.ism]
segkey.ism is a full HC file
Current record length: 50
Keys defined: 2 (56 available to add)
A key can be added with keynum 2
Current keys:
         Key0: 'start=1,length=5,keynum=0'
         Key1: 'start=1:6:11:16,length=5:5:5:4,keynum=1'

Non-qualified duplicate keys

By default, duplicate ISAM keys are stored with a qualifier. If two records are stored with the same value for a key that allows duplicates, each will have a unique qualifier value as well. This qualifier allows SDMS to distinguish between different records and keep the duplicates in a particular order (first-in-first-out or last-in-first-out, depending on DUPLICATE_ORDER).

However, when adding a key to an HC file, we can’t use qualifiers, as they could conflict with existing stored data. For this reason, we’ve introduced a new type of duplicate key: the non-qualified duplicate. Instead of storing qualifiers, this key uses a different mechanism to distinguish between duplicates, which allows us to add a duplicate key to a file without changing how existing records are stored, although duplicate order is undefined.

Virtual keys

When an HC file is created or an existing file is patched to HC, SDMS allocates a certain number of “virtual keys” to reserve space for the maximum number of keys that could potentially be added to the file through hcutl (although you can always delete an existing key to make room for a new one). By default, enough virtual keys will be allocated to fill a header block.

How to get started

You can create an HC file by specifying “,hc” or “, high_cooperative” as part of the filename when calling ISAMC or in a .par file. If you’re using an .xdl file, you can add “HIGH_COOPERATIV yes” to the file specification.

But if you want to take advantage of HC features in an existing application, it’s likely that you already have an ISAM file that wasn’t created as HC. To get such a file ready to work with hcutl, you’ll need to patch it with isutl, like this:

isutl -p -qfile=hc myfile.ism

This operation does require exclusive access to the file, but only long enough to update the file’s header, which is very quick. The file is now a high-cooperative file and can be freely accessed by applications while you run operations with hcutl.

However, a patched HC file is likely to be in an intermediate state at first. HC files store data a bit differently than other ISAM files, and the data file probably hasn’t been converted yet. For small files (less than 10 million records), isutl automatically converts the data during a patch, but for larger files it just updates the header. You can specify “-qfile=hc,noncovert” to prevent data conversion regardless of file size, or “-qfile=hc,convert” to force data conversion regardless of file size, but be aware that data conversion through isutl will require exclusive access and may take a while. To avoid downtime as much as possible, you can stick with the default behavior and finish data conversion using hcutl rather than isutl.

After patching a file, you can check to see if it’s in an intermediate state by running hcutl -i, as shown in “The hcutl utility” above. Running ipar will also inform you if a file is an intermediate HC file, with the comment “Intermediate file, data conversion required.” To finish converting an intermediate HC file, use hcutl -c, like this:

hcutl -c myfile.ism

Now ipar and hcutl should show the file as high-cooperative, without the “Intermediate file” message. Whether creating a new file or converting an existing one, you should now have a fully high-cooperative file, ready for further modification.

Note: High-cooperative operations (e.g., hcutl -c and -ka) are designed for shared file access, whereas isutl operations are designed for speed. For example, with a 4-million-record, 13-key file on a fast machine, using isutl may take 10 seconds with exclusive file access, while the same operation may take 4 minutes with hcutl. In other words, the same operation using isutl will generally be faster but will count against downtime, whereas hcutl can be started at any time and left alone until completed.

How to extend a record

The most basic high-cooperative operation is changing the length of a record. It’s only possible to extend the length of a record, not reduce it. Because of this, and because it would be easy to accidentally run the command twice, the -e command in hcutl requires you to specify the desired final length of the record rather than the number of bytes to add. For example:

hcutl -e 80 myfile.ism
Extended record size from 70 to 80

The utility will update the record size in the header and tell you the result of the extend operation. If the requested size is more than a 20% increase, you’ll be prompted before the operation takes effect.

As soon as you extend the record size of a file, all records effectively have the new record size. If you read an existing record, the extended area will be filled with spaces. When a record is stored or updated, it will be written out to the full extent of the new size.

Extending the length of a record also updates the bucket geometry for the file, which you can examine using the -b or -vb options for isutl. This affects how space from deleted records is reclaimed for new records and ensures the updated file continues to operate normally.

It’s important to note that updated records are treated as having the new size, even though the stored data hasn’t changed. If you extend the record length of a file and then try reading a record into a data area of the old size, a TOOBIG error will occur.

To avoid this error in Synergy DBL programs, you can update your code to use a buffer size sufficient for either record length and then query the current record length with %ISINFO and choose which data area to use based on the results.

How to add or delete a key

Adding a key with hcutl

Adding a key is similar to extending record length. Use hcutl -ka and specify a key using the same syntax as you would with ISAMC. For example:

hcutl -ka "start=10,length=5,type=alpha,dups=noqual,keynum=3" myfile.ism

The keynum keyword is mandatory when adding a key with hcutl, to avoid accidentally adding multiple copies of the same key by re-running the same command.

If you include “dups” as part of the key specification, the duplicates must be non-qualified, e.g., “dups=noqual”. Otherwise, dups are not allowed, and hcutl will throw an error if it encounters any duplicate values for the specified key.

It may take a while for hcutl to finish building the index for the new key, and the key will not be usable for any read operations until the index is finished.

Deleting a key with hcutl

Deleting a key uses the same syntax as adding a key, but with hcutl -kd:

hcutl -kd "start=10,length=5,keynum=3" myfile.ism

To avoid accidentally deleting a key, you must specify the key number and start/length of each segment. As an extra layer of security, you’ll be prompted to confirm that you want to delete the key before it’s removed.

You can’t delete the primary key, only alternative keys. As soon as a key is deleted, it becomes unusable for any file I/O operations.

Adding or deleting a key with isutl

As a convenience, we also updated isutl to allow adding or deleting keys without converting the entire file. The syntax and procedure is virtually the same as with hcutl. For example:

isutl -ka "start=10,length=5,type=alpha,dups=noqual,keynum=3" myfile.ism

or

isutl -kd "start=10,length=5,keynum=3" myfile.ism

The isutl approach works for both HC and non-HC files. However, isutl requires exclusive access to the file while adding or deleting the key and building the index. If a file is HC, isutl will inform you that a key could be added or deleted with hcutl without requiring exclusive access and will ask if you want to continue with isutl.

Considerations and limitations

When considering whether to adopt high-cooperative files, here are some things to keep in mind. First, any updates to files using hcutl will only apply to the files themselves. For instance, if you add a key to a file, you may also need to update your repository, source code, and xfODBC catalogs to use the new key (depending on how your application is set up). You’ll need to update your application after the file header has been updated, although you don’t necessarily have to wait until the operation is finished. Newly stored records will have index entries for the new key, but an application won’t be able to read using the new key until the index is done being constructed by the high-cooperative operation (and it won’t know about the new key unless the application itself has been updated).

Similarly, before extending a record, you may want to update your DBL application so it queries the record size with %ISINFO and decides what to do based on the size. That way, your application can behave appropriately both before and after the length is increased, rather than throwing an error when attempting to access a record of the wrong size. Take care that you don’t try to read using a deleted key, as this could cause an error.

In addition, here are some limitations to consider regarding high-cooperative files:

• As a feature of SDMS, HC ISAM files can’t be used on OpenVMS, which uses native RMS ISAM.
• HC files use an ISAM compat level of 6.5 and can only be accessed by Synergy products that support that compat level. Thus, all Synergy products should have a minimum version of 12.4.1.1005 to use HC ISAM features. The minimum LTS version will be 12.5.
• To patch an existing ISAM file to HC, it must be a full Rev 6 file to begin with.
• HC files use fixed-length records and are incompatible with variable-length or multiple fixed-length records.
• HC files are incompatible with change tracking.
• HC files always use compressed records. If an existing file that doesn’t use compression is patched to HC, the records will be compressed during the conversion process.
• If you add a key to an HC file, that new key must use non-qualified duplicates or disallow duplicates altogether. DUPLICATE_ORDER is ignored.
• Adding an autokey (type SEQUENCE, TIMESTAMP, or CTIMESTAMP) is a destructive process that overwrites any data contained in the specified location in the records. These keys should only be added to blank areas of the record.
• On a big-endian operating system (e.g., AIX), a new key with type integer or unsigned (or a segmented key with a segment of those types) can only be added in a section already designated as a nonkey integer field (PORT_INT).
• You can’t delete or replace the primary key in an HC file, only alternative keys.
• While all HC operations are by definition high-cooperative (including initial conversion), they do require an HC file to begin with. So, if you have an existing ISAM file, isutl briefly needs exclusive access to the file to patch it to HC.
• Only one HC operation is allowed at a time, and it must finish successfully before another one begins.

Conclusion

I hope this article has demonstrated the potential of these new ISAM features and given you some idea of how you can take advantage of them to improve your applications without needing lengthy downtime. While there are some limitations in what these new features can do, they have the potential to make certain operations a lot easier. We hope you’re as excited to try these new ISAM features as we are to present them.