For several years now, Synergex has maintained an open-source example that provides an example of how to implement the replication of a Synergy applications data to a Microsoft SQL Server database, in near-to-real-time. The example environment makes considerable use of CodeGen to generate the bulk of the code needed to implement the interaction with the SQL Server database, and much of the remaining required code can be used out-of-the-box, requiring very little, if any change to the original Synergy application to enable the data replication to take place. The example environment has been used as a template by many customers, and our Professional Services team has assisted many others by delivering either proof of concept examples, or full-scale implementations.
As technologies and product capabilities evolve, we periodically revisit the code to ensure that it is taking advantage of the latest features and adhering to best practices. Good performance is also of critical importance in products like this, so we frequently revisit the code looking for opportunities to make improvements in throughput.
We have just completed the latest review of the code, and on this occasion, we did make some changes, which are briefly described below.
- We now generate an additional function that returns the key number of the first unique key for each ISAM structure. This allows us to avoid the need for code that previously detected the first unique key number at runtime; that code required that the replicator had an open channel to each data file being replicated.
- We also generate an additional function that, when passed a record containing data, returns the key value of the first unique key. Previously, the code used the Synergy routine %KEYVAL for this purpose, but it also requires that the replicator has an open channel to every data file replicated.
- Because of the previous two changes, we were able to remove the replicator’s requirement to open the underlying data files that are being replicated. The only files that the replicator now opens are the instruction queue file and log file.
- We added code to make the replicator more resilient to interruptions to network connections when using xfServer to access the instruction queue file on a remote system. If a network problem is detected, the replicator now closes the instruction queue file and then attempts to re-open it on a new channel. If this operation fails, it will retry several times with a delay between attempts. The number of retries and the delay between retries are both configurable via command-line options or environment variables.
If you already have a SQL Replication environment based on our sample environment, then you might consider checking out the latest version and applying the changes to your own codebase, and if you’d like some help with that, then our Professional Services team will be happy to assist. And if you haven’t yet implemented a SQL Replication environment but are interested in doing so, get in touch with your Synergex account rep and ask them to set up a demo.
In recent weeks we have been working on making improvements to our SQL Replication environment, which demonstrates how to easily replicate your Synergy data to a SQL Server database. Some of these changes were undertaken in collaboration with a customer that uses the environment extensively, and we thank them for their input and support. Other changes have been on our road-map for some time and we think you’ll be excited about what has been achieved. Here’s a summary of the changes:
- Synergy 11 introduced a new SQL Connection API capability which calls the ODBC API function SQLDescribeParam behind the scenes to improve performance and avoid excessive cache memory thrashing for SQL statements that have I/O parameters when accessing SQL Server (VTX12_SQLNATIVE). Synergex recommend setting SSQL_PERFORMANCE_SQL to yes (or setting the SQLPERFORMANCESQL environment variable. We have updated the environment to do this, which should result in improved performance when running in Synergy 11 or higher environments.
- We have added the ability to run multiple instances of the replicator process side-by-side, within a single data set, and to divide up the replication of different files between these multiple replicator instances. Each instance is assigned a unique instance name and has its own associated instruction queue file, the name of which includes the instance name, as does the log file produced by each instance. In a multi-instance environment developers can chose on a file-by-file bases which data files are replicated via which queue file, and therefor via which replicator instance. It is important to understand that in this scenario there is no synchronization of the sequence in which changes are applied to the underlying SQL database between the multiple instances of the replicator.
- We have added the ability to exclude certain fields in a record from being appearing in and being replicated to the associated SQL Server database table. It is important that fields associated with keys not be excluded, unless those keys are also excluded (see the next item). Generally the fewer fields/columns that are replicated the faster the replication will be.
- We have added the ability to exclude certain keys from processing, so that matching database indexes will not be created in the associated database tables. Generally the fewer indexes that exist on a table the faster the replication will be.
- We have added the ability for the replicator to detect some kinds of database failure, caused by network errors, or the database being shut down or otherwise unavailable, and allow it to gracefully disconnect, and then attempt to reconnect, with a configurable number of retries and retry interval. This should make the replicator processes more robust.
- We have added the ability to change the database datatypes of fields. The main use case envisaged is transforming decimal fields into implied decimal fields (adding decimal places in the database data), but there may be other use cases, such as transforming Y/N fields into Boolean true/false fields, etc.
- We have also corrected an issue that could occur if replicator encountered an unexpected error and was configured to use BATCH COMMIT mode. Is some circumstances, if there were uncommitted changes at the time on an error any uncommitted changes could be lost because the change records were deleted from the instruction queue one by one before the batch was committed. Now any instructions related to uncommitted changes remain in the queue file until the batch is committed, and are then all deleted. When the replicator is restarted it will then attempt to reply the changes to the database again. This should help prevent the database getting out of sync if there is a replicator failure.
We’re excited about these new features in our SQL Replication offering and looking forward to hearing your feedback. By the way, many of these new features rely on using the latest and greatest version of CodeGen (5.4.8) which was also released today. If you’re already using an earlier version of the SQL Replication environment, or if you are interested in getting started and would like some assistance, don’t hesitate to contact us through your Synergex Account Representative.
Some time ago a customer posted in the Synergy/DE Ideas forum suggesting that a Synergy DBL language plug-in should be created for the increasingly popular Visual Studio Code editor. Over course of the next few months the post received a significant number of votes, and we also started hearing the same request from others.
Later we heard from the original customer that they had started to define some basic language integration themselves, offered to share that initial code, and agreed that it could be open sourced.
At this time we are pleased to announce that initial support for DBL in VS Code is live! We have developed this functionality as an open source project on GitHub (https://github.com/Synergex/vscode-synergydbl) and welcome your feedback and your contributions.
If you just want to try the extension you can search for “Synergy” in the extension marketplace inside of VS Code, or download and install the VSIX from the GitHub Releases page. If you do try the extension out, and have questions or problems, please use the GitHub Issues page to let us know.
The initial scope of the extension is fairly limited, but we have been using the colorization locally for a while now and it makes a huge difference when you just need to do a quick edit to a source file. We are looking forward to seeing what everyone else has to contribute.