By Steve Ives, Posted on August 26, 2019 at 9:58 pm
For some time CodeGen has shipped with a utility called CreateFile that (given an appropriately configured repository) could be used to create an ISAM file based on a repository FILE or STRUCTURE definition.
A customer recently asked me to create a mechanism that they could use to create multiple files, load data into those files, and also support creating both RELATIVE and ISAM files. Rather than writing something completely new I decided to extend the CreateFile utility, and that new feature has now been released along with the latest version of CodeGen (5.4.2).
Briefly, there is a new command line option -in that allows you to specify an input file containing the details of one or more files to be created and and populated. The file uses a JSON format. If you’re interested in finding out more you can find additional information in the CodeGen documentation.
It’s been a while since a GodeGen release was announced on this BLOG, but rest assured they have been coming thick and fast; in fact there have been 9 interim releases since we last announced here.
So why all the frenetic activity in CodeGen land? Well the primary driving force for so many releases and so many new features is to provide support for the ever evolving Harmony Core RESTful Web Services framework. If you’re not familiar with Harmony Core then you should check it out. More and more Synergy developers are starting to take advantage of it as a way to expose their applications data and business logic in whole new ways and to support a myriad of use cases.
So what’s new in CodeGen 5.4.0? Well the answer to that question is … a LOT! I won’t list all the new features here but if you want to check out the specifics then you can head over to the CodeGen Releases page where you’ll find a detailed change log for this and previous versions. And if you need more information than that then check out the CodeGen Documentation.
We recommend that all developers who are using CodeGen upgrade to this latest version, but particularly if you’re developing with Harmony Core.
In the 1984 movie classic Ghostbusters, we are introduced to Bill Murray’s character Dr. Peter Venkman, a professor of paranormal studies, testing subjects for the gift of clairvoyance—the ability to gain information about an object, person, location, or event through extrasensory perception. While it is clear Dr. Venkman does not take such things very seriously, we can see the advantage of such an ability, particularly for developers.
Stop me if you’ve heard this before: “We upgraded X and now Y is happening.” In this case, Y is usually associated with a negative behavior like slow performance, a slew of errors, or a bad user experience. Events like these may induce weariness, nausea, dry mouth, and other various side effects that are usually listed in American pharmaceutical commercials. In summary, they’re unpleasant. If only there were some means to predict these events and avoid them in the future…
Unfortunately, most developers are not gifted with the power of clairvoyance to anticipate problems with upgrades before they happen. But maybe instead of a crystal ball, there are tools that can help us avoid upgrade failure. Let’s look at some things that can help anticipate and/or prevent such issues from happening.
A relatively recent addition to the Synergy documentation is the Configuring for performance and resiliency topic in the Installation Configuration Guide. This topic discusses things that one should take into consideration when running Synergy on any system, and it’s based on years of experience from the Synergex Development staff and the results of their testing. If you haven’t read this section yet, I highly recommend doing so. If you’ve already read it, I recommend a quick refresher the next time you’re looking at major system or software changes.
In Support, we often see issues when developers virtualize systems that run Synergy or when data is migrated and then accessed across a network rather than being stored locally. Both scenarios are discussed in this topic. And as part of Synergy’s web-based documentation set, it’s updated regularly with the latest information. Make sure you take a look before any major upgrade in case there are changes.
Other useful tools for avoiding problems are the Synergex Blog, the Synergy migration guides, KnowledgeBase articles like Guideline for debugging performance issues, and the release notes that come with each version of Synergy. Remember that even if you are just changing the operating system and/or hardware and not your version of Synergy, you should re-review these materials. Some of the considerations they outline may now be relevant, even if they didn’t affect you previously. Also, when testing, remember to take load testing or a process running over time into account. We commonly see pitfalls when developers neglect these two factors in testing.
Now let’s say that despite your excellent planning, you do see a performance issue. What can you do? Here are some steps I’ve found helpful that might get overlooked. Most have to do with simply eliminating factors that affect performance.
Establish a baseline
If you’re going to diagnose a problem in performance, the first thing to do is isolate code or create a piece of code that demonstrates the problem. Make this your baseline test for all of the various configurations you’re going to test. This will make your tests consistent as well as eliminate code changes as a factor.
Use a metric
Establish a program you’re going to use to measure the difference in performance. If you’re using a traditional Synergy program as your baseline, you can use the Synergy DBL Profiler, which will count CPU time for you. Just make sure you pick the same metric for your testing—CPU time is not the same as real time. This step will enable you to get measurable results to test what is actually making a difference.
One by one, test all the things
I’ve found that the easiest way to plan and visualize testing is to make a tree. Each layer is one aspect you’re testing that continues to branch with every different aspect. For example, I had a situation where a production machine migrated Synergy version, operating system, and hardware and virtualized the OS, all in one move. We picked one thing to change (the virtualization of the OS) and tested it.
By doing this, we established that virtualization was a factor, because a virtualized environment was slower than a non-virtualized one. We then compared those to the old and new Windows versions, but continued with virtualized and non-virtualized environments using the same virtualization software.
In previous table
In previous table
On average, this produced the same result. (It was I/O processing, so we did an average of 10-20 runs based on how volatile the results could be.) Next, we compared the Synergy 10 runtime with the Synergy 9 one.
The tree continued growing until all of the factors were considered and tested.
It can be tedious to test one change at a time, but without that kind of granularity, you can’t establish which change affected performance and by how much. In the example I mentioned above, we established that virtualizing the hardware was causing a problem because of the way the virtual machine software emulated separate cores. We never would have come to such a conclusion without carefully eliminating the many different changes one at a time.
After you’re able to establish exactly which changes caused the performance issue(s) and by how much, you can work on a fix or provide a solid case to whichever software support representative you need to contact to get a fix.
You might know most of this already. You might even know of some methods, tips, etc., for performance issues that I didn’t discuss. Maybe you are clairvoyant and you already knew the contents of this post before I did. Either way, I hope you find this information helpful when you look at performance in the future, in both preventative measures and problem diagnosis.
If you have large (and I mean LARGE) blocks of code in single source files – and by large I mean 20k lines plus – then you may be having compiler issue with “SEGBIG” errors: “Segment too big”. This issue arises because your code segment is just too big for the compiler to handle and is usually the result of many man years of development to a single routine that has just grown over time.
If you encounter SEGBIG issues, as a customer I have recently worked with did, then this quick blog will give you some practical ideas of how to manage the issue and modify the code to allow for future development and expansion, without having to rewrite the world.
First off, it’s not the physical number of lines of code in the source file that’s the issue, it’s the lines of code and data definitions within each routine block: subroutine or function. Developers have encountered the problem for many years and the resolution has previously been to chop out a section of code, make it into a subroutine or function, and somehow pass all the appropriate data to it – usually by large numbers of arguments or common/global data blocks.
The “today” way is not too dissimilar but is a little more refined: turn the code block into a class. The first major advantage is class-based data. This removes the need to create subroutines or functions that accept large numbers of arguments, or to create large common or global data blocks. As an example:
Obviously this code will not give us a SEGBIG issue, but its an example of the structure of the code. The routine has a common data include and private data. In the routine body we make multiple local label calls. When there is too much data and too many lines of code added we will encounter a SEGBIG error.
So to address this, in the same source file, we can create a class with class-level data (the routine level data) and methods for the local call labels. So, for example:
public class BigRoutineClass
private record localData
public method Execute, void
method doSomeLogic, void
method doOtherLogic, void
In this code, the Execute method becomes the entry point. All the existing code that made the label calls is moved into this routine and the calls changed to method invocations;
Then we can change the existing BigRoutine code;
routineInstance = new BigRoutineClass()
Although the code changes I’ve described here sound monumental, if you use Visual Studio to develop your Traditional Synergy code the process is actually quite simple. Once you have created the scaffolding routine and defined the base class with class level data (which really is a case of cutting and pasting the data definition code), there are a few simple regex commands we can use that will basically do the work for us.
To change all the call references to class method invocations you can use:
Find: ([\t ]+)(call )([\w\d]+)
To change the actual labels into class methods, simply use the following regex:
Find: ^([\t ]+)([a-zA-z0-9_]+)[,]
Replace: $1endmethod\n$1method $2, void\n$1proc
And to change the return statements to method returns, use:
These simple steps will allow you to take your large code routines and make manageable classes from them which can be extended as required.
If you have any questions or would like assistance in addressing your SEGBIG issues, please let me know.
If you’re a Porsche enthusiast, you probably know about the Nürburgring record set a few months back by a 911 GT2 RS, the latest iteration of the Porsche 911 (see the Wikipedia article). Like many, I find it interesting that the best* production sports car in the world isn’t a new design, but the result of continuous improvement and development since its introduction over 50 years ago. One company, Singer Vehicle Design, takes old 911s and resurrects them as carbon fiber kinetic sculptures in the aesthetic of the older 911s, but with performance that matches some of the fastest new 911s from the factory. They describe these as “Reimagined,” and you can see a video about a Singer 911 or visit the Singer website for more information.
Here in the Documentation department at Synergex, we’ve been doing some reimagining of our own. The Synergy/DE documentation has been continually improved over the years, but since 10.3.3c, we’ve published the Synergy/DE documentation with a new format, a new look, and a new set of underlying technologies and practices. The goal: documentation that is quickly updated to reflect product changes and user input, that is increasingly optimized for online viewing, and that is increasingly integrated with other Synergy/DE content. (And soon it will be better integrated with Visual Studio as well; see a recent Ideas post for details.) You can access the doc just about anywhere (even without internet access), it offers better viewing on a range of screen sizes, and it’s poised for further improvements. If you haven’t seen our new “reimagined” doc, check it out at synergex.com/docs.
Here are some of the highlights:
A better, more responsive UI for an improved experience when viewing documentation from a desktop system, a laptop, or a tablet.
Technologies that facilitate more frequent updates and allow us to increasingly optimize content for online use.
Improved navigation features. The Contents tab and search field adapt to small screens, the UI includes buttons that enable you to browse through topics, and a Synergy Errors tab makes it easy to locate error documentation.
Quick access to URLs for subheadings. To get the URL for a topic or subheading (so you can share it or save it for later), just right-click the chain-link icon next to a heading and copy the link address.
The ability to print the current topic (without printing the Contents tab, search field, etc.) and to remove highlighting that’s added to topics when you use the search feature.
Local (offline) access. If you’re going somewhere where internet access is limited, download and install the Local Synergy/DE Product Documentation, which is available on the Windows downloads page for 10.3.3.
See the Quick Tips YouTube video for a brief visual tour of the documentation, and let us know what you think. In the footer for every documentation topic, there is a “Comment on this page” link you can use to send us your input. We look forward to hearing from you!
For a recent project I’ve worked with a customer who wished to provide their users with an engaging desktop application that would allow management of product formulations. They had a Synergy UI Toolkit version and also elements of the required application in a third-party system. However, neither met the needs of the users. After a review and discussions about their requirements we agreed on a Synergy .NET Windows Presentation Foundation based application using Infragistics tooling for the User Experience.
The basic requirements of the application where to allow the creation and maintenance of formulations. A formulation contains the components required to make a finished product. For this customer the final product is an aerosol.
Let’s take a look
The basic interface is built using the Infragistics control to handle navigation (Ribbon menu control), listing and selection of data (powerful DataGrid), hierarchical representation of the formulation components (TreeView) and management of finished product details (Property Grid);
Of course, using the Infragistics DockManager allows the user to drag and reposition all the available windows to their liking.
There are powerful searching facilities, or QBE (Query By Example) controls. These allow the user to provide snippets of information and the application will query the Synergy DBMS database using Symphony Harmony and the Synergex.SynergyDE.Select class;
The top line of the QBE controls allow the user to enter the data in the columns they wish to search for and so only select the data they require and not have to filter through a list of thousands of formulations.
Because the application is written in Synergy, the existing printing capabilities from the original UI Toolkit application have been retained without change;
The whole application is written in Synergy .NET and utilises the Symphony Framework for controlling the data access and presentation. If you would like more details, or would like to know how you can build modern applications with Synergy .NET please drop me an email.
By Steve Ives, Posted on December 1, 2017 at 11:18 pm
We are pleased to announce the release of CodeGen version 5.2.3. The new release includes new features, addresses some issues found with previous releases, and also paves the way for once again being able to use CodeGen on non-Windows platforms through experimental support for the .NET Core environment.
As always, you can download the latest version of CodeGen from here.
CodeGen Version 5.2.3 Release Notes
Added a new experimental utility to the distribution. The Code Converter utility can be used to automate bulk searches within and bulk edits to an applications code. This utility is in a usable form. However, it is still a work in progress and is likely to undergo substantial changes as it evolves.
Added two new utility routines (IsDate.dbl and IsTime.dbl) that are referenced by some of the supplied sample template files.
Corrected a regression that was introduced in the previous release which caused the field loop expansion token <FIELD_SQL_ALTNAME> not to default to using the actual field name if no alternate name was present.
Performed an extensive code review and cleanup, updating the code in several areas to take advantage of new features available in the Synergy compiler, and also improving efficiency.
Fixed an issue that was causing the CreateFile utility -r (replace file) option to fail, an existing file would not be replaced even if the -r option was specified.
Fixed an issue in the CreateFile utility that would result in an unhanded exception in the event that invalid key information was passed to XCALL ISAMC.
Made some minor code changes to allow CodeGen to be built in a .NET Core environment and we hope to be able to leverage .NET Core to once again support the use of CodeGen on non-Windows systems (starting with Linux) in the near future.
This version of CodeGen was built with Synergy/DE 10.3.3d and requires a minimum Synergy version of 10.1.1 to operate.
Symphony Framework Components
We no longer ship the Symphony Framework sample templates with CodeGen. You can obtain the latest Symphony Framework templates from the Symphony Framework web site (http://www.symphonyframework.net).
There were no Symphony Orchestrator changes in this release.
There were no Symphony Framework CodeGen Extensions changes in this release.
By Steve Ives, Posted on October 25, 2017 at 3:00 pm
We are delighted to announce the availability of the CodeGen version 5.2.2 release that includes the following enhancements and changes:
CodeGen Version 5.2.2 Release Notes
Added a new field loop expansion token <FIELD_FORMATSTRING> which can be used to access a fields format string value.
Added a new command-line option -utpp which instructs CodeGen to treat user-defined tokens as pre-processor tokens. This means that user-defined tokens are expanded much earlier during the initial tokenization phase, which in turn means that other expansion tokens may be embedded within the values of user-defined tokens.
Removed the RpsBrowser utility from the distribution; it was an experimental project that was never completed.
This version of CodeGen was built with Synergy/DE 10.3.3d and requires a minimum Synergy version of 10.1.1 in order to operate.
Most Synergy developers would love to use the very latest and greatest development tools to develop and maintain their Synergy applications, but how do you get started? At the recent DevPartner conference in Atlanta product manager Marty Lewis not only discussed the concepts of how to get started, but actually demonstrated the entire process with a real Synergy application. Check out his presentation entitled Developing Synergy Code in Visual Studio:
I am pleased to announce that we have just released a new version of CodeGen (5.1.9) that contains some new features that were requested by customers. The changes in this release are as follows:
We added two new structure expansion tokens <FILE_ODBC_NAME> and <FILE_RPS_NAME> that expands to the repository ODBC table name or structure name of the first file definition that is assigned to the structure being processed.
We made a slight change to the way that the multiple structures command line option (-ms) is processed, allowing it to be used when only one repository structure is specified. This allows for templates that use the <STRUCTURE_LOOP> construct to be used when only one structure is being processed.
We also fixed an issue that was causing the <FIELD_SPEC> token to produce incorrect values for auto-sequence and auto-timestamp fields. Previously the value 8 would be inserted, now the correct value i8 is inserted.
This version of CodeGen was built with Synergy/DE 10.3.3c and requires a minimum Synergy runtime version of 10.1.1. You can download the new version directly from the CodeGen Github Repository.
By Steve Ives, Posted on February 7, 2017 at 10:25 am
We are pleased to announce that Professional Services has just released CodeGen 5.1.7. The main feature of the release is the addition of experimental support for generating code for the MySQL and PostgreSQL relational databases. Developers can use a new command line option -database to specify their database of choice. This causes the SQL-compatible data types that are injected by the field loop expansion token <FIELD_SQLTYPE> to be customized based on the chosen database. The default database continues to be Microsoft SQL Server.
Before we consider support for these new databases to be final we would appreciate any feedback from developers working with MySQL or PostgreSQL to confirm whether we have chosen appropriate data type mappings. Additional information can be found in the CodeGen documentation.
By Steve Ives, Posted on July 29, 2016 at 10:42 am
We are pleased to announce that we have just released CodeGen V5.1.4. The main change in this version is an alteration to the way that CodeGen maps Synergy time fields, i.e. TM4 (HHMM) and TM6 (HHMMSS) fields, to corresponding SQL data types via the <FIELD_SQLTYPE> field loop expansion token. Previously these fields would be mapped to DECIMAL(4) and DECIMAL(6) fields, resulting in time fields being exposed as simple integer values in an underlying database. With this change it is now possible to correctly export time data to relational databases.
We also made a small change to the CodeGen installation so that the changes that it makes to the system PATH environment variable occur immediately after the installation completes, meaning that it is no longer necessary to reboot the system after installing CodeGen on a system for the first time.
This version of CodeGen is built with Synergy/DE 10.3.3a and requires a minimum Synergy runtime version of 10.1.1.
When I introduced the PSG PDF API during the recent DevPartner Conference in Washington DC I received several questions about whether it was possible to define the layout of something like a standard form using one PDF file, and then simply merge in data in order to create another PDF file. I also received some suggestions about how this might be done, and I am pleased to report that one of those suggestions panned out into a workable solution, at least on the Windows platform.
The solution involves the use of a third-party product named PDFtk Pro. The bad news is that this one isn’t open source and neither is it free. But the good news is it only costs US$ 3.99, which I figured wouldn’t be a problem if you need the functionality that it provides.
Once you have PDFtk Pro installed and in your PATH you can then call the new SetBackgroundFile method on your PdfFile object, specifying the name of the existing PDF file to use as the page background for the pages in the PDF file that you are currently creating. All that actually happens is when you subsequently save your PDF file, by calling one of the Print, Preview or Save methods, the code executes a PDFtk Pro command that merges your PDF file with the background file that you specified earlier. Here’s an example of what the code looks like:
;;Create an instance of the PdfFile class pdf = new PdfFile()
;;Name the other PDF file that defines page background content if (!pdf.SetBackgroundFile(“FORMS:DeliveryTicketForm.pdf”,errorMessage) throw new Exception(errorMessage)
;;Other code to define the content of the PDF file
;;Show the results pdf.Preview()
There are several possible benefits of using this approach, not least of which is the potential for a significant reduction in processing overhead when creating complex forms. Another tangible benefit will be the ability to create background forms and other documents using any Windows application that can create print output; Microsoft Word or Excel for example. Remember that in Windows 10 Microsoft has included the “Print to PDF” option, so now any Windows application that can create print output can be used to create PDF background documents.
I have re-worked the existing Delivery Ticket example that is distributed with the PDF API so that it first creates a “form” in one PDF file, then creates a second PDF file containing an actual delivery ticket with data, using the form created earlier as a page background.
I have just checked the code changes into the GitHub repository so this new feature is available for use right away, and I am looking forward to receiving any feedback that you may have. I will of course continue to research possible ways of doing this on the other platforms (Unix, Linux and OpenVMS) but for now at least we have a solution for the Windows platform that most of us are using.
The Symphony Harmony namespace allows access to data and logic through an SQL like syntax. For example you can select records from a file using a query such as “SELECT ID, DESCRIPTION FROM PART WHERE QUANTITY > 12 ORDER BY QUANTITY”. All matching records are returned from the query in the form of Symphony Data Objects. The data can be local or accessed via Synergy xfServer. The Symphony Bridge utility allows you to expose your query-able database via a standard Windows Communication Foundation (WCF) web service. So far so good.
Steve Ives and I recently had the opportunity to spend a week working together in the UK to “bang” heads together. Steve has always been an exponent of providing RESTful services to access logic and data which can be consumed by just about anything. So we set about using CodeGen to build a standard Restful service that will utilize the Symphony Framework to enable dynamic access to data and ultimately logic.
We soon had the basic service up and running. Out first implementation handled the standard GET verb – and returned all the records in the file. No filtering, no selection, just all the records retuned as a JSON collection. This is the standard API;
Now remember that Symphony Harmony allows you to filter the data you are requesting, so we next implemented the ability to provide the “where” clause to the query. So for example;
And using ARC (Advanced Rest Client which is a Google Chrome plug-in) we can test and query the service;
And we get back just the selected customer details – all those customers where CUSTST has value of CA.
As well as being able to filter the data we can also limit the results returned by Harmony to just the fields we need; this has the benefit of reducing the data being brought across the wire. But how can our REST server build the required data objects to just include the fields we select? By doing runtime code generation! Within our code generated data objects we added the ability to dynamically build the response data object to only include those fields requested by the client. The calling syntax, as provided by the API, is;
And again using ARC to test our server we can issue a command like;
This is requesting all records from CUSMAS where the CUSNM2 field contains the word “LAWN” and limiting the response data object to just three fields. The response JSON looks like;
Two perfectly formed data object that are limited by the fields in the selection list. If your Symphony Harmony connection to your data is via xfServer then only those selected fields will have been loaded from the file, again improving performance.
We also added the ability to limit the amount of data retuned by adding a “maxrows” option;
We have already added the ability to the Symphony Harmony namespace to perform inserts, updates and deletes using standard SQL syntax and we’ll be adding these capabilities to the appropriate rest verbs POST, PUT and DELETE. Watch this blog feed for more information.
Tomorrow morning I’m heading back home to California having spent the last two weeks in the United Kingdom. The second week was totally chill time; I spent time with family and caught up with some old friends. But the first week was all about work; I spent a few days working with Richard Morris (it’s been WAY too long since that happened) and I can tell you that we worked on some pretty cool stuff. I’m not going to tell you what that is right now, but It’s something that many of you may be able to leverage in the not too distant future, and you’ll be able to read all about it in the coming weeks. For now I wanted to let you know that we found that we needed to add some new features to CodeGen to achieve what we were trying to do, so I am happy to announce that CodeGen 5.1.3 is now available for download.