Phone800.366.3472 SupportGet Support DocumentationDocumentation Resource CenterResource Center
search
close
Open Menu

Synergex Blog


Synergy DBL and Visual Studio Code

By Steve Ives, Posted on October 10, 2019 at 2:52 pm

Steve Ives

Some time ago a customer posted in the Synergy/DE Ideas forum suggesting that a Synergy DBL language plug-in should be created for the increasingly popular Visual Studio Code editor. Over course of the next few months the post received a significant number of votes, and we also started hearing the same request from others.

Later we heard from the original customer that they had started to define some basic language integration themselves, offered to share that initial code, and agreed that it could be open sourced.

At this time we are pleased to announce that initial support for DBL in VS Code is live! We have developed this functionality as an open source project on GitHub (https://github.com/Synergex/vscode-synergydbl) and welcome your feedback and your contributions.

If you just want to try the extension you can search for “Synergy” in the extension marketplace inside of VS Code, or download and install the VSIX from the GitHub Releases page. If you do try the extension out, and have questions or problems, please use the GitHub Issues page to let us know.

The initial scope of the extension is fairly limited, but we have been using the colorization locally for a while now and it makes a huge difference when you just need to do a quick edit to a source file. We are looking forward to seeing what everyone else has to contribute.


Synergy/DE 11 and REV11 licensing are released

By Roger Andrews, Posted on October 1, 2019 at 2:51 pm

Avatar

We’re very pleased to announce the release of Synergy/DE 11. Version 11 delivers many features, including important security updates, exciting ISAM resilience features, and runtime performance improvements. It also includes new Synergy DBL and Visual Studio integration features. See our Synergy/DE 11 page for a full list of features.

Synergy/DE 11 also includes our new REV11 licensing, which automates your product key updates. REV11 licensing is used with Synergy/DE 11, and it also supports Synergy/DE 9.3 – 10.3 with a licensing upgrade package. See our REV11 licensing page for more information.

Many of the new features in Synergy/DE 11 were suggested on our Ideas site. We would like to thank everyone for their input, and we encourage you to continue to post, comment, and vote on that site to let us know what features are most important to you.

Be sure to review the new requirements before you upgrade to Synergy/DE 11. For example, version 11 doesn’t support Windows 7, Windows Server 2008 R2, or Visual Studio 2015.

We strongly encourage you to upgrade to Synergy/DE 11 for enhanced application security, resilience, and performance; a better developer experience; and a much easier process for managing product keys. Take a look, and let us know if you have any questions or if you need any assistance.


CodeGen 5.4.3 Released

By Steve Ives, Posted on September 22, 2019 at 1:01 pm

Steve Ives

I am pleased to announce that another new version of CodeGen has been released, is available for immediate use, and contains several powerful new features:

  • We added a new structure expression token <IF STRUCTURE_PII_FIELDS> that indicates whether any field in a structure has been tagged as containing PII (Personally Identifiable Information). We also added a new field loop expression token <IF PII_SCRUB> that indicates whether a field has been tagged as containing PII data, and a new field loop expansion token <FIELD_PII_TYPE> which inserts the type of PII data a field contains.
  • We added a new field loop expression token <IF USED_IN_RELATION> that can be used to determine whether the field currently being processed is involved in any relations to other structures.
  • We added a new custom field loop expression token <IF HARMONYCORE_CUSTOM_FIELD> and a new custom field loop expansion token <HARMONYCORE_CUSTOM_FIELD_TYPE> to the Harmony Core extensions assembly. In a future release Harmony Core will use these new features to allow developers to easily implement support for custom field types that may be implemented within their applications data set.
  • We added a new tag loop expansion token <TAGLOOP_FIELD_SNTYPE> and relation key segment loop expansion token <LITERAL_SEGMENT_SNTYPE>.

Additionally, because the Symphony Framework is no longer maintained by Synergex, and is no longer an open source development effort, we have decided to discontinue the distribution of the “Symphony Framework CodeGen Extensions” assembly that was formerly distributed along with CodeGen. We continue to support the addition of custom extensions and the Symphony Framework extensions can be added back at any time by any Synergy developer with the necessary Synergy .NET development tools.

This version of CodeGen was built with Synergy/DE 11.1.1 and requires a minimum of version 10.1.1 to operate.



Harmony Core Project Update, September 2019

By Harmony Core Team, Posted on September 16, 2019 at 4:17 pm

Avatar

The Harmony Core project has been steadily gaining momentum as we improve our processes, work on support issues, and continue with user engagements.

To start with, we’ve been working on our continuous integration and continuous delivery (CI/CD) pipeline in Azure DevOps. This pipeline serves a dual purpose. It allows us to demonstrate software development best practices applicable to Harmony Core users, and it enables our team to be more productive. Whenever code is committed or a pull request is created, the full test suite is run to ensure that the code is good and that there are no regressions. Automating this has improved our ability to take pull requests from outside contributors.

Significant effort has gone into offering a nearly seamless experience for xfServerPlus users who are looking to expose their Synergy routines via RESTful web services. We have updated CodeGen, our CodeGen templates, and the underlying support libraries. CodeGen can now read SMC interface definitions, and there are new tokens for iterating over xfServerPlus methods and parameters. Additionally, CodeGen now uses SMC interface definitions as input when determining which repository structures to iterate over. We use this functionality to generate the data objects required to wrap structures passed as parameters.

As part of a consulting engagement, we built a mechanism to support custom loading and custom validation of data objects and transactions (Issue 107). We enhanced CodeGen and the CodeGen templates in Harmony Core to support certain types of relations to ensure that required relations have the necessary resources in the database or the current transaction.

Traditional Bridge Improvements

Up to this point, we’ve been exclusively promoting RESTful web services, but now we’re introducing an alternative. We’ve created CodeGen templates for exposing Traditional Bridge services via SignalR, a technology provided by Microsoft. SignalR allows for two-way communication in or outside of the browser without compromising performance, reach, or standards. We’re recommending SignalR in cases where an application can’t operate statelessly or the server needs to push updates to clients rather than requiring them poll or refresh for changes.

On the xfServerPlus front, we implemented encoding in Traditional Bridge for all currently known scenarios and data types that are supported for xfServerPlus. This includes arrays, handles, ArrayLists, and scalar types. We also implemented a secondary calling convention that uses the Synergy routine call block API for those of you that use the optional parameter support offered only in xfNetLink COM clients. And we’re actively working on porting the full test suite used by xfServerPlus and xfNetLink.NET to work as unit tests for Traditional Bridge. Swagger Docs

Over the last few weeks, we’ve been working on improving our API version support. We found an open-source Microsoft library for this, and we have created integration samples. This library does two things for us: it takes care of versioning, and it produces Swagger docs for versions, enabling us to move away from statically generated Swagger files. This dramatically improves the quality of Harmony Core Swagger documentation for items that fall outside the normal set of GET and POST entities. And it has the added benefit of allowing you to expose source code doc comments as part of your API documentation.

Traditional Runtime Updates

The traditional Synergy runtime for version 11 includes support for reading and writing JSON using a subset of the new System.Text.Json namespace from .NET Core 3.0. We have decided to use this functionality because it is designed for speed. Even for moderately-sized JSON documents, we’re seeing upwards of 15x faster parsing, and the number gets larger for larger documents. In the coming weeks, we’ll be implementing a dual pathway for JSON reading and writing in Traditional Bridge. If you’re on a runtime older than 11, we’ll fall back to the implementation written in Synergy DBL. But if you’re using the latest runtime, you’ll be able to take advantage of the massive performance increase.

A New Harmony Core Example

We’ve received some requests for a complete end-to-end Harmony Core example that demonstrates how to develop, test, and deploy a web application backed by a Harmony Core web service. We’ve come up with an idea for an application we’re calling xfBBQ. At our offices during the summer, the development team frequently hosts barbeques for the company, and one of the challenges we face is taking orders. This is currently managed using a Google Sheets spreadsheet, which isn’t ideal. But it gave us the idea to build an example app to improve this process. xfBBQ will be implemented as a React + Redux Single Page Application. We’re planning to pack large amounts of sometimes gratuitous functionality, but we’ll stay within the bounds of best practices.

Synergex Company BBQ

Utilizing xfBBQ for planning company BBQs

Resolving Your Harmony Core Issues

There are two things we need to know when prioritizing Harmony Core development: what issues you have and what features you need. And we can more quickly resolve your issues and implement the features you need when we know what they are. The following are the best ways to reach us.

We host online office hours every six weeks. During this time, we deliver a short update on what we have improved in the Harmony Core framework. At the end of each session, we answer questions and work through issues you may have. This is a great opportunity to get direct and live feedback from us. During our last online office hours session, we fielded some support questions that we were subsequently able to resolve. We will continue to offer these sessions every six weeks. If you have any questions for us, please sign up for the next office hours session so we can answer your questions directly.

The next best way to reach us is through GitHub issues. Here we can easily keep track of your issues and make sure they are resolved as quickly as possible. This is also helpful because it creates a repository of answers for other users to search through. Recently we were able to resolve questions regarding ISMFILE.txt and .XDL files and where to put logicals by directing the user to our wiki documentation.

If you have sensitive corporate information that you don’t want publicly available, you can send your issues to the Synergy/DE Developer Support team. You may contact them at support@synergex.com.


Enhanced CodeGen CreateFile utility

By Steve Ives, Posted on August 26, 2019 at 9:58 pm

Steve Ives

For some time CodeGen has shipped with a utility called CreateFile that (given an appropriately configured repository) could be used to create an ISAM file based on a repository FILE or STRUCTURE definition.

A customer recently asked me to create a mechanism that they could use to create multiple files, load data into those files, and also support creating both RELATIVE and ISAM files. Rather than writing something completely new I decided to extend the CreateFile utility, and that new feature has now been released along with the latest version of CodeGen (5.4.2).

Briefly, there is a new command line option -in that allows you to specify an input file containing the details of one or more files to be created and and populated. The file uses a JSON format. If you’re interested in finding out more you can find additional information in the CodeGen documentation.


New string features coming with Synergy/DE 11

By Jerry Fawcett, Posted on August 19, 2019 at 8:58 am

Avatar

Synergy/DE 11 is bringing many new features, and among those additions are some exciting new abilities within the String class. All of these powerful new features are available in both traditional and Synergy .NET.

When dealing with strings, keep in mind that a string is immutable (fancy word for unchangeable), so any time a string variable is changed, a new string variable is created.

First, let me list the new methods added, along with existing methods that include new overloads to the String class: 

String.IsNullorEmpty(@string) , boolean
  • Parameter: String to interrogate for null or empty value
  • Return value: Boolean value indicating whether the string is null or empty
String.IsNullorWhitespace(@string) , boolean
  • Parameter: String to interrogate for null or white-space characters
  • Return value: Boolean value indicating whether the string contains null or white-space characters

New overloads to give precise control on how a string is trimmed:

String.Trim() ,string			(exists prior to version 11)
  • Parameter: None
  • Return value: String with leading and trailing white-space characters removed
String.Trim(@string) ,string 
  • Parameter: Character(s) to be removed from the string’s beginning and end
  • Return value: Returned string with passed-in character(s) removed from the beginning and end
String.TrimStart(@string) ,string
String.TrimEnd(@string) ,string
  • Parameter: Character(s) to remove from either the string’s start or end
  • Return value: String with all instances of character(s) removed from start or end

New methods to control how characters are removed or inserted into a string:

String.Remove(N) ,string
  • Parameter: Starting position in the string to remove all characters until its end
  • Return value: String with all characters from passed starting position to end removed
String.Remove(N) ,string
  • Parameter: Starting position in the string to remove all characters until its end
  • Return value: String with all characters from passed starting position to end removed
String.Insert(N, @string) ,string
  • Parameters:
    1. Starting position to insert string passed as parameter 2
    2. String to be inserted
  • Return value: Returned string with sting passed in parameter 2 inserted

New overloads to give more control on concatenating strings together:

String.Concat(@string, @string) ,string	(exists prior to version 11)
String.Concat(@string, @string, @string) ,string
String.Concat(@string, @string, @string, @string) ,string
String.Concat([#]@string) ,string
  • Parameters:
    1. Two strings to concatenate together
    2. Three strings to concatenate together
    3. Four strings to concatenate together
    4. Dynamic string array to concatenate all elements together
  • Return value: Returned string after concatenation operation

New overloads to locate the position of a substring in a string:

String.IndexOf(A1) ,int			(exists prior to version 11)
String.IndexOf(A1, N) ,int
String.IndexOf(A1, N, N) ,int		(exists prior to version 11)
  • Parameters:
    1. Character(s) to find in string
    2. Character(s) to find in string starting at position specified in parameter 2
    3. Character(s) to find in string starting at position specified in parameter 2 for the length of parameter 3
  • Return value: Integer value with first position of passed character(s)

New overloads to find the last occurrence of a substring in a string:

String.LastIndexOf(A1) ,int		(exists prior to version 11)
String.LastIndexOf(A1,N) ,int
String.LastIndexOf(A1,N,N) ,int		(exists prior to version 11)

Parameters:

  • Parameters
    1. Numeric value of the starting position to begin returning the substring
    2. Numeric value of the length of the substring to return
  • Return value: Returned substring with value from the specified starting and length values

New methods to create an array from a delimited string:

String.Split(A1) ,[#]string
String.Split(A1, StringSplitOptions) ,[#]string (see below for the StringSplitOptions enumeration)
  • Parameters:
    1. Character(s) to use a delimiter
    2. Enumeration StringSplitOptions to control if empty elements are created
    3. Return value: Dynamic string array to be populated with the contents of the passed string

New overload to use the replace method on alphas:

String.Replace(A, A) ,string
  • Parameters:
    1. Alpha value to be replaced by parameter 2
    2. Alpha value to replace all occurrences of parameter 1 in string
  • Return value: Returned string with all occurrences of parameter 1 replaced by parameter 2
String.Replace(@string, @string) ,string 	(exists prior to version 11)
  • Parameters
    1. String value to be replaced by parameter 2
    2. String value to replace all occurrences of parameter 1 in string
  • Return value: Returned string with all occurrences of parameter 1 replaced by parameter 2

And finally, the enumeration StringSplitOptions, which is used with the Split method to control if empty elements are created when creating the string array:

public enum StringSplitOptions
    None		,0
    RemoveEmptyEntries	,1

You can find details on all String members in the System.String topic in the Synergy/DE documentation.

As I mentioned earlier, some of the above are brand new methods (Insert, IsNullorWhitespace, Remove, Split, TrimEnd, TrimStart) added to the System.String class, while others are new overload methods added to existing methods to provide additional functionality. Interestingly, the runtime itself will make use of the String.Concat method when strings are added together (e.g., string1 + string), as it is efficient and generates fewer temporary variables.

Now let’s take a look at one of the new methods. Using the new Split method, we can populate a dynamic array from a delimited string. Say we have a string containing the following:

mystring = “Bob, Mary, Joe, Jack,,,, Fred, Henry, Tom”

We’d like for the values between the commas to be elements in an array, but we don’t want the commas without values between them to become empty elements in the array. This can easily be accomplished with the String.Split method using the syntax below:

stringarray = mystring.split(“,” , StringSplitOptions.RemoveEmptyEntries)

Passing the quoted comma as the first parameter tells the Split method that a comma is the delimiter to look for in the string. Also, for even more control, the delimiter can consist of any number of characters.  There is no need to instantiate the dynamic array, as the dynamic array will automatically be instantiated by the Split method. The result of the above operation will be that the dynamic array stringarray will contain elements with the names that are separated by commas in the string mystring and no empty elements (due to passing StringSplitOptions.RemoveEmptyEntries as the second parameter) for the commas with no values between them:

stringarray[1] = “Bob”
stringarray[2] = “Mary”
stringarray[3] = “Joe”
stringarray[4] = “Jack”
stringarray[5] = “Fred”
stringarray[6] = “Henry”
stringarray[7] = “Tom”

As you can see, the enhancements to the String class are more than enough reason to upgrade to Synergy/DE 11 when it’s released. If you can’t wait and want to try these new capabilities now, the version 11 beta is available in the Resource Center for download. Don’t forget to set your Synergy/DE documentation to the “Beta” version in the version drop-down near the top of the screen.

And wait, there’s more! Now that you’re aware of the new and powerful methods added to the String class, be sure to also check out the new StringBuilder class added to version 11.



Are you protecting yourself and your customers from upcoming security threats?

By Roger Andrews, Posted on August 1, 2019 at 10:07 am

Avatar

Thanks to those of you who joined us last week for our security webinar. We talked about some of the big security threats we all face today, and how to protect ourselves from ransomware/malware, endpoint attacks, and phishing. Keeping current with security updates is one important step, and this includes operating systems, applications, firewalls, and other components. Also, if you’re using HTTPS or SSL communication, you may need to encrypt your data between computers and when writing to disk (as well as when transferring externally).

Security webinar, July 24, 2019

If you use Windows, make sure you have a plan to upgrade any Windows 7 or Server 2008R2 systems. Microsoft is ending support for these at the end of this year (and for some of their other products soon). This means there will be no more security patches. This will increase the likelihood that your systems will be compromised with ransomware. Also, if you’re using OpenSSL on Windows, you’ll need to upgrade to Synergy/DE 11 (scheduled for release later this year) as it supports the new OpenSSL requirements. Synergy/DE 11 has other security updates, including the ability for services to be configured to run as a user other than system/root.

Here’s an article from our last Synergy-e-News for additional security information. And our Synergy/DE 11 page has for more information about our upcoming release. We’ll continue to provide you with information about these security topics and others. In the meantime, please contact us if you have questions or if you would like more information.


CodeGen 5.4.0 Released

By Steve Ives, Posted on July 3, 2019 at 9:59 am

Steve Ives

It’s been a while since a GodeGen release was announced on this BLOG, but rest assured they have been coming thick and fast; in fact there have been 9 interim releases since we last announced here.

So why all the frenetic activity in CodeGen land? Well the primary driving force for so many releases and so many new features is to provide support for the ever evolving Harmony Core RESTful Web Services framework. If you’re not familiar with Harmony Core then you should check it out. More and more Synergy developers are starting to take advantage of it as a way to expose their applications data and business logic in whole new ways and to support a myriad of use cases.

So what’s new in CodeGen 5.4.0? Well the answer to that question is … a LOT! I won’t list all the new features here but if you want to check out the specifics then you can head over to the CodeGen Releases page where you’ll find a detailed change log for this and previous versions. And if you need more information than that then check out the CodeGen Documentation.

We recommend that all developers who are using CodeGen upgrade to this latest version, but particularly if you’re developing with Harmony Core.

Happy 4th Everyone!




An open letter to Synergy developer hiring managers

By Jacklin Garcia, Posted on May 2, 2019 at 12:58 pm

Jacklin Garcia

Dear Hiring Manager,

Congrats on your new Synergy developer hire! Recruiting isn’t easy, but now that you’ve found the right person for the job, the real hard work begins. The onboarding period should be used for training your new hire and setting their expectations about the job. This time is crucial, not only to ensure a smooth transition for your team, but also to make your new employee feel welcome. So where do you start?

Tell your Synergex account executive about your new hire

New Synergy developers in the DevPartner program are entitled to a welcome box full of goodies. This box contains educational materials like laminated cheat sheets as well as some fun Synergex-branded items. Don’t forget to mention the new hire’s T-shirt size when you reach out.

Download and modify the suggested 90-day plan

At Synergex, we use 90-day plans for all of our new hires. They are a key part of our onboarding strategy. We’ve created a template plan for you to download and modify from our education library. At the very least, we suggest grabbing the link for the new hire onboarding playlist on YouTube that is referenced in the plan and reviewing the other materials available in library.

Send your new hire to a Synergy DBL Language Essentials class

The best way to learn Synergy DBL is to take this class. Many developers prefer hands-on learning, and in this class, students will build a simple application in Synergy. You can either send developers to our headquarters for the standard five-day training or request a custom class at your location. If a class at HQ isn’t being advertised when you need one, that’s ok! Reach out to classes@synergex.com and we’ll see what we can do.

Have your new hire subscribe, join, or follow us on various platforms

We don’t like to bombard our customers with emails, so connecting with us on sites like YouTube and GitHub is the best way to stay in the know about new educational content. The Synergex Community is also the place for Synergy developers to get advice from their peers and make suggestions for Synergy/DE enhancements.  Bookmarking our online documentation is also a great idea!

Call support early and often

Our support team is incredible, and their services are free with your Synergy DevPartner subscription. Encourage new hires to call in (toll-free from the US and Canada 800.366.3472 or +1.916.635.7300 for all other countries) or email (support@synergex.com) with any Synergy-related problems they may encounter. Our support team is made up of engineers who are more than capable of troubleshooting issues and helping you track down bugs in your codebase. They are also happy to report any product bugs you may encounter directly to our development team. 

Attend a Synergy DevPartner conference

Our conferences provide a dedicated time to learn and immerse yourself in the latest and greatest in the Synergy/DE ecosystem. It’s free for DevPartner subscribers, and it’s a unique opportunity for your new developers to get face-to-face interaction with their Synergy developer peers and the Synergex team.

If you need help identifying any additional educational materials, please reach out to us! Congratulations again on your new Synergy developer, and best of luck with the onboarding process.

Sincerely,

The Synergex Education Team


What’s Up with Harmony Core?

By Harmony Core Team, Posted on March 22, 2019 at 2:49 pm

Avatar

If you attended the 2018 Synergy DevPartner Conference in New Orleans, you probably remember hearing about Harmony Core. Harmony Core is Synergex’s open-source solution for allowing access to the data and logic in your Synergy applications. We had roughly six conference sessions on this topic, so if you weren’t at the conference or need a refresher, you can check out the videos. Since then we have been working fast and furious on our backlog, ideas that surfaced at the conference, and adjustments to Harmony Core based on early adopter experiences. In this post, we’ll attempt to shed light on some of our recent decisions, struggles, and successes. We also want to share our plans for the near future so you know what to look forward to and what to expect from us.

From the Backlog

When we announced the release of Harmony Core at the conference, we discussed the backlog of tasks we wanted to complete. In the months since we shared the following slide, many of these items have been delivered.

What's next

Later in this blog post, we’ll discuss some of the enhancements that have been implemented, but we’d like to start by mentioning one of the more important changes we’ve made for optimization: implementing sparse Select. Without this, Harmony Core could never realize its performance potential because of the extra data sent over xfServer connections. Fortunately, the implementation turned out to be fairly straightforward. Harmony Core simply detects whether a channel is remote by using %ISINFO. If the channel is remote, Harmony Core builds a sparse Select object by processing the supplied EF Core LINQ expression and searching for field and object references that are required to create the result. If the channel is local, Harmony Core won’t build a sparse Select object because the setup required for the object would incur a small performance cost.

Conference Conversations

At the conference, we talked with many of you about how we could improve and extend Harmony Core. These conversations were invaluable in helping us prioritize items in the backlog, and they got us thinking about how we could better take advantage of open standards. One thing we started thinking about was our custom protocol for Traditional Bridge, which is the Harmony Core component that exposes traditional Synergy logic. This custom protocol was well documented and made use of JSON, so it wasn’t difficult to understand. However, it certainly seemed odd to have something custom like this, given our focus on open standards. We wanted to find an open solution, and as we looked around, JSON-RPC 2.0 caught our attention due to its use as the base for Microsoft’s Language Server Protocol. Microsoft uses this protocol for remote procedure calls across standard in/out, which is very similar to our use case. And on further investigation, we found that there was an open-source project that implemented JSON-RPC 2.0 in .NET Core. It’s written by Andrew Arnott, a long time Microsoft employee with whom we’ve interacted extensively, making this an even more enticing choice.

Having a top-quality high-performance library available sealed the deal, and we set out to switch the protocol. After some limited changes to Harmony Core server code (written in traditional Synergy) we had a working JSON-RPC 2.0 server and were able to communicate with it using vs-streamjsonrpc. Using someone else’s client to talk to our traditional Synergy server gave us a great deal of confidence in our adherence to the standard and ensured that we won’t have any weird behavior if you want your server to interface with some other non-Synergy system using the same mechanism. This is just one of the many steps we’ve taken to make Harmony Core open and accessible for all of you.

Real-World Findings

Early on, a lot of our decision-making for Harmony Core was based on our experience developing and supporting xfServerPlus, xfODBC, and Symphony Harmony. Since the conference, we’ve been able to get Harmony Core into the hands of some early adopters, which has enabled us to make more targeted decisions based on real-world situations. We’ve learned a lot so far, and we’re looking forward to tackling any new scenarios that arise in the future. For now, we’ve got a solid list of improvements that should make the implementation process even smoother for future adopters.

Exposing Synergy Code as a Web Service

With our first early adopter, we exposed preexisting Synergy code as a web service. We had expected this to be a use case, but hadn’t yet worked through all the implications for the project layout and how to bring together all the CodeGen templates. We ended up making quite a few changes to the project template layout and tweaks to the CodeGen batch file included with the project template.

Initially, we weren’t sure whether we wanted to use .NET Framework or .NET Core for this project. Did we need .NET Framework AppDomains for code isolation? Or could we get away with the much more performant AssemblyLoadContext available with .NET Core? After extensive discussions, we determined that we needed only global/common/static isolation, so .NET Core proved to be a perfect fit. Since we were wrapping preexisting code that wouldn’t normally compile on .NET Core, we needed to make a .NET Standard version of the customer’s core libraries. This went pretty smoothly once we sprinkled a few .IFDEFs around to remove code related to WinForms and Windows services.

Once everything was running, we were greeted by a debugger that was behaving strangely. We looked at our isolated objects in the debugger, but the debugger wouldn’t display any values, and we weren’t able to expand any children. It turned out there was a bug in the .NET Core debugger that prevented evaluation of objects that existed in multiple AssemblyLoadContexts. This issue was reported to Microsoft and was fixed (in Visual Studio 2019). This marked our first of many successful Harmony Core implementations.

Querying with OData

After tackling all the bugs and features for our first adopter, we started to consider how best to expose search functionality. To implement things like search, we generally recommend using the Harmony Core EF Provider, which eliminates the need for programmer intervention. In this case, however, we couldn’t use the Harmony Core EF Provider due to the nature of the project’s file/data layout. A comprehensive search function was already built into the application, but we wouldn’t really be implementing OData if we just exposed custom methods that didn’t offer any OData query options. Thus, the idea for adapters was born!

The existing search had a particular set of operations, numeric ranges, field matches, “or field matches”, and arbitrary parameters. These sorts of operations can be expressed in a single flat data structure, so we created the dispatch machinery to convert OData queries into a single class instance and call the requested Synergy method, passing that instance to it. To maintain high performance when dispatching, we generated the dispatcher using low-level LINQ expressions to take advantage of their precompilation features. This means that Harmony Core does the complicated task of generating code to map an OData query to a Synergy data structure the first time the request is made. After that, it just runs the compiled code at full speed. This was a bit of a challenge to implement, but we think it will make web services much more performant.

Repository

After the conference, we picked up a few more early adopters, including several with repositories that hadn’t been used for xfODBC so they didn’t have any relations defined. As we worked with these repositories, we came to realize that Harmony Core could not describe a two-way relationship when a tag field was involved. A tag field is a literal that indicates record type when multiple record types are stored in a single ISAM file. Only foreign keys can specify tag literals, but repositories can’t have foreign-to-foreign key relations, which would allow for two-way relationships. (Repositories can have only access-to-access key relations and foreign-to-access key relations.) After some back and forth on this, we came up with an answer: enhance CodeGen. CodeGen now looks across relations defined for structures and glues both foreign keys together in a foreign-to-access/access-to-foreign key relationship. To do this, CodeGen uses either the foreign key naming convention or description fields for the foreign keys.

In addition to fun with foreign keys, we ran into some interesting EF Core limitations when describing the fields in a join. Synergy allows keys to be comprised of up to eight segments that can total up to 256 bytes. EF Core has a loosely similar concept, composite keys. When using composite keys to describe a relationship, however, the parts of a composite key must have matching types. This is not a restriction in Synergy, so we had to figure a way around this limitation. The solution we came up with was to add another type of field metadata: composite fields. These fields in the generated DataObjectMetadataBase class allow us to keep track of the size, position, and type of each supplied key segment, while exposing only a single dummy field to the EF Core Navigation property.

The work we did to create composite fields in metadata opened up an additional opportunity. When we introduced Harmony Core at the conference, it was possible to create calculated fields simply by defining a property with a getter that returned the data in question along with a setter that either threw an exception or wrote to the actual underlying field, depending on your use case. But Harmony Core didn’t yet understand which fields were required when selecting a calculated field while using sparse Select with xfServer for remote data access. Thanks to composite fields, this is now implemented. To take advantage of this enhancement, you’ll need to create a property in your DataObject’s partial class and implement a partial class for its corresponding DataObjectMetadataBase with a private void method called InitializeCustomFields. InitializeCustomFields just needs to call AddFieldInfo (which creates a composite field with the same name as the calculated field), passing references to the field metadata for each of the fields required to calculate the field.

Over the past few months, we’ve been polishing our templates, building out our documentation, and fixing bugs reported by our early adopters. With all these enhancements, Harmony Core should be a Synergy developer’s best bet for adding Synergy web services. Still not sure if this is the right choice for your applications? We’re working on so much more.

Our Plan Going Forward

Because Harmony Core is an open-source project, we want to share not only our technical roadmap, but also our goals for the project. We also want to set expectations up front so you can help keep us accountable and on track.

Technical Direction

There are so many things we can do to extend the Harmony Core framework, but we’ve chosen a few large items and a smattering of smaller ones to focus on over the next 12 months. We’ll continue to post blog updates as features are delivered and the direction evolves.

Webhooks

Webhooks are user-defined HTTP callbacks that are usually triggered by some event, such as a status update for an order, or when the quantity of something drops below a certain level. Webhooks are very useful when you have to glue two complex systems together. Imagine your company has grown by acquisition and you find yourself needing to merge the underlying data and business functions of two different IT systems. If you’re trying to build a unified RESTful view of both systems, your view is going to need to be notified when important events occur in the underlying system. Webhooks are the best way to do this. We think this sort of functionality will be crucial in many scenarios. If you have thoughts about how you would like to use this technology, please post on https://github.com/Synergex/HarmonyCore/issues/61. We aren’t certain what shape this support will take, but we will be actively pursuing this. Large (1-3 months)

Adapters

While we’ve already implemented adapters, there is so much more we can do to extend their uses. Currently, adaptors apply only to filters, but we can further blur the line between operations provided by EF Core and custom code. This would entail implementing additional attributes to indicate that a particular field or collection needs to be expanded. In general, our design philosophy is to flatten OData queries into the sort of data structures that customers currently use to implement custom solutions. Supporting custom Synergy code is a very important part of Harmony Core; many of you have decades of very important business logic that you want to expose to the world. We would be doing you a disservice if the full OData query experience were not applicable to your important business logic. Adaptor enhancements will enable you to take full advantage of uniform OData query syntax, whether you’ve implemented an endpoint with custom code or are using the Harmony Core EF Provider. See https://github.com/Synergex/HarmonyCore/issues/85. Large (1-2 months)

Joins

Not all customer data is laid out the way we developers would like it to be. Sometimes this is for historical reasons, or it might be the result of business requirements. Whatever the reason, we want to make sure it’s possible to fully expose your data via OData and EF Core. This means we need to add at least some support for nested subqueries. The main use case we’re thinking of is when data for different branch locations is stored in separate files. In order to join across all branches, we will have to split the query into several parts, storing some intermediate results in a hash lookup table. While this doesn’t amount to support for completely arbitrary subqueries, we think it will solve most customer needs. See https://github.com/Synergex/HarmonyCore/issues/83. Large (1-3 months)

A Migration Path for xfServerPlus

Prior to the conference, we added support for reading xfServerPlus method catalogs. We also added initial support for generating the required wrappers and glue to allow existing xfServerPlus methods to be called using the JSON-RPC 2.0 protocol and the Traditional Bridge portion of Harmony Core. This needs to be extended to support all the data types and coercion that exist in the wild. See https://github.com/Synergex/HarmonyCore/issues/84. Large (1-3 months)

Durable Transaction Log

At the conference we rhetorically asked, “Is this ACID?” Spoiler alert: we’re pretty close but not quite all the way there. The missing piece of the puzzle is a durable transaction log, which is really just a fancy name for an on-disk data structure used as part of a two-phase commit. As transactions are being committed to the underlying ISAM files, Harmony Core would write a log of the operations that it’s performing with information on the state they are in should there be a hardware or software fault that interrupts ISAM operations. In addition to its verification duties, the transaction log would also enable us to undo any changes that were only partially committed. When combined with innovations at the operating system level (such as Storage Spaces Direct in Windows Server 2019) and ISAM resiliency in Synergy/DE 11, we think the transaction log will provide the foundation for highly available, fault-tolerant Synergy applications and web services. See https://github.com/Synergex/HarmonyCore/issues/82. Large (2-4 months)

Tooling Improvements

CodeGen integration with MSBuild – We’ve wanted to improve the process of building projects that use CodeGen templates. The degree to which we rely on CodeGen templates for Harmony Core projects is staggering. Keeping generated code up to date should be a build system operation rather than something for humans to manage (i.e., forget). Small (1-2 weeks)

CodeGen integration with NuGet – Delivering updates to CodeGen templates is tricky and manual. Those are a pair of attributes that should never be associated with tasks that must be performed frequently. Our answer to this problem is to ship template (.tpl) files using NuGet packages. This way, your build system can reference a specific version number for the templates you’re using. If updates are available, all you have to do is bump the requested version, and CodeGen will go download the requested package of templates. Small (1-2 weeks)

.NET CLI scaffolding tool – The command line interface for .NET Core introduced a concept known as scaffolding. Rather than shipping complex logic in wizards, project templates, or item templates, the dotnet tool allows you to offer a scaffolding generator. Conceptually, it’s very similar to a wizard in Visual Studio, but it’s much easier to work with and can be used without Visual Studio. Because Harmony Core relies heavily on CodeGen templates and source conventions, it’s currently more difficult than it should be to add new functionality to an existing project. Once we have a scaffolding tool, it will be much easier to add new endpoints, new structures, custom code in partial classes, and wrappers around existing code. Small (1-2 weeks)

Concurrency Improvements

Multiple xfServer connections – Currently the Harmony Core EF provider uses only a single xfServer connection. We need to build additional infrastructure into the provider to maintain a pool of worker threads that each have an open xfServer connection. And the FileChannelManager class will need some additional smarts to deal with request queuing and to use existing threads with requested channels. See https://github.com/Synergex/HarmonyCore/issues/79. Small (1-2 weeks)

Additional configuration options for FileChannelManager and, more generally, the pooling mechanisms – Currently we open file channels and never close them for the duration of a FileChannelManager object. This improves the performance of individual requests, but it’s wasteful in situations where there are hundreds of files, but few that are commonly accessed. The plan for this is to implement a least-recently-used (LRU) eviction strategy. Keeping track of the total list of all files that have ever been opened and the relative frequency of file opens should allow us to auto-tune the number of channels to keep in the LRU list. See https://github.com/Synergex/HarmonyCore/issues/81. Small (1-2 weeks)

Traditional Bridge SSH connection multiplexing – Currently, we open one SSH connection for each Traditional Bridge process on the remote server. For a very large number of processes, this is wasteful because connections are sitting idle most of the time. We can instead create up to 10 sessions per SSH connection (or more if the operating system is configured for it), which would enable us to have many more worker processes without the additional SSH overhead. This would be configurable for cases where SSH connection scaling isn’t the bottleneck. See https://github.com/Synergex/HarmonyCore/issues/80. Small (less than 1 week)

Better Debugging

Diagnosing issues in production is one of the most challenging tasks for a developer. We’re considering two features to make this less daunting: per-request logging levels and per-request Traditional Bridge debugging.

Per-request logging levels – Harmony Core offers extensive logging along with a logging framework that works well with Synergy data types. This is great in development, but in production it can destroy performance or collect sensitive data that shouldn’t be logged. By providing a custom HTTP header with a request, the logging level could be set specifically for that request. This means you could get full fat logging from a production server while the enhanced logging is turned off for the rest of your requests. See https://github.com/Synergex/HarmonyCore/issues/77. Small (less than 1 week)

Per-request Traditional Bridge debugging – Similar to per-request logging levels, we can use custom HTTP headers to cause the current request to launch a debuggable Traditional Bridge process. This would cause a separate .dbr and command line to be run on the remote system for this one request, enabling you to attach a debugger from Visual Studio to the running remote process, even in a production environment with thousands of normal requests being serviced concurrently. See https://github.com/Synergex/HarmonyCore/issues/78. Small (less than 1 week)

Harmony Core Goals

What will determine the success of this project? Ultimately, it will be you, the customer. So we want to make sure that implementing and interacting with the web services you build with Harmony Core is as painless as possible.

For 2019, our goal is to have at least four customers that have successfully deployed Harmony Core web services or are in the final stages of developing real-world solutions that make extensive use of Harmony Core. This is not as aggressive as it sounds; we already have four customers in various stages of development, and we aren’t just aiming to get four customers; we want those customers to be happy with the quality of Harmony Core and with the features it provides. To determine if these customers are happy, we’ll keep track of the net promotor score for all the different parts of Harmony Core. For example, if you have a complaint about the documentation or need more help than we expected in order to implement something, that’s a sign that we are missing documentation, examples, educational material, blog posts, or functionality. We’ll need to hear this from you to know how to do better. Once an issue is found, we’ll address it and follow up to determine if the solution actually worked. The number of adopters can’t and won’t be our only metric for success. Additionally, we anticipate that we’ll need to interact with non-technical decision makers and non-Synergy developers who in some cases will know best if Harmony Core is actually solving their business needs.

Open-Source Technology: The New Frontier

We feel that open-source development is a two-way street; in order to be successful, we need non-Synergexian contributors to create pull requests and issues on GitHub. We’re hoping for at least four non-Synergexian contributors to submit pull requests or issues by the end of 2019. The contributions can be for documentation fixes, new samples, or even new core functionality. We will tag GitHub issues that we think are a good fit for external contributors. For a non-tagged issue, we would ask that you give us a heads-up on GitHub to make sure we aren’t currently working on a solution for the issue. In general, we’re looking to add features and functionality that will be applicable to the majority of users, so we won’t want to incorporate code that is specific to a company or line of business.

What You Can Expect from Synergex

When it comes to open-source products, Synergex’s stance has generally been to provide the software as-is with no guarantees. For Harmony Core, however, you can expect Synergex to provide more robust support. Although the development path for Harmony Core is different than for our proprietary products, we want companies of all sizes to feel comfortable taking such a large dependency on Harmony Core. We think this can be accomplished using GitHub issues, but we’re looking to you, the Harmony Core community, to tell us what kind of service-level guarantees your management team will need.

Support

Our Developer Support team is already training up on Harmony Core, and we plan to train them to write and understand CodeGen templates. There are quite a few new areas inside CodeGen that have been added as a result of the Harmony Core project, and you can expect the support team to have up-to-date knowledge on those enhancements. We will also begin looping Support into our ongoing development and deployment efforts with customers. We believe there is no training substitute for extensive guidance while getting your hands dirty on the inner workings of a product, and we think the team will be very well equipped to tackle and triage Harmony Core support cases in the near future.

In addition to providing help though Developer Support, we’re planning on incorporating some educational events and materials to ensure that you can successfully implement and interact with Harmony Core web services. We will be hosting office hours monthly for anyone interested in a chance to hop on a GoToWebinar session and ask questions or work through implementation issues. Office hours are a common feature in open-source projects run by companies who want make sure complex, difficult-to-solve issues aren’t ignored or missed or otherwise languish without solutions. You can also expect regular blog posts recapping the development progress that has taken place, and we’re interested in posting project post mortems (with identifying information scrubbed, of course). We think letting the Harmony Core community see the good, the bad, and the difficult in project implementation will give developers and management confidence that they will also be successful.

Documentation

Open-source products are usually documented by their creators, and this will be true of Harmony Core. Our tech writing team will be involved (organizing, editing, and helping direct the documentation), but all documentation and samples will come from developers working on Harmony Core. This is a departure from how documentation has traditionally been developed at Synergex, but we think this will be the best way to document this project.

Meet the Harmony Core Team

Jeff Greene, Harmony Core Product Manager

Jeff GreeneJeff joined Synergex in 2006 and he brought his passion for open source and standards-based technologies with him.

His first contribution to our product suite was Synergy DBL Integration for Visual Studio, for which he architected and implemented items involving Intellisense, debugging, and migration tools. He championed the importance of running Synergy on .NET Core and was the lead developer of that project.

Jeff has been instrumental in Synergex’s embrace of Service Oriented Architecture (SOA) and standards-based technologies. When the need for a unified solution to creating web services for all Synergy/DE applications arose, Jeff and Steve Ives created the Harmony Core Framework with an emphasis on using standardized technologies such as OData, Swagger, Asp.NET Core, .NET Core and Entity Framework Core.

Jeff has worked with various departments within Microsoft over the years, and has made significant contributions to the open source Microsoft CoreRT project.

Steve Ives, Harmony Core Co-Architect

Steve IvesFollowing an initial background in operating systems, networking and systems administration early in his career, Steve realized that his real passion and talent was in the field of software development.

Having worked for one of the largest Synergy/DE developers in the United Kingdom, Synergex frequently contracted Steve to present on Synergex locally in Europe. These engagements consisted of presenting on the company and products during meetings, delivering training seminars on the use of the products, and consulting with other customers about how to address specific technology requirements.

Steve formally joined the Synergex Professional Services group in 1997. Having held the position of Senior Consultant for over 20 years, Steve has assisted many Synergy developers to architect, design and implement a wide variety of software projects. Each of these projects involves Synergy/DE in some way, and many involve a wide range of other development languages and technologies.

In recent years, Steve has been a major contributor to several Synergy-related open source development projects, including:

  • The CodeGen code generator which can be used to automate the generation of Synergy DBL and other types of code in a wide variety of software scenarios
  • The SQL Replication reference implementation—this provides a working example and most of the code needed to replicate a Synergy application’s ISAM and Relative data to a SQL Server database, in near-real-time
  • A PDF API that allows Synergy developers to easily add PDF file creation to their applications
  • The Harmony Core framework which provides Synergy developers the ability to quickly and easily build RESTful Web Service APIs to expose their Synergy data and business logic to a wide variety of applications and environments

Steve is passionate about assisting software developers to continue to leverage the investment that they have made in their Synergy code base over the years, and to continue to see a return on their ongoing investments in their code.

Johnson Luong, Harmony Core developer

Johnson LuongJohnson joined Synergex in 2014 as a QA engineer. He got his start by testing Synergy releases and the Synergex Resource Center Community, as well as verifying trackers and automating the testing of those trackers. You can read about his testing techniques and automated testing at Synergex.

In addition, after moving into a software development role, Johnson made many contributions to our Synergy DBL Integration for Visual Studio (SDI) product including migrating from Windows Forms to the WPF property pages and the product installers from InstallShield to a newer technology. He has since developed all of our Synergy product installers with the open-source language, WiX. He worked closely with Fire Giant, the custodian of WiX, to ensure the Synergy installers are using their most up-to-date technology.

Prior to joining Synergex, Johnson was an active member of the Association for Computing Machinery (ACM) at his alma mater. Along with his team, he even won their biannual International Collegiate Programming Contest. His background in test along with experience in Visual Studio development and solid foundational knowledge of algorithm design make Johnson the perfect addition to round out the Harmony Core team.


Highlights from the 2018 Synergy DevPartner Conference

By Synergex, Posted on January 14, 2019 at 3:26 pm

Synergex

As we head into 2019, we reflect back on one of our favorite events of the year: the 2018 Synergy DevPartner Conference in New Orleans.

Packed with sessions on topics ranging from modern and agile development practices to RESTful web services to QA and learning culture, not to mention an engaging and illuminating keynote presentation from Microsoft’s Donovan Brown, the conference offered a plethora of info, ideas, tips, tricks, and plans for the future. Case studies and customer demos provided insight into practical implementations of technical concepts, and we were jazzed to introduce our new open source RESTful web services project, Harmony Core. Bourbon Street wasn’t too bad either! The legendary food, music, and culture of one of America’s most historic cities provided a great backdrop to a fun and productive week.

Here are some key takeaways from the conference:

  • Improve your productivity and practices by adopting more efficient development methodologies.
  • Enhance years of Synergy data and code with new technologies, including enabling connectivity through RESTful web services.
  • Security and disaster recovery are important for compliance—stay up to date with Synergy SSL and operating system security patches.
  • Use traditional Synergy in Visual Studio (it’s not as hard as it seems!) to significantly boost productivity, lower the barrier to continuous code integration, and improve your processes and software quality.
  • Move to the cloud. Developing and running your Synergy application in the cloud is relatively easy and provides a convenient path to expand your infrastructure to meet demands.
  • Education in the workplace is important—create an onboarding program that includes presentations, videos, and discussion.

The Synergex team came back to the office energized and ready to implement big plans for Synergy in the new year. We look forward to seeing you at the 2020 conference!

You can check out videos and slides of conference sessions here.

(more…)


Don't miss a post!

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Recent Posts Categories Tag Cloud Archives