A quick post to announce the availability of CodeGen 5.6.6. This is a quality release that addresses an issue that could occur when generating code from templates containing certain complex expressions. In particular, several Harmony Core templates contain such complex expressions, so we recommend that all Harmony Core developers upgrade to this new version of CodeGen at their earliest convenience. As always, documentation for this new release can be found at https://codegen.synergex.com.
Ahh… summer BBQs. Is there anything that defines the summer season better than a good BBQ? Yes, yes, there is: A browser-based application, made with React and Redux, that helps you organize your BBQs. Don’t you agree? How else are you going to keep track of who ordered what and with which toppings? Did Aunt Vi want one or two hot dogs? Did she order them burnt or regular? Did you buy enough turkey burger patties? You forgot that Manny and Bigbah both wanted two veggie burgers each! How could you have accounted for that?
Enter xfBBQ, the solution to all your existential BBQ woes. I built xfBBQ using the latest and greatest in web application technology, React and Redux. It’s the ultimate answer to managing your perfect BBQ. As its back end, xfBBQ uses the excellent Harmony Core framework—your solution to modernizing your Synergy applications with web services!
Is all this hyperbole not enough for you? Do you also need to know how you can get your hands on this wonderous application? Easy peasy: It’s on our GitHub, of course! Just clone it and see for yourself. You can check out the application live, too. xfBBQ is for everyone!
You may be wondering, how did I xfBBQ? What do you mean “how” did I xfBBQ? That’s not even a valid question. I don’t think xfBBQ is a verb. But maybe you meant how did I create xfBBQ? Now that’s an excellent question, which calls for an equally excellent answer! Fortunately, you can find my four-part series about xfBBQ on YouTube.
Part 1 explains how to set up your Harmony Core and React environments from scratch. I talk about what prerequisites you need for the app. I also explain in detail how to edit your regen.bat to get Postman working with Harmony Core. In addition, I talk about the first stages of getting the front end and back end to talk to each other. By the end of this segment, the application can display raw JSON data coming from Harmony Core to the front end.
In Part 2, I clarify how to add a primary key generator in order to POST new data to the back end. I add a user registration form to show off this new feature. Then, Redux comes crashing into the BBQ party, so I need to explain how it all works. Redux is here to stay, so I integrate Redux into our app and show it working in a new login form. Based on the permissions set on who logs in, the application will display different pages for different users, all thanks to Redux.
Part 3 introduces reCAPTCHA into the picture. I show how to use reCAPTCHA to protect both the front and back end from user registration bot attacks. Afterwards, I add batch order processing to the mix. This allows a user to add multiple orders at the same time, all as one POST, leading to less strain on the back end. In addition, I update the UI a little bit to make the data coming from the back end easier to read.
Lastly, Part 4 explains why xfBBQ needs user authentication. Because the back end is public, it needs protection from all sorts of nasties out there in the wild. User authentication helps by forcing a user to sign in from the front end before being able to do anything with the back end. Finally, I go over styling the application using Bootstrap so that it looks all nice and fancy. As a bonus, Bootstrap is responsive, meaning that the web application now works on your phones! As an ultimate step, I discuss how to use create-react-app to build a production version of xfBBQ. I set up Harmony Core to use this production build, too.
So, what are you waiting for? xfBBQ is the perfect example of how you can modernize your applications and bring them to the web. In my webinars, I explain everything in depth, from initial setup to final commit. Plus, with a few more steps, your applications can be mobile friendly! Now, who’s ready to put their hot dogs on the grill? I’ll be waiting… Hold the mustard, please.
On Thursday, June 18th we will launch a new series of webinars called the xfBBQ Webinars, and I’m guessing this may require some explanation. So here goes:
What is xfBBQ?
xfBBQ is a browser-based application, designed and created by Johnson Luong, Synergex Software Developer, (with minor input from Jeff Greene, Harmony Core Technical Lead, and Synergex Principal Engineer) to help us plan for our regular company BBQs. The ones that we are NOT currently enjoying because of the COVID-19 pandemic! Generally though, we host around six BBQs outside our office at lunchtime during the summer season, but most of you won’t care about that because you’re not invited 😉!
Who is Johnson Luong?
Johnson is a valuable member of our development team who, since joining Synergex in 2014, has worked on many different parts of the Synergy/DE toolset. Recently he has played a major role in the creation of our Windows installations using the WiX Toolset, while at the same time having made significant personal contributions to the development of the entire Harmony Core framework.
What does xfBBQ do?
The app allows customers (employees) to register for access, then informs them when BBQs are scheduled, and allows them to select from, and customize, various menu selections for each BBQ. For example, for a given BBQ, I might use the app to express my preference for a “medium rare cheeseburger with hot peppers and a well-done dog.” I could even opt for two burgers, or two dogs, or both … but in doing so I may risk the wrath of Samantha, our wellness coordinator! Johnson assures me she doesn’t have a special “back door” into the data, but I’m not convinced!
Anyhoo, the app collates the orders, presenting the BBQ officials (all who lavish in Synergex VIP status, and rightly so) with the required lists of requirements and ingredients for each BBQ event. Yada, yada, yada.
Why did we do this?
Because having developed an awesome (our words, but we’re confident) web-services framework called Harmony Core, customers were asking for real-world examples of how it might be used. And … we had a real-world problem to solve, namely that we really needed to control the amount of burger patties and sausages that we were purchasing from Costco!
Just for fun! Johnson believes that he has now defined the actual meaning of “xf”, but trust me, he has no clue! Until it’s been ratified by Marketing it means nothing, and that process could take years!
But seriously, this webinar series presents to you an end-to-end view of how a modern browser-based application is developed. You will see how the Harmony Core back-end is built, but you have seen that in previous webinars. What is much more important here is the focus on the development of the front-end application that interacts with the Harmony Core service.
This will be a series of four webinars, as follows:
Thursday June 18 @ 9 am PDT
Creating a Harmony Core solution and hooking it up to a new React app.
Register for part one here.
Thursday June 25 @ 9 am PDT
POSTing data in React, and an introduction to Redux.
Register for part two here.
Wednesday July 1@ 9 am PDT
Captchas, batch processing, and other CRUD.
Register for part three here.
Wednesday July 8@ 9 am PDT
User authentication, styling with Bootstrap, and making a React production build.
Register for part four here.
We are completely excited that over the last few months we have built up a marvelous following for our Harmony Core webinars, but I’ll be honest, there won’t be much DBL code to see here! Regardless, I encourage all of you to join us for this webinar series, and to stick with it to the end.
If you’re a DBL dev you may not learn much about your own part of the puzzle that is building a modern distributed application. But I promise you, by the end of the process you will have learned a lot about what it takes to pull together all the other pieces of that puzzle. Building a modern, distributed application requires skills in many different areas, and this webinar series touches on many of them.
It is most important that we get your non-Synergy developers to participate in these webinars. Your web devs will be right at home here, but if you don’t have web devs then invite your Windows devs, your iOS and Android devs, and your MacOS devs.
Web services are accessed via HTTP, and pretty much every development language can do that! The perfect target audience for these webinars is anyone that is involved in developing your next-generation apps. It’s not about any one part of the process, it’s about the entire process, end-to-end, and that will require devs with many different skill sets get together, and to work together to create success!
Following the holidays, we didn’t have much time for major updates to Harmony Core since our last office hours. Our focus has been on the EF Core 3.1 provider, which is necessary for us to support ASP.NET Core 3.1. This is should be finished in the next few days, and this will probably resolve a few open issues on GitHub.
Jeff and Steve give an update on Harmony Core development activities over the last six weeks, which included the following.
Traditional Bridge has been updated to use the new system.text.json parser and writer. This reduces overhead when running traditional Synergy methods, resulting in significant performance improvements (a full order of magnitude from what we’ve seen). This enhancement is built on the version 11 Synergy/DE runtime, so it requires version 11 on the traditional side of the bridge. Traditional Bridge still works with prior versions, but these versions can’t take advantage of the built-in JSON libraries that make the performance improvements possible.
We created a prototype with EF Core 3.1 support. Microsoft has completely reworked the model for providers, so this prototype took quite a bit of work. The reworked model is an improvement; it makes query analysis requirements more explicit, so the Harmony Core EF Provider does not have to do as much pattern mapping for optimization. We will continue to improve this support, and we’ll transition the project from prototype to production.
We also made several improvements to project templates. The metadata that is injected into data models has been extended, and some data models have additional attributes that improve the fidelity of generated documentation. These attributes provide additional information about the HTTP responses each operation endpoint can generate, along with the shape of the data that is returned by each operation. This should provide much better information for Swagger documentation. These changes will be checked into the Harmony Core Github repository in the next couple of days.
Finally, we fixed a customer-reported bug that caused duplicate items in multi-leg joins (issue 131), and we continued work on Harmony Core Generator, a utility designed to replace regen.bat and display relations.
Jeff and Steve give a short update on the Harmony Core project, including what will be changing with Synergy/DE 11. They also introduce the xfBBQ project. The team then reviews an issue and answers several questions on the project.
Jeff and Steve give an update on the Harmony Core project. Jeff also validates resolving issues that have been raised in GitHub. Lastly, Jeff and Steve discuss adding documentation to let users know how to enable STDOUT logging inside IIS.
The Harmony Core project has been steadily gaining momentum as we improve our processes, work on support issues, and continue with user engagements.
To start with, we’ve been working on our continuous integration and continuous delivery (CI/CD) pipeline in Azure DevOps. This pipeline serves a dual purpose. It allows us to demonstrate software development best practices applicable to Harmony Core users, and it enables our team to be more productive. Whenever code is committed or a pull request is created, the full test suite is run to ensure that the code is good and that there are no regressions. Automating this has improved our ability to take pull requests from outside contributors.
Significant effort has gone into offering a nearly seamless experience for xfServerPlus users who are looking to expose their Synergy routines via RESTful web services. We have updated CodeGen, our CodeGen templates, and the underlying support libraries. CodeGen can now read SMC interface definitions, and there are new tokens for iterating over xfServerPlus methods and parameters. Additionally, CodeGen now uses SMC interface definitions as input when determining which repository structures to iterate over. We use this functionality to generate the data objects required to wrap structures passed as parameters.
As part of a consulting engagement, we built a mechanism to support custom loading and custom validation of data objects and transactions (Issue 107). We enhanced CodeGen and the CodeGen templates in Harmony Core to support certain types of relations to ensure that required relations have the necessary resources in the database or the current transaction.
Traditional Bridge Improvements
Up to this point, we’ve been exclusively promoting RESTful web services, but now we’re introducing an alternative. We’ve created CodeGen templates for exposing Traditional Bridge services via SignalR, a technology provided by Microsoft. SignalR allows for two-way communication in or outside of the browser without compromising performance, reach, or standards. We’re recommending SignalR in cases where an application can’t operate statelessly or the server needs to push updates to clients rather than requiring them poll or refresh for changes.
On the xfServerPlus front, we implemented encoding in Traditional Bridge for all currently known scenarios and data types that are supported for xfServerPlus. This includes arrays, handles, ArrayLists, and scalar types. We also implemented a secondary calling convention that uses the Synergy routine call block API for those of you that use the optional parameter support offered only in xfNetLink COM clients. And we’re actively working on porting the full test suite used by xfServerPlus and xfNetLink.NET to work as unit tests for Traditional Bridge. Swagger Docs
Over the last few weeks, we’ve been working on improving our API version support. We found an open-source Microsoft library for this, and we have created integration samples. This library does two things for us: it takes care of versioning, and it produces Swagger docs for versions, enabling us to move away from statically generated Swagger files. This dramatically improves the quality of Harmony Core Swagger documentation for items that fall outside the normal set of GET and POST entities. And it has the added benefit of allowing you to expose source code doc comments as part of your API documentation.
Traditional Runtime Updates
The traditional Synergy runtime for version 11 includes support for reading and writing JSON using a subset of the new System.Text.Json namespace from .NET Core 3.0. We have decided to use this functionality because it is designed for speed. Even for moderately-sized JSON documents, we’re seeing upwards of 15x faster parsing, and the number gets larger for larger documents. In the coming weeks, we’ll be implementing a dual pathway for JSON reading and writing in Traditional Bridge. If you’re on a runtime older than 11, we’ll fall back to the implementation written in Synergy DBL. But if you’re using the latest runtime, you’ll be able to take advantage of the massive performance increase.
A New Harmony Core Example
We’ve received some requests for a complete end-to-end Harmony Core example that demonstrates how to develop, test, and deploy a web application backed by a Harmony Core web service. We’ve come up with an idea for an application we’re calling xfBBQ. At our offices during the summer, the development team frequently hosts barbeques for the company, and one of the challenges we face is taking orders. This is currently managed using a Google Sheets spreadsheet, which isn’t ideal. But it gave us the idea to build an example app to improve this process. xfBBQ will be implemented as a React + Redux Single Page Application. We’re planning to pack large amounts of sometimes gratuitous functionality, but we’ll stay within the bounds of best practices.
Utilizing xfBBQ for planning company BBQs
Resolving Your Harmony Core Issues
There are two things we need to know when prioritizing Harmony Core development: what issues you have and what features you need. And we can more quickly resolve your issues and implement the features you need when we know what they are. The following are the best ways to reach us.
We host online office hours every six weeks. During this time, we deliver a short update on what we have improved in the Harmony Core framework. At the end of each session, we answer questions and work through issues you may have. This is a great opportunity to get direct and live feedback from us. During our last online office hours session, we fielded some support questions that we were subsequently able to resolve. We will continue to offer these sessions every six weeks. If you have any questions for us, please sign up for the next office hours session so we can answer your questions directly.
The next best way to reach us is through GitHub issues. Here we can easily keep track of your issues and make sure they are resolved as quickly as possible. This is also helpful because it creates a repository of answers for other users to search through. Recently we were able to resolve questions regarding ISMFILE.txt and .XDL files and where to put logicals by directing the user to our wiki documentation.
Jeff Greene discusses some recent updates to Harmony Core. Jeff also asks the audience if there is any interest in an Entity Framework Core project template, and guidance for stand-alone use.
Jeff and Steve discuss some recent updates to Harmony Core and review a few use cases used by others.
Jeff Greene and Steve Ives talk to Chris about the progress of his project. Steve also presents a demonstration on the basics of Harmony Core to Deborah.
If you attended the 2018 Synergy DevPartner Conference in New Orleans, you probably remember hearing about Harmony Core. Harmony Core is Synergex’s open-source solution for allowing access to the data and logic in your Synergy applications. We had roughly six conference sessions on this topic, so if you weren’t at the conference or need a refresher, you can check out the videos. Since then we have been working fast and furious on our backlog, ideas that surfaced at the conference, and adjustments to Harmony Core based on early adopter experiences. In this post, we’ll attempt to shed light on some of our recent decisions, struggles, and successes. We also want to share our plans for the near future so you know what to look forward to and what to expect from us.
From the Backlog
When we announced the release of Harmony Core at the conference, we discussed the backlog of tasks we wanted to complete. In the months since we shared the following slide, many of these items have been delivered.
Later in this blog post, we’ll discuss some of the enhancements that have been implemented, but we’d like to start by mentioning one of the more important changes we’ve made for optimization: implementing sparse Select. Without this, Harmony Core could never realize its performance potential because of the extra data sent over xfServer connections. Fortunately, the implementation turned out to be fairly straightforward. Harmony Core simply detects whether a channel is remote by using %ISINFO. If the channel is remote, Harmony Core builds a sparse Select object by processing the supplied EF Core LINQ expression and searching for field and object references that are required to create the result. If the channel is local, Harmony Core won’t build a sparse Select object because the setup required for the object would incur a small performance cost.
At the conference, we talked with many of you about how we could improve and extend Harmony Core. These conversations were invaluable in helping us prioritize items in the backlog, and they got us thinking about how we could better take advantage of open standards. One thing we started thinking about was our custom protocol for Traditional Bridge, which is the Harmony Core component that exposes traditional Synergy logic. This custom protocol was well documented and made use of JSON, so it wasn’t difficult to understand. However, it certainly seemed odd to have something custom like this, given our focus on open standards. We wanted to find an open solution, and as we looked around, JSON-RPC 2.0 caught our attention due to its use as the base for Microsoft’s Language Server Protocol. Microsoft uses this protocol for remote procedure calls across standard in/out, which is very similar to our use case. And on further investigation, we found that there was an open-source project that implemented JSON-RPC 2.0 in .NET Core. It’s written by Andrew Arnott, a long time Microsoft employee with whom we’ve interacted extensively, making this an even more enticing choice.
Having a top-quality high-performance library available sealed the deal, and we set out to switch the protocol. After some limited changes to Harmony Core server code (written in traditional Synergy) we had a working JSON-RPC 2.0 server and were able to communicate with it using vs-streamjsonrpc. Using someone else’s client to talk to our traditional Synergy server gave us a great deal of confidence in our adherence to the standard and ensured that we won’t have any weird behavior if you want your server to interface with some other non-Synergy system using the same mechanism. This is just one of the many steps we’ve taken to make Harmony Core open and accessible for all of you.
Early on, a lot of our decision-making for Harmony Core was based on our experience developing and supporting xfServerPlus, xfODBC, and Symphony Harmony. Since the conference, we’ve been able to get Harmony Core into the hands of some early adopters, which has enabled us to make more targeted decisions based on real-world situations. We’ve learned a lot so far, and we’re looking forward to tackling any new scenarios that arise in the future. For now, we’ve got a solid list of improvements that should make the implementation process even smoother for future adopters.
Exposing Synergy Code as a Web Service
With our first early adopter, we exposed preexisting Synergy code as a web service. We had expected this to be a use case, but hadn’t yet worked through all the implications for the project layout and how to bring together all the CodeGen templates. We ended up making quite a few changes to the project template layout and tweaks to the CodeGen batch file included with the project template.
Initially, we weren’t sure whether we wanted to use .NET Framework or .NET Core for this project. Did we need .NET Framework AppDomains for code isolation? Or could we get away with the much more performant AssemblyLoadContext available with .NET Core? After extensive discussions, we determined that we needed only global/common/static isolation, so .NET Core proved to be a perfect fit. Since we were wrapping preexisting code that wouldn’t normally compile on .NET Core, we needed to make a .NET Standard version of the customer’s core libraries. This went pretty smoothly once we sprinkled a few .IFDEFs around to remove code related to WinForms and Windows services.
Once everything was running, we were greeted by a debugger that was behaving strangely. We looked at our isolated objects in the debugger, but the debugger wouldn’t display any values, and we weren’t able to expand any children. It turned out there was a bug in the .NET Core debugger that prevented evaluation of objects that existed in multiple AssemblyLoadContexts. This issue was reported to Microsoft and was fixed (in Visual Studio 2019). This marked our first of many successful Harmony Core implementations.
Querying with OData
After tackling all the bugs and features for our first adopter, we started to consider how best to expose search functionality. To implement things like search, we generally recommend using the Harmony Core EF Provider, which eliminates the need for programmer intervention. In this case, however, we couldn’t use the Harmony Core EF Provider due to the nature of the project’s file/data layout. A comprehensive search function was already built into the application, but we wouldn’t really be implementing OData if we just exposed custom methods that didn’t offer any OData query options. Thus, the idea for adapters was born!
The existing search had a particular set of operations, numeric ranges, field matches, “or field matches”, and arbitrary parameters. These sorts of operations can be expressed in a single flat data structure, so we created the dispatch machinery to convert OData queries into a single class instance and call the requested Synergy method, passing that instance to it. To maintain high performance when dispatching, we generated the dispatcher using low-level LINQ expressions to take advantage of their precompilation features. This means that Harmony Core does the complicated task of generating code to map an OData query to a Synergy data structure the first time the request is made. After that, it just runs the compiled code at full speed. This was a bit of a challenge to implement, but we think it will make web services much more performant.
After the conference, we picked up a few more early adopters, including several with repositories that hadn’t been used for xfODBC so they didn’t have any relations defined. As we worked with these repositories, we came to realize that Harmony Core could not describe a two-way relationship when a tag field was involved. A tag field is a literal that indicates record type when multiple record types are stored in a single ISAM file. Only foreign keys can specify tag literals, but repositories can’t have foreign-to-foreign key relations, which would allow for two-way relationships. (Repositories can have only access-to-access key relations and foreign-to-access key relations.) After some back and forth on this, we came up with an answer: enhance CodeGen. CodeGen now looks across relations defined for structures and glues both foreign keys together in a foreign-to-access/access-to-foreign key relationship. To do this, CodeGen uses either the foreign key naming convention or description fields for the foreign keys.
In addition to fun with foreign keys, we ran into some interesting EF Core limitations when describing the fields in a join. Synergy allows keys to be comprised of up to eight segments that can total up to 256 bytes. EF Core has a loosely similar concept, composite keys. When using composite keys to describe a relationship, however, the parts of a composite key must have matching types. This is not a restriction in Synergy, so we had to figure a way around this limitation. The solution we came up with was to add another type of field metadata: composite fields. These fields in the generated DataObjectMetadataBase class allow us to keep track of the size, position, and type of each supplied key segment, while exposing only a single dummy field to the EF Core Navigation property.
The work we did to create composite fields in metadata opened up an additional opportunity. When we introduced Harmony Core at the conference, it was possible to create calculated fields simply by defining a property with a getter that returned the data in question along with a setter that either threw an exception or wrote to the actual underlying field, depending on your use case. But Harmony Core didn’t yet understand which fields were required when selecting a calculated field while using sparse Select with xfServer for remote data access. Thanks to composite fields, this is now implemented. To take advantage of this enhancement, you’ll need to create a property in your DataObject’s partial class and implement a partial class for its corresponding DataObjectMetadataBase with a private void method called InitializeCustomFields. InitializeCustomFields just needs to call AddFieldInfo (which creates a composite field with the same name as the calculated field), passing references to the field metadata for each of the fields required to calculate the field.
Over the past few months, we’ve been polishing our templates, building out our documentation, and fixing bugs reported by our early adopters. With all these enhancements, Harmony Core should be a Synergy developer’s best bet for adding Synergy web services. Still not sure if this is the right choice for your applications? We’re working on so much more.
Our Plan Going Forward
Because Harmony Core is an open-source project, we want to share not only our technical roadmap, but also our goals for the project. We also want to set expectations up front so you can help keep us accountable and on track.
There are so many things we can do to extend the Harmony Core framework, but we’ve chosen a few large items and a smattering of smaller ones to focus on over the next 12 months. We’ll continue to post blog updates as features are delivered and the direction evolves.
Webhooks are user-defined HTTP callbacks that are usually triggered by some event, such as a status update for an order, or when the quantity of something drops below a certain level. Webhooks are very useful when you have to glue two complex systems together. Imagine your company has grown by acquisition and you find yourself needing to merge the underlying data and business functions of two different IT systems. If you’re trying to build a unified RESTful view of both systems, your view is going to need to be notified when important events occur in the underlying system. Webhooks are the best way to do this. We think this sort of functionality will be crucial in many scenarios. If you have thoughts about how you would like to use this technology, please post on https://github.com/Synergex/HarmonyCore/issues/61. We aren’t certain what shape this support will take, but we will be actively pursuing this. Large (1-3 months)
While we’ve already implemented adapters, there is so much more we can do to extend their uses. Currently, adaptors apply only to filters, but we can further blur the line between operations provided by EF Core and custom code. This would entail implementing additional attributes to indicate that a particular field or collection needs to be expanded. In general, our design philosophy is to flatten OData queries into the sort of data structures that customers currently use to implement custom solutions. Supporting custom Synergy code is a very important part of Harmony Core; many of you have decades of very important business logic that you want to expose to the world. We would be doing you a disservice if the full OData query experience were not applicable to your important business logic. Adaptor enhancements will enable you to take full advantage of uniform OData query syntax, whether you’ve implemented an endpoint with custom code or are using the Harmony Core EF Provider. See https://github.com/Synergex/HarmonyCore/issues/85. Large (1-2 months)
Not all customer data is laid out the way we developers would like it to be. Sometimes this is for historical reasons, or it might be the result of business requirements. Whatever the reason, we want to make sure it’s possible to fully expose your data via OData and EF Core. This means we need to add at least some support for nested subqueries. The main use case we’re thinking of is when data for different branch locations is stored in separate files. In order to join across all branches, we will have to split the query into several parts, storing some intermediate results in a hash lookup table. While this doesn’t amount to support for completely arbitrary subqueries, we think it will solve most customer needs. See https://github.com/Synergex/HarmonyCore/issues/83. Large (1-3 months)
A Migration Path for xfServerPlus
Prior to the conference, we added support for reading xfServerPlus method catalogs. We also added initial support for generating the required wrappers and glue to allow existing xfServerPlus methods to be called using the JSON-RPC 2.0 protocol and the Traditional Bridge portion of Harmony Core. This needs to be extended to support all the data types and coercion that exist in the wild. See https://github.com/Synergex/HarmonyCore/issues/84. Large (1-3 months)
Durable Transaction Log
At the conference we rhetorically asked, “Is this ACID?” Spoiler alert: we’re pretty close but not quite all the way there. The missing piece of the puzzle is a durable transaction log, which is really just a fancy name for an on-disk data structure used as part of a two-phase commit. As transactions are being committed to the underlying ISAM files, Harmony Core would write a log of the operations that it’s performing with information on the state they are in should there be a hardware or software fault that interrupts ISAM operations. In addition to its verification duties, the transaction log would also enable us to undo any changes that were only partially committed. When combined with innovations at the operating system level (such as Storage Spaces Direct in Windows Server 2019) and ISAM resiliency in Synergy/DE 11, we think the transaction log will provide the foundation for highly available, fault-tolerant Synergy applications and web services. See https://github.com/Synergex/HarmonyCore/issues/82. Large (2-4 months)
CodeGen integration with MSBuild – We’ve wanted to improve the process of building projects that use CodeGen templates. The degree to which we rely on CodeGen templates for Harmony Core projects is staggering. Keeping generated code up to date should be a build system operation rather than something for humans to manage (i.e., forget). Small (1-2 weeks)
CodeGen integration with NuGet – Delivering updates to CodeGen templates is tricky and manual. Those are a pair of attributes that should never be associated with tasks that must be performed frequently. Our answer to this problem is to ship template (.tpl) files using NuGet packages. This way, your build system can reference a specific version number for the templates you’re using. If updates are available, all you have to do is bump the requested version, and CodeGen will go download the requested package of templates. Small (1-2 weeks)
.NET CLI scaffolding tool – The command line interface for .NET Core introduced a concept known as scaffolding. Rather than shipping complex logic in wizards, project templates, or item templates, the dotnet tool allows you to offer a scaffolding generator. Conceptually, it’s very similar to a wizard in Visual Studio, but it’s much easier to work with and can be used without Visual Studio. Because Harmony Core relies heavily on CodeGen templates and source conventions, it’s currently more difficult than it should be to add new functionality to an existing project. Once we have a scaffolding tool, it will be much easier to add new endpoints, new structures, custom code in partial classes, and wrappers around existing code. Small (1-2 weeks)
Multiple xfServer connections – Currently the Harmony Core EF provider uses only a single xfServer connection. We need to build additional infrastructure into the provider to maintain a pool of worker threads that each have an open xfServer connection. And the FileChannelManager class will need some additional smarts to deal with request queuing and to use existing threads with requested channels. See https://github.com/Synergex/HarmonyCore/issues/79. Small (1-2 weeks)
Additional configuration options for FileChannelManager and, more generally, the pooling mechanisms – Currently we open file channels and never close them for the duration of a FileChannelManager object. This improves the performance of individual requests, but it’s wasteful in situations where there are hundreds of files, but few that are commonly accessed. The plan for this is to implement a least-recently-used (LRU) eviction strategy. Keeping track of the total list of all files that have ever been opened and the relative frequency of file opens should allow us to auto-tune the number of channels to keep in the LRU list. See https://github.com/Synergex/HarmonyCore/issues/81. Small (1-2 weeks)
Traditional Bridge SSH connection multiplexing – Currently, we open one SSH connection for each Traditional Bridge process on the remote server. For a very large number of processes, this is wasteful because connections are sitting idle most of the time. We can instead create up to 10 sessions per SSH connection (or more if the operating system is configured for it), which would enable us to have many more worker processes without the additional SSH overhead. This would be configurable for cases where SSH connection scaling isn’t the bottleneck. See https://github.com/Synergex/HarmonyCore/issues/80. Small (less than 1 week)
Diagnosing issues in production is one of the most challenging tasks for a developer. We’re considering two features to make this less daunting: per-request logging levels and per-request Traditional Bridge debugging.
Per-request logging levels – Harmony Core offers extensive logging along with a logging framework that works well with Synergy data types. This is great in development, but in production it can destroy performance or collect sensitive data that shouldn’t be logged. By providing a custom HTTP header with a request, the logging level could be set specifically for that request. This means you could get full fat logging from a production server while the enhanced logging is turned off for the rest of your requests. See https://github.com/Synergex/HarmonyCore/issues/77. Small (less than 1 week)
Per-request Traditional Bridge debugging – Similar to per-request logging levels, we can use custom HTTP headers to cause the current request to launch a debuggable Traditional Bridge process. This would cause a separate .dbr and command line to be run on the remote system for this one request, enabling you to attach a debugger from Visual Studio to the running remote process, even in a production environment with thousands of normal requests being serviced concurrently. See https://github.com/Synergex/HarmonyCore/issues/78. Small (less than 1 week)
Harmony Core Goals
What will determine the success of this project? Ultimately, it will be you, the customer. So we want to make sure that implementing and interacting with the web services you build with Harmony Core is as painless as possible.
For 2019, our goal is to have at least four customers that have successfully deployed Harmony Core web services or are in the final stages of developing real-world solutions that make extensive use of Harmony Core. This is not as aggressive as it sounds; we already have four customers in various stages of development, and we aren’t just aiming to get four customers; we want those customers to be happy with the quality of Harmony Core and with the features it provides. To determine if these customers are happy, we’ll keep track of the net promotor score for all the different parts of Harmony Core. For example, if you have a complaint about the documentation or need more help than we expected in order to implement something, that’s a sign that we are missing documentation, examples, educational material, blog posts, or functionality. We’ll need to hear this from you to know how to do better. Once an issue is found, we’ll address it and follow up to determine if the solution actually worked. The number of adopters can’t and won’t be our only metric for success. Additionally, we anticipate that we’ll need to interact with non-technical decision makers and non-Synergy developers who in some cases will know best if Harmony Core is actually solving their business needs.
Open-Source Technology: The New Frontier
We feel that open-source development is a two-way street; in order to be successful, we need non-Synergexian contributors to create pull requests and issues on GitHub. We’re hoping for at least four non-Synergexian contributors to submit pull requests or issues by the end of 2019. The contributions can be for documentation fixes, new samples, or even new core functionality. We will tag GitHub issues that we think are a good fit for external contributors. For a non-tagged issue, we would ask that you give us a heads-up on GitHub to make sure we aren’t currently working on a solution for the issue. In general, we’re looking to add features and functionality that will be applicable to the majority of users, so we won’t want to incorporate code that is specific to a company or line of business.
What You Can Expect from Synergex
When it comes to open-source products, Synergex’s stance has generally been to provide the software as-is with no guarantees. For Harmony Core, however, you can expect Synergex to provide more robust support. Although the development path for Harmony Core is different than for our proprietary products, we want companies of all sizes to feel comfortable taking such a large dependency on Harmony Core. We think this can be accomplished using GitHub issues, but we’re looking to you, the Harmony Core community, to tell us what kind of service-level guarantees your management team will need.
Our Developer Support team is already training up on Harmony Core, and we plan to train them to write and understand CodeGen templates. There are quite a few new areas inside CodeGen that have been added as a result of the Harmony Core project, and you can expect the support team to have up-to-date knowledge on those enhancements. We will also begin looping Support into our ongoing development and deployment efforts with customers. We believe there is no training substitute for extensive guidance while getting your hands dirty on the inner workings of a product, and we think the team will be very well equipped to tackle and triage Harmony Core support cases in the near future.
In addition to providing help though Developer Support, we’re planning on incorporating some educational events and materials to ensure that you can successfully implement and interact with Harmony Core web services. We will be hosting office hours monthly for anyone interested in a chance to hop on a GoToWebinar session and ask questions or work through implementation issues. Office hours are a common feature in open-source projects run by companies who want make sure complex, difficult-to-solve issues aren’t ignored or missed or otherwise languish without solutions. You can also expect regular blog posts recapping the development progress that has taken place, and we’re interested in posting project post mortems (with identifying information scrubbed, of course). We think letting the Harmony Core community see the good, the bad, and the difficult in project implementation will give developers and management confidence that they will also be successful.
Open-source products are usually documented by their creators, and this will be true of Harmony Core. Our tech writing team will be involved (organizing, editing, and helping direct the documentation), but all documentation and samples will come from developers working on Harmony Core. This is a departure from how documentation has traditionally been developed at Synergex, but we think this will be the best way to document this project.
Meet the Harmony Core Team
Jeff Greene, Harmony Core Product Manager
Jeff joined Synergex in 2006 and he brought his passion for open source and standards-based technologies with him.
His first contribution to our product suite was Synergy DBL Integration for Visual Studio, for which he architected and implemented items involving Intellisense, debugging, and migration tools. He championed the importance of running Synergy on .NET Core and was the lead developer of that project.
Jeff has been instrumental in Synergex’s embrace of Service Oriented Architecture (SOA) and standards-based technologies. When the need for a unified solution to creating web services for all Synergy/DE applications arose, Jeff and Steve Ives created the Harmony Core Framework with an emphasis on using standardized technologies such as OData, Swagger, Asp.NET Core, .NET Core and Entity Framework Core.
Jeff has worked with various departments within Microsoft over the years, and has made significant contributions to the open source Microsoft CoreRT project.
Steve Ives, Harmony Core Co-Architect
Following an initial background in operating systems, networking and systems administration early in his career, Steve realized that his real passion and talent was in the field of software development.
Having worked for one of the largest Synergy/DE developers in the United Kingdom, Synergex frequently contracted Steve to present on Synergex locally in Europe. These engagements consisted of presenting on the company and products during meetings, delivering training seminars on the use of the products, and consulting with other customers about how to address specific technology requirements.
Steve formally joined the Synergex Professional Services group in 1997. Having held the position of Senior Consultant for over 20 years, Steve has assisted many Synergy developers to architect, design and implement a wide variety of software projects. Each of these projects involves Synergy/DE in some way, and many involve a wide range of other development languages and technologies.
In recent years, Steve has been a major contributor to several Synergy-related open source development projects, including:
- The CodeGen code generator which can be used to automate the generation of Synergy DBL and other types of code in a wide variety of software scenarios
- The SQL Replication reference implementation—this provides a working example and most of the code needed to replicate a Synergy application’s ISAM and Relative data to a SQL Server database, in near-real-time
- A PDF API that allows Synergy developers to easily add PDF file creation to their applications
- The Harmony Core framework which provides Synergy developers the ability to quickly and easily build RESTful Web Service APIs to expose their Synergy data and business logic to a wide variety of applications and environments
Steve is passionate about assisting software developers to continue to leverage the investment that they have made in their Synergy code base over the years, and to continue to see a return on their ongoing investments in their code.
Johnson Luong, Harmony Core developer
Johnson joined Synergex in 2014 as a QA engineer. He got his start by testing Synergy releases and the Synergex Resource Center Community, as well as verifying trackers and automating the testing of those trackers. You can read about his testing techniques and automated testing at Synergex.
In addition, after moving into a software development role, Johnson made many contributions to our Synergy DBL Integration for Visual Studio (SDI) product including migrating from Windows Forms to the WPF property pages and the product installers from InstallShield to a newer technology. He has since developed all of our Synergy product installers with the open-source language, WiX. He worked closely with Fire Giant, the custodian of WiX, to ensure the Synergy installers are using their most up-to-date technology.
Prior to joining Synergex, Johnson was an active member of the Association for Computing Machinery (ACM) at his alma mater. Along with his team, he even won their biannual International Collegiate Programming Contest. His background in test along with experience in Visual Studio development and solid foundational knowledge of algorithm design make Johnson the perfect addition to round out the Harmony Core team.