Conformance
The primary use case for the FixSpec platform is to automate API conformance; something we’ve successfully delivered to everybody from start-ups to large exchanges, and in every asset class.
The precise nature of conformance and the extent to which it is mandated by regulators varies by country and asset class. There is, however, a broad consensus that API conformance is a best practice, especially for the FIX protocol.
Hierarchy
A Conformance is an official test of a customer or vendor’s capabilities.
It consists of one or more Capabilities, each of which represent a meaningful set of business workflows for your particular trading model. How you organise capabilities is up to you (although of course we can help). The first Capability that most platforms set up, however, is one around the FIX Session Layer, which - in turn - is marked mandatory for new customers.
Each Capability consists of one or more Scenarios, which tests a particular sequence of API messages. Continuing our Session Layer example, scenarios may include Logon, Resetting Sequence numbers, and Logout.
Each Scenario is then made up of one or more Steps, which are either:
- Automated steps which look for a specific message exchanged between the applications which meets a set of defined conditions, or
- Manual steps which require manual approval by a team member.
💡 Flexible Scope
When you start a Conformance, we start with the expected scope. Customers rarely develop to the full scope of course, and so the platform allows for easy scope adjustment once this becomes clearer during conformance.
Defining Conformance Services
If we think of a Conformance report as an exam, we need some way of knowing the subject (or spec in our case) that we are examining. This is, of course, a moving target as API documentation is an evolving, versioned concept. To add further complexity, how do we associate a jumbled mass of FIX messages to the correct API document and certification?
The answer is by defining a Service; a small piece of configuration that links a Document to API messages.
The process starts with configuring how API traffic should be allocated to a Document (Services) at the top level. In the case of FIX, this means defining the TargetCompID (56) values, and for REST this is typically the root URL.
ℹ️ Set Service Defaults Then Override
The core Service definition sets the defaults for receiving traffic. Our platform is, however, capable of handling bespoke overrides.
Once in place, you then enable services for customers, restricting the set of SenderCompID (49) values that they will use for this Document (if relevant).
Once these configurations are in place, you are able to start a conformance.
Certification Windows
Certifications are intended to be a point-in-time test of a customer’s capabilities, which means they are naturally time-boxed to a specific (short) window to reduce the risk of clients changing their software during the window and invalidating prior results.
As part of the platform implementation project, we will set the default certification window. We typically recommend a window of no more than 5 days.
All new Certifications acquire this default window when they are created, but you can use the window slider in the Certifications report itself to view results over a longer or shorter timeframe. Using the icon will permanently store an overridden window for the certification.
Tips For Writing Scenarios
We’ve been writing and reviewing conformance scenarios for well over a decade. Here are some golden rules that we always recommend.
- Try to test one specific workflow per scenario as opposed to chaining them together.
- Think carefully about which Scenarios should be considered core (required) within a Capability, and which are optional.
- While there is no limit to the number of Steps a Scenario may have, we discourage more than 4 or 5 steps as it is a symptom of over-complex scenarios.
- Try not to repeat steps across scenarios; limit them to the key interactions required for that test to pass. For example, there is no need to specify an “enter order” step on an order amend scenario - it is implied.
- Unless your test environment needs a very specific trigger for a canned response, be as permissive as possible on fields that do not directly impact functionality under test. For example, most order type scenarios should not need to specify side, price or symbol.
- Defining functional views will significantly simplify your Scenarios, as all of their conditions are automatically applied to the step.
- A scenario can only have one manual Step and it can only be the last Step in the Scenario’s sequence.
ℹ️ Initial Scenario Import
The initial import of scenarios is typically handled by FixSpec as part of a standard implementation; you provide the inputs and we set everything for you before training your team to take over responsibility as your service evolves.
💡 Automatic Chaining
With manual testing, onboarders spend much of their time tracking through linked message “chains” using various identifiers. For example a request to amend an order using a FIX OrderCancelReplaceRequest [G] message links to the original NewOrderSingle [D] message using the fields ClOrdId (11) and OrigClOrdID (41).
Our platform screens for this chaining automatically (including custom linkages), completely removing the need for these conditions to be specified in the scenarios and saving time.
Frequently Asked Questions
The extent to which it is a regulatory obligation varies by country and asset class. It is broadly accepted, however, that API conformance - especially for FIX - is an important risk control for platforms.
Of all the protocols typically found in trading, FIX is the one which is most commonly certified. Part of the reason is simple precedent; FIX always has been certified and so customers now expect it. As FIX knowledge becomes more rare over time, and the more old-fashioned it appears to recent graduates, so the need for conformance grows as human developers of tomorrow may be less experienced and therefore make more mistakes.
Conformance is a risk mitigation control, and so this is largely a judgement call about:
- How confident you are in counterparties’ developer knowledge,
- How confident you are in your API’s in-built validation, and
- Risk associated with the API actions themselves.
Knowledge of REST is now widespread, and a spec-first approach which drives documentation, validation and QA from a complete and accurate OpenAPI will mean (1) and (2) are largely addressed. The most suitable REST workflows are non latency-sensitive workflows which do not impact underlying data, such as retrieving static data or querying status.
While there is always a case for risk controls, we acknowledge that the urgency for REST conformance is lower than for FIX.
If the binary API is proprietary to your platform, then the answer is a clear YES.
If you are using schema-driven binary protocols such as gRPC, Simple Binary Encoding, or Protocol Buffers then the answer is more nuanced. These frameworks consume the digital “spec” and auto-generate software encoder and decoders, which typically guarantee that only well-formed, compliant messages make it through to the receiving application. This process completely eliminates a large class of conformance tests - making sure that applications can correctly communicate using well-formed messages - and leaves only behavioural tests.