Trading data providers: overcoming the obstacles

Oleg’s diverse experience has allowed him not only to successfully apply approaches used by others, but also to develop his own ways of solving problems. In this interview, Oleg shares his experience in overcoming difficulties with trading data providers.

Q: Thank you for taking the time to discuss EffectiveSoft’s process for developing trading solutions, including the complexities involved and technological strategies utilized. Could you describe the landscape you operate in and the particular challenges you’ve faced?

A: Absolutely, and I appreciate the opportunity. At EffectiveSoft, we’ve created a number of trading systems, including notable projects like City Index and CNote. Through decades of experience, we’ve identified the key issues in trading solutions development. I’m going to touch on some of the challenges these obstacles can create.

One issue is integrating with multiple data providers. Due to differences in coverage, latency, and quality, regularly switching providers exposes the system to the risk of data inconsistencies or interruptions resulting in further instability. Another obstacle is efficiently processing vast volumes of real-time market data for critical determinations. Compliance with constantly evolving regulations in different jurisdictions is also a challenge.

Q: This is a bouquet of thorns. Let’s start from the beginning by focusing on the particularly prickly issue of data providers. Could you expand on the difficulties you’ve encountered in dealing with them and how EffectiveSoft addresses these issues?

A: Of course. As I’ve mentioned, the quality of data providers varies. We’ve encountered providers with incomplete datasets, unreliable service, or inadequate price filtering. Discovering these limitations can be discouraging, especially if you’re halfway through development. These inconsistencies could pose risks and challenges down the line if the data providers are not properly isolated.

So, we’ve learned to tackle this proactively — with microservice architecture. Essentially, we encapsulate each data provider within a separate service. This isolation is key, as it allows us to contain any potential issues exclusively within that provider’s service.

If a provider becomes unreliable, it can be swapped out independently without affecting the rest of the platform. Our standardized data model then ensures a seamless integration of alternative providers.

Q: Working with real-time data at scale is another issue you’ve mentioned. Can you tell me more about how your team is able to efficiently handle a massive stream of calculations, like margin calls or stop losses?

A: I must say that in this regard, the choice of technology and architecture proves vital. We rely heavily on the Go programming language, which supports concurrent operations via goroutines. Processing is broken into concurrent sub-tasks to maximize output. This allows us to process real-time data seamlessly, ensuring that vast volumes of information are digested accurately without performance hiccups. Combined with Kafka’s distributed messaging queues, which create a flexible and scalable data pipeline, this setup creates data loss or mismanagement while providing the high throughput crucial for the real-time financial decisions made on our platforms.

Q: Finally, could you reflect on how EffectiveSoft has advanced over the years in developing trading systems that are robust, compliant, and scalable?

A: With years of iterative improvements and learning, we’ve refined our blueprint for trading systems. The architecture of our trading solutions has evolved to become increasingly modular, reactive, and cloud-native. This enables advanced data querying and analytics for intelligent order routing, market making, and other sophisticated strategies.

Our regulatory reporting capabilities have also expanded significantly alongside new compliance frameworks. Furthermore, our expertise extends to both spot and margin trading and we are adept at managing both A-book and B-book models at scale.

Q: Thank you for such insightful responses.