Grid Computing in the Investment World: Adapting to the New Cloud Landscape

Traditionally, we’ve seen various companies falter without being able to streamline their business operations or properly handle their workloads. With the growth of cloud technology, some tremendously powerful solutions have surfaced. On top of that, grid computing offers an extra element of elasticity that comes with the underlying technology resources. We have already covered some of the basics in our previous blog post, Infinitely Scalable Clusters, Grid Computing 101. As we explore this topic further, we unveil the benefits of grid computing, cloud tech, and serverless tech.

How Grid Computing Yields Faster Results

Businesses but more crucially financial services are constantly challenged with managing ever growing data sets, in addition to facing massive overheads and processes that are generally time-consuming. As a result, most of these companies are now relying on some form of HPC (high performance computing) grids for valuing portfolios, calculating risk, etc.

There are some key underlying common factors to consider when moving grid computing to the cloud:

  • Scale: anything that has flexible workloads; running small workloads or running a huge workload that scales out across a massive resource, or even not running at all, which leads us into the next factor;
  • Intermittency: either from scheduled batch operations that come into play on a once-a-night, or once-a-week basis or workloads which are triggered by certain external irregular events. You have to admire the simplicity of it – resources are spun up and distributed across the grid only when needed. This leads us into the following factor;
  • Parallelism: this means running operations concurrently, hence the overall problem is split up into smaller pieces so that it can be distributed across multiple worker nodes in parallel streams. The result is numerous nodes working together efficiently towards the answer. That may not be feasible in some workloads as it depends on the underlying logic, code, data and cost/benefit.

In practice, what all of this applies to is kind of irrelevant – so, it doesn’t matter if it’s trading, portfolio analysis, finance, or backend accounting. We’ve even seen it in use cases such as sales and marketing. At the end of the day, grid computing is a powerful weapon only when it improves the business and adds value. Naturally, it all falls into place when you are solving business problems, better, quicker, and at a lower cost. As it’s known from the project management triangle (link) ticking all three boxes is impossible, however this doesn’t seem to be a limitation with cloud grid computing and we’ve seen scenarios where all three have been achieved – better results, at a cheaper rate reducing the time from days to hours.

Valuable Use Cases

From our experience, even though the cloud significantly lowers the barriers to entry into grid computing, the cloud adoption process can be a challenge. We have seen some recurring themes to the choices firms face: what cloud provider to opt for, what data storage to use, what types of compute, how to transition from legacy languages to more modern, data gravity and governance?

There is no single solution for all cases, but we have to focus on the nature of the business, its data and the output it requires out of that data. Then it is a balancing act between the many variables how to achieve this. The more challenging scenarios are migrating existing legacy workloads from on-premise into a hybrid solution between on-premise and cloud. This often requires compromises around data management and compute resources.

Here are some of the approaches we use at Hentsu to solve these challenges:

  • Multi-cloud load balancing across different clouds for resource capacity/cost
  • Or multi-cloud to use specific compute or data management features for certain workloads
  • Abstracting workloads to portable patterns – Docker/Kubernetes, Terraform
  • Serverless workloads, even down to Function as a Service

Discover more about the serverless approach and refactoring legacy applications in the cloud.

More often the migrations are completely away from on-premise, skipping even the hybrid model, simply because of the added complexity dealing with on-premise equipment. There are some approaches to the hybrid model using solutions such as AWS Outposts, Azure Stack and Google Anthos, as well as solutions such as OpenStack.

Spinning Up Resources, and Empowering Users

Various organizations are utilizing different analytic tools on separate clusters, and this can cause a so-called “cluster sprawl.” Cloud computing is something of a turn-key solution in that regard, thanks to various cloud-based PaaS offerings.

“In some ways, the cluster sprawl has been enabled through the ease with which end users can spin up resources and access different cloud platforms. We’ve seen this across clients, with the democratization of technology and empowerment of teams and users. We work to mitigate this cluster sprawl through templating, reusable patterns, and tight ‘business guardrails’ around what users can do. The other key approach we take is having tight configuration management. This can mean anything from basic having awareness of where things are running, to moving the entire platform into code in Git, and we extensively use Terraform to keep on top of that. That helps keep in check of what the infrastructure is and how it is spun up, deployed and used across different cloud providers and teams. At the end of the day, as tech we need balance empower of the users with also getting the right business value as quickly and efficiently as possible” Hentsu CEO, Marko Djukic.

There is a fine line between too much locking down and too much freedom. The cloud tech is always innovating and evolving, which is one of the key attractions of cloud platforms. What used to be great a year ago, is suddenly replaced with something better and evolved, and tech needs to enable getting that into the end user’s as quickly as possible.

Three Key Recommendations for Cloud Grid Computing

  1. Understanding workloads and the target cloud architectures, thus avoiding the many pitfalls of misapplied cloud tech
  2. Knowing the nature and value of your business.
  3. Looking for agile solutions that bring security, scalability, and cost-effectiveness.

As cloud technology grows and evolves, our knowledge expands, and Hentsu can help with challenges you may be facing as you form your own business strategy around the public cloud space.

To learn even more about the advantages Grid Computing, check out full interviews with Marko Djukic and our other Hentsu cloud experts.

Tell us about the technology challenges that you face – we would love to hear from you: