Grid Computing: Scale and Automate with Slick Cloud Tools

Cloud technology has already helped define the ‘new normal’ and shape the digital landscape. It’s a way to improve how people communicate, operate and develop within the competitive business environment. By introducing grid computing into the cloud, you are effectively escalating, scaling and generally improving your own businesses operations. With grid computing you get speedy research performance. In other words, farming out large problems to be solved using parallel processing across many worker nodes or running multiple different jobs at once, speeds up research throughput. Also, the elastic capacity of the cloud allows for the freedom to adjust your cloud grid computing resources based on immediate needs. This denotes that you can scale up or down as necessary, of course. In addition you unlock a wide range of powerful cloud tools. Things also become cost-efficient very fast, as cloud solutions eliminate the upfront cost of IT – unlike traditional research computing that may require upfront decisions on the technology and the purchasing of all the required machines before they can be used. Rather than being tied down to one location, your data and applications are available and accessible wherever you are and easily shared across multiple teams and offices. Where do we go from here? Well, for the past few years it has become obvious that there are other methodologies that can enhance the process of running complex workloads. Staying serverless is the focus, but beyond that, we should look at a variety of valuable cloud-powered tools and Kubernetes (or more widely known as k8s). Kubernetes is an open-source system that automates your application deployment, scaling, and management.

Incorporating Cloud Tools: Azure Batch and Azure App Services

The most important facet of serverless compute platforms is giving developers a chance to concentrate on application and script development, versus spending time and energy managing its underlying infrastructure. The goal is to handle huge workloads via a platform that was crafted as an elegant serverless solution; the sort of solution we facilitate regularly here at Hentsu. So, when moving to the cloud environment, the customary method is rewriting the code, thus creating a modernized cloud native stack. For example, batch work can utilize Azure Batch; scripts are turned into Docker containers, which are scheduled onto temporary machines. The key term there is ‘temporary’ – the machines are used and paid for only as much as necessary, with provisioning handled completely automatically. The main drive here is easy management, lower cost, and parallelism. A parallel for application servers is hosting them with Azure App Service, which denotes reliability and even more useful management tools. However, if you are looking to utilize powerful systems such as Kubernetes instead, you must understand certain basics first. Kubernetes is currently one of the most popular methods of running production workloads at scale, but more than that it brings other big cloud-based advantages, as well as vital improvements for enterprises. Kubernetes is great for building containerized and resilient applications.

Kubernetes in the Cloud: 4 Essential Benefits for Enterprises

When it comes to data processing, the huge Kubernetes ecosystem has truly turned things around, offering a variety of ways to reduce its generally high complexity. The immense advantages of Kubernetes may get you to where you want to go, however managing it is a great challenge, especially if you are doing things on-premise, where you may well need someone to manage the infrastructure full-time. In contrast, AWS and Azure can do that bit for you; a reliable, managed cluster is only a few clicks away. Let’s have a peek at the biggest advantages of Kubernetes to any business or enterprise: Scalable When you’re developing applications and running workloads, Kubernetes delegates infrastructure utilization and boasts the ability to scale up in case CPU usage threshold is surpassed. As soon as the workload is reduced, Kubernetes scales back the application. Flexible and Cloud-friendly The massive benefit is being able to utilize several existing tools for cloud-native software. Staying true to the general scalability and flexible nature of the public cloud, Kubernetes is a great addition when resolving issues with clusters and deployments. Apps You Run Are Stable With Kubernetes applications are up and running quite quickly and they are much more stable. In addition, updates and changes can roll out without any downtime. Smooth Cloud Migration Since K8s runs consistently across all environments, on-premise and clouds like AWS, Azure and GCP, Kubernetes provides a more seamless and prescriptive path to port your application from on-premise to cloud environments.

Utilizing Databricks and Other Azure Collaboration Cloud Tools

Azure Databricks is another different and rather vital product for data experts, focused on allowing them to analyze data efficiently and smoothly. One of the biggest additional advantages is having actionable insights, which are understandable to non-data individuals within your company. For instance, if the data team delivers new data, business execs, marketers or sales can go over the data without requiring too much technical knowhow. Benefits for data engineers: a clear major advantage for data engineers is the ability to create, clone and edit clusters of intricate data, which they can transform and deliver effectively for review to data scientists and data analysts. Benefits for data scientists: they run advanced analysis on the same cluster of data in one interface - Databricks auto-scales within the cloud, thus decreasing the resources needed for optimized performance. Azure Databricks is easily integrated with numerous data analysis and storage tools from the Azure library, such as Azure Cosmos DB, Azure Data Lake Storage (ADLS), Azure Blob Storage,  Azure SQL Data Warehouse, or reporting via Power BI which is known for its user-friendly dashboards. Thanks to the elasticity of the cloud no matter how complex the data is, all the gathered information becomes much more accessible for non-data experts.

Adopt Your Own Cloud Sanctuary

Cloud grid computing is the future for research, and it helps companies shift from traditional workstations to the grid, thus leading to lower operational costs, auto-scaling, and speedy research performance. We have already taken a closer look at why grid computing is important for today’s large financial companies (hedge funds, and financial services). To learn even more, feel free to check out our recent post: Infinitely Scalable Clusters: Grid Computing 101. The bottom line is, if you have open-source tools such as Kubernetes by your side, they will no doubt help reorganize your business towards better production. However, the ultimate goal is quality, saving time through automation, and managing things more efficiently. Utilizing reliable tools that are cloud-based, means you have all the heavy lifting done for you. That’s where Hentsu and public cloud services’ features and functionalities come into play, complete with a solid infrastructure foundation.

Date/Time

Date(s) - 01/01/1970
12:00 AM - 12:00 AM

Location

600 5th ave. NY, NY

Grid Computing in the Investment World: Adapting to the New Cloud Landscape

[et_pb_section fb_built="1" admin_label="section" _builder_version="3.22"][et_pb_row admin_label="row" _builder_version="3.25" background_size="initial" background_position="top_left" background_repeat="repeat"][et_pb_column type="4_4" _builder_version="3.25" custom_padding="|||" custom_padding__hover="|||"][et_pb_text admin_label="Text" _builder_version="4.1" background_size="initial" background_position="top_left" background_repeat="repeat"]

Traditionally, we’ve seen various companies falter without being able to streamline their business operations or properly handle their workloads. With the growth of cloud technology, some tremendously powerful solutions have surfaced. On top of that, grid computing offers an extra element of elasticity that comes with the underlying technology resources. We have already covered some of the basics in our previous blog post, Infinitely Scalable Clusters, Grid Computing 101. As we explore this topic further, we unveil the benefits of grid computing, cloud tech, and serverless tech.

How Grid Computing Yields Faster Results

Businesses, but more crucially financial services, are constantly challenged with managing ever growing data sets. Additionally, they are facing massive overheads and processes that are generally time-consuming. As a result, most of these companies are now relying on some form of HPC (high performance computing) grids for valuing portfolios, calculating risk, etc.

There are some key underlying common factors to consider when moving grid computing to the cloud:

  • Scale: anything that has flexible workloads; running small workloads or running a huge workload that scales out across a massive resource, or even not running at all, which leads us into the next factor;
  • Intermittency: either from scheduled batch operations that come into play on a once-a-night, or once-a-week basis or workloads which are triggered by certain external irregular events. You have to admire the simplicity of it – resources are spun up and distributed across the grid only when needed. This leads us into the following factor;
  • Parallelism: this means running operations concurrently, hence the overall problem is split up into smaller pieces so that it can be distributed across multiple worker nodes in parallel streams. The result is numerous nodes working together efficiently towards the answer. That may not be feasible in some workloads as it depends on the underlying logic, code, data and cost/benefit.

In practice, what all of this applies to is kind of irrelevant – so, it doesn’t matter if it’s trading, portfolio analysis, finance, or backend accounting. We’ve even seen it in use cases such as sales and marketing. Ultimately, cloud grid computing is a powerful weapon only when it improves the business and adds value. Naturally, it all falls into place when you are solving business problems, better, quicker, and at a lower cost. As it’s known from the project management triangle (link) ticking all three boxes is impossible, however this doesn’t seem to be a limitation with cloud grid computing and we’ve seen scenarios where all three have been achieved – better results, at a cheaper rate reducing the time from days to hours.

Valuable Use Cases

From our experience, even though the cloud significantly lowers the barriers to entry into grid computing, the cloud adoption process can be a challenge. We have seen some recurring themes to the choices firms face: what cloud provider to opt for, what data storage to use, what types of compute, how to transition from legacy languages to more modern, data gravity and governance?

There is no single solution for all cases, but we have to focus on the nature of the business, its data and the output it requires out of that data. Then it is a balancing act between the many variables how to achieve this. The more challenging scenarios are migrating existing legacy workloads from on-premise into a hybrid solution between on-premise and cloud. This often requires compromises around data management and compute resources.

Here are some of the approaches we use at Hentsu to solve these challenges:

  • Multi-cloud load balancing across different clouds for resource capacity/cost
  • Or multi-cloud to use specific compute or data management features for certain workloads
  • Abstracting workloads to portable patterns – Docker/Kubernetes, Terraform
  • Serverless workloads, even down to Function as a Service

Discover more about the serverless approach and refactoring legacy applications in the cloud.

More often the migrations are completely away from on-premise, skipping even the hybrid model, simply because of the added complexity dealing with on-premise equipment. There are some approaches to the hybrid model using solutions such as AWS Outposts, Azure Stack and Google Anthos, as well as solutions such as OpenStack.

Spinning Up Resources, and Empowering Users

Various organizations are utilizing different analytic tools on separate clusters, and this can cause a so-called “cluster sprawl.” Cloud computing is something of a turn-key solution in that regard, thanks to various cloud-based PaaS offerings.

“In some ways, the cluster sprawl has been enabled through the ease with which end users can spin up resources and access different cloud platforms. We’ve seen this across clients, with the democratization of technology and empowerment of teams and users. We work to mitigate this cluster sprawl through templating, reusable patterns, and tight ‘business guardrails’ around what users can do. The other key approach we take is having tight configuration management. This can mean anything from basic having awareness of where things are running, to moving the entire platform into code in Git, and we extensively use Terraform to keep on top of that. That helps keep in check of what the infrastructure is and how it is spun up, deployed and used across different cloud providers and teams. At the end of the day, as tech we need balance empower of the users with also getting the right business value as quickly and efficiently as possible” Hentsu CEO, Marko Djukic.

There is a fine line between too much locking down and too much freedom. The cloud tech is always innovating and evolving, which is one of the key attractions of cloud platforms. What used to be great a year ago, is suddenly replaced with something better and evolved, and tech needs to enable getting that into the end user’s as quickly as possible.

Three Key Recommendations for Cloud Grid Computing

So, let's examine certain essentials when it comes to cloud grid computing:

  1. Understanding workloads and the target cloud architectures, thus avoiding the many pitfalls of misapplied cloud tech
  2. Knowing the nature and value of your business.
  3. Looking for agile solutions that bring security, scalability, and cost-effectiveness.

As cloud technology grows and evolves, our knowledge expands, and Hentsu can help with challenges you may be facing as you form your own business strategy around the public cloud space.

To learn even more about the advantages Grid Computing, check out full interviews with Marko Djukic and our other Hentsu cloud experts.

Tell us about the technology challenges that you face – we would love to hear from you: hello@hentsu.com

[/et_pb_text][/et_pb_column][/et_pb_row][/et_pb_section][et_pb_section fb_built="1" _builder_version="4.1"][et_pb_row _builder_version="4.1"][et_pb_column type="4_4" _builder_version="4.1"][et_pb_video src="https://www.youtube.com/watch?v=9eHStLqY4go&feature=youtu.be" _builder_version="4.1"][/et_pb_video][et_pb_text _builder_version="4.1" header_font="|||||||#000000|" hover_enabled="0" locked="off"]

 

 Download interview with our CEO Marko Djukić

[/et_pb_text][/et_pb_column][/et_pb_row][/et_pb_section]

Date/Time

Date(s) - 01/01/1970
12:00 AM - 12:00 AM

Location

600 5th ave. NY, NY

Infinitely Scalable Clusters: Grid Computing 101

[et_pb_section fb_built="1" admin_label="section" _builder_version="3.22"][et_pb_row admin_label="row" _builder_version="4.1" background_size="initial" background_position="top_left" background_repeat="repeat"][et_pb_column type="4_4" _builder_version="3.25" custom_padding="|||" custom_padding__hover="|||"][et_pb_text _builder_version="4.1"]

We are currently seeing a major change in the IT industry, and the cloud environment has turned into a safe haven. With a variety of major businesses switching to the public cloud, we have also seen a huge drive towards scaling, and automation. Generally, both the technology and financial sectors are facing a huge shift from having complex computing tasks done manually towards more streamlined and automated processes. That's where grid computing comes in.

When going for cloud adoption, “businesses are pushing PaaS first and that has a lot of positive effects. To begin with, things are made easier right off the bat, because essentially all the building blocks are there, so any initial business can run workloads and get going within a day or two, rather than wasting too much time to set up the foundation,” stated Hentsu CEO, Marko Djukic.

An increasing amount of businesses are implementing grid computing clusters to handle massive workloads.

What is Grid Computing?

Grid Computing refers to making use of the shared power of a cluster of computers to process computationally intensive tasks that would otherwise bog down a single workstation. To put it simply, jobs can be submitted from one computer to the grid, which then processes the data and returns the output to the user. Importantly, multiple people can simultaneously make use of the same grid by intelligently managing how the computers in the grid allocate resources. This allows for significantly improved workflow for companies whose work is optimized for grid computing.

Opening with a much needed refresher on public and private cloud, as well as defining key terminology, the talk flowed into an investigation of the challenges companies face when working with different data sets and a look at solutions currently available to companies.

How Grid Computing Works - Grid Architecture Explained

To recap, grid computing denotes one main computer that distributes information and tasks across multiple networked computers – all of that usually towards one objective. Let us simplify a bit and illustrate how things operate. Grid computing network frequently has three types of machines:

  • Central Control Node - Server/computer or a group of servers working to controls the entire network and preserving the account of the resources within the network pool. Again this part is used for controlling and not for processing.
  • Provider - A computer working to give resources within the network.
  • User/Client – In basics terms, clients or users can actually utilize the computer on the network regardless of geographical location.

Key Advantages of Grid Computing

One of the key aspects of grid computing is flexibility, and more importantly, computing power. In other words, it boils down to having a single computer grid for large amounts of data rather than placing the demand on a single supercomputer. You can also find out how Grid Computing yields faster results for your business. 

But for now, let's focus on the most frequently asked questions related to grid computing.

Why grid is computing important?
What are the real benefits and biggest advantages of grid computing?

Well, the reasons why so many businesses rely on this particular method of completing joint tasks because it denotes following key advantages:

  • Improved use of existing hardware.
  • General performance increase and quicker handling of complex problems.
  • Easier to collaborate with other organizations.
  • Massive servers for applications can be divided into smaller commodity type servers.
  • With grid environments lead to reduced chances of failure - if one desktop/server fails, other resources pick up the workload.
  • Jobs can be executed in parallel speeding performance.

The Agility of Ephemeral Computing

It has to be highlighted that ephemeral computing has also been a huge part of the innovation process in modern-day tech. It carries tremendous advantages, albeit we have to ask the simple and most obvious question here: what does that mean for businesses and the SaaS industry in particular?

When you describe a process as “ephemeral” it denotes something temporary and brief. Essentially, the notion of dealing with a surplus of servers or indeed a shortage of servers is something that’s not an issue with ephemeral computing. To put it into perspective, ephemeral computing services are agile and will adjust according to the problems and needs at hand.

The Power of Code and Automation

Utilizing ephemeral clusters that scale up and down as needed is quite a boost to handling workloads in general. In short, it means you limit or eliminate convoluted pre-planning and relying on heavy server power.

“It’s basically all about serverless. Code that is distributed across ephemeral compute that handles the analysis and then churns out the answers without having to deal with what’s actually the underlying compute,” says Marko Djukic.

He added: “Compute is just a utility you consume as needed, nothing exists permanently."

To summarize, in PaaS and SaaS scenarios, giving your business operations and workloads the power to shrink automatically, can completely remove any scalability issues you may be experiencing with traditional server-heavy computing methodology.

Traditional tools and PaaS Offerings

There is a breadth of choices for grid computing and how to migrate workloads into cloud environments. We covered some of those, looking at traditional MATLAB setups to more extreme Platform as a Service (PaaS) environments from Google using their Bigquery and Datalab, and ran some live demos ripping though 2TB of full depth market data.

Key points to take away:

  • Horses for courses - What spec’d machine does your task work best in? Many cores in fewer machines, or many smaller cores on multiple machines? Consider how best to configure your cluster.
  • Reduced upfront costs - Unlike traditional Grid Computing clusters that require upfront purchasing of all the machines needed before they can actually be used, cloud solutions let you skip out on paying for hardware.
  • Flexibility - Publicly available solutions can allow you to quickly scale the size and power of your cluster to ensure that you can crunch the data in the time you need.
  • Reducing worker downtime - Having employees sitting around waiting for their code to execute is wasting time and money. By pushing tasks off individual computers and onto the cloud, the bottleneck is alleviated and workers can continue with work.
  • Full Platform as a Service (PaaS) - Allows some very dynamic and fast access to compute across vast data sets, but will usually require significant re-tooling for major hedge fund production environments. When implementing PaaS type grid computing this also involves a more holistic approach across people, processes and technology.
[/et_pb_text][/et_pb_column][/et_pb_row][et_pb_row _builder_version="4.1"][et_pb_column type="4_4" _builder_version="4.1"][et_pb_text _builder_version="4.1"]

How Can Hentsu Help?

We've built up a wealth of in-house expertise running grid computing workloads across all three major public clouds - Amazon AWS, Microsoft Azure and Google Compute Engine. We can get you up and running quickly with pre-tested designs and architectures, greatly eroding the overall traditional pains and TCO for running grid computing.

[/et_pb_text][/et_pb_column][/et_pb_row][et_pb_row _builder_version="4.1"][et_pb_column type="4_4" _builder_version="4.1"][et_pb_text _builder_version="4.1"]

Get in touch: hello@hentsu.com, we'd love to hear about your grid computing challenges.

[/et_pb_text][et_pb_video src="https://www.youtube.com/watch?v=zNDkFyVxYc4" _builder_version="4.1"][/et_pb_video][et_pb_text _builder_version="4.1" header_font="|||||||#000000|" hover_enabled="0"]

 

 Download interview with our CEO Marko Djukić

[/et_pb_text][/et_pb_column][/et_pb_row][/et_pb_section]

Date/Time

Date(s) - 01/01/1970
12:00 AM - 12:00 AM

Location

600 5th ave. NY, NY

Embracing Cloud for your Enterprise

It appears that just about any technology you use in your personal life touts some aspect of “cloud”. From Google Drive and DropBox to iCloud and OneDrive, companies are allowing users to wizard away their data to these services that magically don’t exist within their personal devices. One may assume that cloud services are simply storage solutions where you can move things into the safekeeping of a company. But of course, this is just the tip of the cloud iceberg. At Hentsu the cloud is our business so we’ve written a few tips to help clear up some of the confusion regarding what else a “cloud” service can be.  
  1. Grid Computing: Fancy speeding up number crunching and relieving your poor hardware from stressful workloads? Then move your work off to a grid computing service! These clusters of computers share a workload between themselves, freeing up your machine’s capacity. Services such as Google’s BigQuery and DataLab allow for large sets of data to be processed and analysed with all the power Google can muster, in a cost-effective manner. Provisioning compute clusters in the cloud has far less set up time and upfront costs compared to their in-office counterparts.
  2. Software as a Service (SaaS). Need some software now? You may not need to download it! Some companies offer fully functional software from within browsers or via remote connection. Office365 and similar services allow users to create Office documents from their web browser or phone without downloading anything to their machines. These documents are even backed up online and easily sharable with URL links, allowing multiple people to edit the same document at once. SaaS isn’t limited to documents, anything from monitoring tools to communication can be provided as a cloud based service.  Better still, the software provider will take care of all maintenance and updates.
  3. Infrastructure as a Service (IaaS). Companies and individuals don’t always have the right hardware they need to operate as desired. Hardware also represents a big upfront expense and a maintenance burden. How about you let someone else take care of all that? Whereas SaaS allows you to make use of software not on your machine, IaaS lets you make use of infrastructure you don’t physically have. Companies such as Amazon and Microsoft have created professional grade virtual networking environments where you can deploy virtual machines and large data stores without the overhead of maintaining a datacentre.
  So yes, the cloud can be where you store and share your photos, but it can be so much more. Here at Hentsu we specialise in cloud technologies and have a wealth of experience helping organisations move their operations from traditional platforms to the cloud. If you think your business may benefit from any aspect of the cloud then feel free to get in touch, we can ensure you end up with the services right for you.

Date/Time

Date(s) - 01/01/1970
12:00 AM - 12:00 AM

Location

600 5th ave. NY, NY

AWS for Hedge Funds

Cloud infrastructure is ready for prime time - AWS for hedge funds is rapidly becoming "the new normal" no-brainer, providing an enabling technology platform to address many of the business challenges faced by managers. Running a hedge fund has never been more challenging. To start with, markets have been very unpredictable. Central bank policies across the world have played a key role in driving (or arguably distorting) asset prices, not only in fixed income and currency markets but also in other asset classes. The competition has also evolved. The rise of the ETF industry has made it cheaper for investors to track benchmark indices, and even introduce some degree of model-based active management through intelligent beta finds. At the same time, an industry that historically has been unregulated is now coming under the watchful eye of financial watchdogs – triggering a corresponding increase in compliance costs for fund managers and operational burdens spanning a range of activities, from registration through to record-keeping and reporting. So with alpha discovery proving more difficult, the pressure to justify fees coming from cheaper investment vehicles, and costs on the rise because of new compliance requirements – smarter and more agile use of technology is more critical than ever. Sadly, many fund managers and vendors are still bogged down with legacy systems and solutions that are complex and inefficient to manage. They are under the illusion that owning and operating one’s own server and network infrastructure offers some form of an advantage when the opposite is in fact true. What’s needed is a new approach; an agile way of solving fund manager problems using new technology architectures. That is why we founded Hentsū. Our vision is to deliver hedge fund technologies that are agile, secure, compliant, scalable, resilient, and cost-effective. Building solutions that maximize the benefit offered by cloud infrastructure providers like AWS, for hedge funds.

3 Quickfire Benefits of AWS for Hedge Funds and Financial Services

Here's a quick overview of the public cloud benefits for financial services and hedge funds: Grid Computing Similarly to financial services firms, hedge funds can utilize AWS to expand on-premises grids to the public cloud. As an alternative, they can build cloud-native grids. The objective here is to right-size compute for factors such as risk and stress testing, trading strategy development, and portfolio management. The result is gaining the adaptability to regulatory changes. Other crucial benefits are getting more precise risk insights, quick responses to market events, and so on. What is Grid Computing? ML (Machine Learning) When utilizing AWS for a hedge fund business, Machine Learning can be a huge component to drive things forward. AWS includes various machine learning (ML) services, which focus on digital transformation. The result is improving algorithmic models, portfolio management, and data analysis. ML in Trading Modernizing Legacy Systems with Agility and Scale Public cloud services such as AWS, help hedge funds, and financial services modernize their legacy systems (and applications), as they wade through the dynamic market. With the tools and services offered within the AWS suite, these companies can accomplish attain better operational flexibility, while decreasing cost substantially. Refactoring Legacy Applications If those qualities are what you look for in your IT systems, get in touch. Chart your own cloud adoption strategy and roadmap by discovering more valuable benefits of the public cloud.

Date/Time

Date(s) - 01/01/1970
12:00 AM - 12:00 AM

Location

600 5th ave. NY, NY