It appears that just about any technology you use in your personal life touts some aspect of “cloud”. From Google Drive and DropBox to iCloud and OneDrive, companies are allowing users to wizard away their data to these services that magically don’t exist within their personal devices. One may assume that cloud services are simply storage solutions where you can move things into the safekeeping of a company. But of course, this is just the tip of the cloud iceberg. At Hentsu the cloud is our business so we’ve written a few tips to help clear up some of the confusion regarding what else a “cloud” service can be.
Grid Computing: Fancy speeding up number crunching and relieving your poor hardware from stressful workloads? Then move your work off to a grid computing service! These clusters of computers share a workload between themselves, freeing up your machine’s capacity. Services such as Google’s BigQuery and DataLab allow for large sets of data to be processed and analysed with all the power Google can muster, in a cost-effective manner. Provisioning compute clusters in the cloud has far less set up time and upfront costs compared to their in-office counterparts.
Software as a Service(SaaS). Need some software now? You may not need to download it! Some companies offer fully functional software from within browsers or via remote connection. Office365 and similar services allow users to create Office documents from their web browser or phone without downloading anything to their machines. These documents are even backed up online and easily sharable with URL links, allowing multiple people to edit the same document at once. SaaS isn’t limited to documents, anything from monitoring tools to communication can be provided as a cloud based service. Better still, the software provider will take care of all maintenance and updates.
Infrastructure as a Service (IaaS). Companies and individuals don’t always have the right hardware they need to operate as desired. Hardware also represents a big upfront expense and a maintenance burden. How about you let someone else take care of all that? Whereas SaaS allows you to make use of software not on your machine, IaaS lets you make use of infrastructure you don’t physically have. Companies such as Amazon and Microsoft have created professional grade virtual networking environments where you can deploy virtual machines and large data stores without the overhead of maintaining a datacentre.
So yes, the cloud can be where you store and share your photos, but it can be so much more. Here at Hentsu we specialise in cloud technologies and have a wealth of experience helping organisations move their operations from traditional platforms to the cloud. If you think your business may benefit from any aspect of the cloud then feel free to get in touch, we can ensure you end up with the services right for you.
On Wednesday the 2nd of November Hentsū hosted keen minds from around the financial technology sector for a talk about both the benefits and implementation of grid computing clusters. This talk aimed to give people a taste of what Grid Computing has to offer by:
Discussing how they operate
Providing worked examples of Google Datalab and MATLAB Distributed Computing toolkit
What is Grid Computing?
For the uninitiated, Grid Computing refers to making use of the shared power of a cluster of computers to process computationally intensive tasks that would otherwise bog down a single work station. Jobs can be submitted from one computer to the grid, which then processes the data and returns the output to the user. Importantly, multiple people can simultaneously make use of the same grid by intelligently managing how the computers in the grid allocate resources. This allows for significantly improved workflow for companies whose work is optimised for grid computing. Opening with a much needed refresher on public and private cloud, as well as defining key terminology, the talk flowed into an investigation of the challenges companies face when working with different data sets and a look at solutions currently available to companies. This investigation, backed up by the live demonstrations, left those who attended with a solid understanding of what Grid Computing can do for them.
Traditional tools and PaaS Offerings
There is a breadth of choices for grid computing and how to migrate workloads into cloud environments. We covered some of those, looking at traditional MATLAB setups to more extreme Platform as a Service (PaaS) environments from Google using their Bigquery and Datalab, and ran some live demos ripping though 2TB of full depth market data.
Key points to take away:
Horses for courses - What spec’d machine does your task work best in? Many cores in fewer machines, or many smaller cores on multiple machines? Consider how best to configure your cluster.
Reduced upfront costs - Unlike traditional Grid Computing clusters that require upfront purchasing of all the machines needed before they can actually be used, cloud solutions let you skip out on paying for hardware.
Flexibility - Publicly available solutions can allow you to quickly scale the size and power of your cluster to ensure that you can crunch the data in the time you need.
Reducing worker downtime - Having employees sitting around waiting for their code to execute is wasting time and money. By pushing tasks off individual computers and onto the cloud, the bottleneck is alleviated and workers can continue with work.
Full Platform as a Service (PaaS) - Allows some very dynamic and fast access to compute across vast data sets, but will usually require significant re-tooling for major hedge fund production environments. When implementing PaaS type grid computing this also involves a more holistic approach across people, processes and technology.
We've built up a wealth of in-house expertise running grid computing workloads across all three major public clouds - Amazon AWS, Microsoft Azure and Google Compute Engine. We can get you up and running quickly with pre-tested designs and architectures, greatly eroding the overall traditional pains and TCO for running grid computing. Get in touch:firstname.lastname@example.org, we'd love to hear about your grid computing challenges.