DevOps Playground #23: Hands-on with AWS Serverless

DevOps Playground #23: Hands-on with AWS Serverless

No comments

In most IT departments you usually have an individual who’s responsibility it is to manage the servers. When your software or website has a particularly high load they can rearchitect the site to manage it.

“In essence, “serverless” is a computing model where the existence of servers is hidden. So you don’t need to worry about servers at all. All you have to worry about is  the function and programming.” – Sunil Tailor

Serverless computing is a cloud-computing execution model in which the cloud provider acts as the server, dynamically managing the allocation of machine resources. Pricing is based on the actual amount of resources consumed by an application, rather than on pre-purchased units of capacity.[1] It is a form of utility computing.In this DevOps playground, Sunil will be getting hands on with AWS’s serverless technology.

Interested in attending our next DevOps Playground events. Follow us on Meetup to receive a notification about the next event

Sunil TailorDevOps Playground #23: Hands-on with AWS Serverless
read more
DevOps Playground Singapore #2: Hashicorp Consul & Smashing

DevOps Playground Singapore #2: Hashicorp Consul & Smashing

No comments

Following the success our last DevOps Playground in Singapore, the team were at it again with another playground showing the power of service discovery and monitoring using Hashicorp Consul.

A typical DevOps Engineer could be responsible for the maintenance of many services in a DevOps pipeline such as a build server, binary storage solution, a a code repository service, wiki and a ticket tracking service.

High agility and a mature DevOps capability require these services to be running 100% of the time. An outage of any service will impact the route to live of any change. People will be walking up to your desk, wondering what is going on!

Using Hashicorp’s Consul, we can intelligently monitor the status of these services and react in real time to unpredicted behaviour. You can therefore identify the issue before the rest of your end users do.

However, there will be occasions when outages require time to fix. How do you keep your end users informed without having to update them on a continual basis? Display this information from Consul in an easy to view dashboard on TVs around your office using Smashing.

It looks smart, tidy, and provides everyone the information they need so you can be left to bring the services back on line in the quickest time possible.

At the event, we went through the steps to install and run Hashicorp Consul and registered both our Jenkins and Artifactory services by configuration a service definition file and loading that into Consul. We validated the health check feature of Consul by taking Jenkins offline and seeing this outage being reflected in the User Interface.

We then installed a Smashing Dashboard and saw how easy it was to post updates to Smashing by executing a simple curl command. With all the core pieces in place, we implemented a watch that pushed an update to Smashing whenever a service went offline.  This result in an eye pleasing dashboard that instantly highlighted the status of your pipeline in real time.

 Following this video you should be able to run through this playground from the comfort of your own home.

Thank you to everyone who attended. We look forward to seeing you again at our next DevOps Playground – keep an eye out for the next event!

ECS DigitalDevOps Playground Singapore #2: Hashicorp Consul & Smashing
read more
Open source. Are you part of the community?

Open source. Are you part of the community?

No comments

Open source is a type of licensing agreement – not very exciting. The exciting bit is that it allows users to create and publish work that can be freely used, modified, integrated into larger projects or derived into new work based on the original by other users.

In an age of trade secrets and profit-driven professions, this is a unique platform that actively promotes a global exchange of innovation. It has been specifically designed to encourage contributions so that the software doesn’t stand still. The collective goal of this barrier-free community is the advancement of creative, scientific and technological tools and applications – which for many is more important than a price tag.

Who uses open source?

Although, it is most commonly used in the software industry, professionals adopt open source licenses in many industries including biotech, fashion, robotics and teaching. This article will focus solely on software applications.

What’s interesting is that more and more businesses are contributing their own source code to the community – Facebook, Airbnb, Cyprus are leading examples. According to a 2018 Tidelift Professional Open Source Survey, 92% of projects amongst European respondents contain open source libraries. Whilst on the surface this contradicts conventional commercial instinct, businesses gain a lot by giving away a little. Whilst the benefits are vast, we are going to focus on five:

  1. Competition:

Since the late 90’s and the advancement of the digital age, competition no longer resides simply between two rival companies. Businesses today also find themselves competing with open source software projects that are free, open to the public and constantly evolving.

Due to the current scale of open source contribution, even the giants in the tech industry are struggling to devote the resources or teams large enough to compete with their community counterparts.

Turning to the open source community enables businesses to outsource resource rich projects to a bottomless sea of innovative capabilities. This potentially reduces cost, pressure and speeds up the feedback loop considerably.

  1. Reputation:

In the same way the Big Bang Theory made traditional science nerds cool, the open source community can boost a business’ profile on the cool/not-cool spectrum.

Not only do businesses become more attractive to potential employees, by initiating an open source software project, or contributing to an existing one, they make their mark on an additional and power channel popular within IT circles. If done well, this has the potential to establish, maintain or improve a brand’s image, as well as attract new business.

  1. Advancement:

Helping to advance something as big as the technology industry isn’t something to turn your nose up at. In fact, businesses revel with the idea of having their name against a leading piece of software that has the potential to make history.

But history moves fast. And building software inhouse can be stifled by other business priorities, resource restrictions and other competitors beating you to the finish line.

Rather than building behind closed doors and waiting until your software it is perfect, opening your source code to the community in its earlier stages has two benefits:

  1. You can plant your flag earlier
  2. You invite an endless list of innovative capability to help advance your idea at a rate unlikely attainable behind closed doors

It also acts as an incentive for individuals to feel part of a project than extends far beyond the business they work for.

  1. Trust:

Fake news, data breaches, shady deals – all of these have encouraged people to lose trust in businesses. Including open source projects in company policy encourages business to be more transparent with its consumers. Whilst it is naive to believe a company will lay down all their cards, companies such as Facebook made 15,682 contributions in 2016. Automattic created WordPress as an open source project and currently powers 31% of the internet, and Netflix frequently open sources the tools they develop in-house.

Not only are they strengthening their brand, sharing is showing the world they have nothing to hide – which is a proven way to start winning back trust.

A great example of building this trust through transparency is the cryptocurrency space where many projects including Bitcoin allow you to browse through the project’s source. A very different approach to their corporate counterparts.

  1. Speed:

Many companies face the same problems. Sometimes companies are kind enough to share the solution. If a problem has been solved before and will provide business value in a fraction of the time and half the man power everybody wins.

Contributing to the community also gives you the capability to ask the projects contributors directly questions, ask for features or raise issues enabling you fast feedback which keeps your project moving

How does open source work? 

Contributors create a project and solve a problem. They realise that other people might benefit from this project to solve their problems. The project is shared on an open platform such as GitHub which can be downloaded and used by other users interested in the project.

If users wish to contribute, they can do this by downloading the project, creating a fork (which is an exact replica of a certain part of the pipeline) and editing the code until they are happy with the changes. Users can then request a pull request which notifies the authors that a suggested change is requested.

It is up to the author to approve the change, before deciding whether they want to include the changes. If they do, this usually becomes part of the next version, which is released at the author’s discretion.

The problem is, this could take some time. The author is under no obligation to release new versions or accept proposed changes. In fact, this is one of the limitations of the open source community. People will only give up as much information as they want to / their projects need. Authors are not there to solve specific problems, and often release software that focuses on their needs rather than trying to create something too generic.

This can be frustrating if an open source project only solves half your problem, however, the community can help bridge knowledge gaps. Users also have the option to download, build and run the project locally in the interim whilst waiting for the official new version – meaning they don’t need to wait for the software to be released with the changes they need.

How it is viable?

Whilst it doesn’t make economic sense on the surface, the community have found a way to make open source viable from a business and individual perspective. Some have capitalised on their projects, making basic versions available at no cost to the user, but adding a price tag to different versions or ‘add-ons’.

Other businesses or individuals actively contributing to the platform have benefited from angel investments, as well as new business after demonstrating successful projects.

It is also often a side project for businesses and individuals. Due to the legal freedom attributed to an open source platform, you’re able to modify the code of the product you’re using endlessly, for free, at no risk of breaching privacy policies or user agreements. This makes it the opportune ‘playground’ for those looking to get into the industry or develop new skills. According to LinkedIn:

“We believe that open sourcing projects makes our engineers better at what they do best. Engineers grow in their craft by having their work shared with the entire community.”

Risks:

With all open platforms, there is a risk of abuse. Open source communities are no different and have certainly experienced their fair share of malicious activity. However, it is the open source approach that significantly increases the reliability of the projects available to the public.

By establishing a community who believe in the future potential of the projects produced, you immediately have a security indicator in place. Many of them in fact. And with so many eyes looking at projects, malicious activity is quick to be spotted and remedied. This is because open source platforms embody an agile mentality, applied in a community wide approach. Rather than make one big change and focus on ensuring it is okay for the next six-months, contributors and authors are interested in making changes quickly, so things get fixed and evolved just as quick.

******

ECS Digital love to find value for our clients and give it back to the wider community, which is why we make tools available on open source platforms such as GitHub and NPM.

We will also be hosting a hands-on session and demonstration of AyeSpy– a visual regression testing tool – at an upcoming DevOps Playground on the 29th of November. Come along to learn more about what the AyeSpy has to offer!

Matt LowryOpen source. Are you part of the community?
read more
Our first DevOps Playground in Singapore

Our first DevOps Playground in Singapore

No comments

Our very first DevOps Playground took place in September in Singapore, at the Lifelong Learning Institute. We had fun with Amazon Connect!

At the event, we created a working solution utilising Amazon Connect, Amazon Lex, AWS Lambda, S3 and SNS. Cloudformation was used to help configure these services with the minimum of effort. Unionising these Amazon services allowed us to create an environment that enabled us to have an interactive telephone conversation through Amazon Connect, powered by Amazon Lex. Once the call was complete, the conversation was transcribed using AWS Lambda and sent via SNS to a subscribed email.

Watching the video below, you can run through the Playground from the comfort of your home!

Thank you to everyone who attended and made our very first DevOps Playground in Singapore a huge success.

Cy NichollsOur first DevOps Playground in Singapore
read more
DevOps Playground #21 – Google Kubernetes Engine on Google Cloud Platform

DevOps Playground #21 – Google Kubernetes Engine on Google Cloud Platform

No comments

Our 21st DevOps Playground took place last evening in Edinburgh, in our ECS office. We explored Google Kubernetes Engine on Google Cloud Platform.

We created a GKE cluster, using preemptible instances, then created an application and built a Docker image from it, which we pushed to the Google Container Registry. We then ran this image on our GKE cluster.

We did run into some issues with the limits on Google Cloud, be warned, you need to upgrade your account or you will be limited to 1 Kubernetes Cluster. Lesson learned! 😊

The remaining of the meet-up was to run some load testing with Locust, playing around with the performance of the application, allowing everyone to see how GKE auto-scaled the cluster and the deployment. Lowering the performance of the app lead to pods being created by the deployment, and as the number of pods became too large for the cluster, the cluster itself scaled, growing from 3 to 7 instances. Improving the performance of the app then lead to the cluster being able to scale back to only 3 instances.

All the changes to the application were made using rolling updates, minimising the negative effects of changing an application version while serving live traffic.

 

 

 

 

 

 

This was our busiest meetup so far in Edinburgh, and we are looking forward to meet with everyone again next month, in our ECS office near Haymarket.

Register for our next DevOps Playground in Edinburgh here, and in London here if you are around.

Interested in attending one of our DevOps Playground events? Follow up on Meetup to receive a notification about the next event – Join us!

 

Michel LebeauDevOps Playground #21 – Google Kubernetes Engine on Google Cloud Platform
read more

DevOps Playground #18 – All Hands on Prometheus

No comments

Earlier this year we had our 18th DevOps Playground focusing on the very powerful monitoring tool – Prometheus, that has been gaining a lot of attention in recent years.

At the event, we first looked at a brief history of Prometheus, then our participants learned how to setup and start a Prometheus server, configure targets and basic alerts. We’ve also looked at PromQL, the powerful query language, and learned the most basic functions, and expressions we can use to get started on using Prometheus.

Following this video you should be able to run through this introduction to Prometheus from the comfort of your home!

For the next playground we will be focusing on managing micro-services on Kubernetes using Istio.

We’re looking forward to seeing you there!

Interested in attending one of our DevOps Playground events? Follow up on Meetup to receive a notification about the next event – Join us!

Daniel MeszarosDevOps Playground #18 – All Hands on Prometheus
read more
DevOps Playground #20 – Jenkins & Kubernetes

DevOps Playground #20 – Jenkins & Kubernetes

No comments

The 20th DevOps Playground took place at the ECS Digital offices on Bermondsey Street. The focus for the event was a hands-on demonstration of running a scalable Jenkins cluster on top of Kubernetes, provisioning pods on demand, containing a Jenkins slave image intended to execute a given pipeline.

The key points covered in this DevOps Playground were:

  • Custom Jenkins Docker image preinstalled with recommended plugins and more.
  • Creating Deployments and Services in Kubernetes.
  • Deploying a Jenkins master node to the cluster.
  • Configuring the master node to be able to provision containers.
  • Defining pipelines utilising the Jenkins Kubernetes plugin.

The event was a nice introduction to both tools, as well as it being a useful real-world scenario. The feedback from the attendees was very positive, with many of them not having much Kubernetes or Jenkins experience.

Overall, the event was a huge success and it was great to have the opportunity to meet many of you, and give back to the community.

We are looking forward to seeing you in May at our next meet-up; topic and dates to follow shortly!

Interested in attending one of our DevOps Playground events? Follow up on Meetup to receive a notification about the next event – Join us!

Anthony ForsterDevOps Playground #20 – Jenkins & Kubernetes
read more
DevOps Playground #19 – AWS CloudFormation

DevOps Playground #19 – AWS CloudFormation

No comments

The 19th DevOps Playground took place at the ECS Digital office’s in Bermondsey Street. The topic was AWS CloudFormation with a focus on Infrastructure as Code (IaC). This was my second DevOps Playground, last year I ran a meet-up showing off Serverless with AWS Lambda, available here.

The playground was composed of a number of sections, with an introduction to Infrastructure as Code, with labs to create S3 bucket to launching an EC2 instance and managing security groups. The idea behind this DevOps Playground was to introduce the basic fundamentals with enough hints for further deep dive studies people could make on their own, using the templates and links mentioned in the labs.

There was a special mention of what’s coming next in the CloudFormation in the form of Configuration Drift, which was mentioned and previewed at Re:Invent 2017.

DevOps Playground 19

Some of the keytake aways from this DevOps Playground were:

  • CloudFormation Fundamentals
  • Service Roles
  • Creating & Updating Stacks
  • Using Change Sets
  • Launching EC2 instances with bootstrap scripts
  • Creating and Updating Security Groups

The feedback from the meet-up was very positive throughout the evening, ending with pizza, beer and networking. Overall, another successful DevOps Playground where everyone present had a good time, and hopefully learned something new.

We are looking forward to seeing everyone again at our next meet-up: Hands on with Kubernetes and Jenkins.

Interested in attending one of our DevOps Playground events? Follow up on Meetup to receive a notification about the next event – Join us!

Sunil TailorDevOps Playground #19 – AWS CloudFormation
read more
Our first DevOps Playground in Edinburgh – Hands on with AWS Fargate

Our first DevOps Playground in Edinburgh – Hands on with AWS Fargate

No comments

On the 27th February, we had the privilege of launching ECS Digital’s very first DevOps Playground in Edinburgh. Such an occasion called for a special topic, and this time we focused on AWS Fargate, a brand-new and often misunderstood technology from Amazon Web Services.

For those who don’t know, DevOps Playground is a small meetup run by ECS-Digital consultants to give people a hands-on practical introduction to DevOps technologies and help to evangelise them.

 

So, what is Fargate?

Picture1-12.png

Amazon Web Services defines it as a ‘technology that allows you to run containers without managing the underlying instances’ – this means that by using Fargate as launch type, your container will be downloaded and launched in some physical place inside the AWS datacenters, away from your direct management.

The Playground

As is tradition at DevOps Playground, we prepared a hands-on example to give the audience a taste of the functionality of this technology. There was a brief introduction to AWS Elastic Container Services, with an explanation on the logical division between Clusters, Services and Tasks. Then we proceeded to create the Cluster, register the container specifications (Task Definitions), and finally run a service. A single instance of the famous Ghost blog engine.

The foundation that makes all of this possible is the new networking mode called awsvpc, which allows us to attach an Elastic Network Interface to a container, rather than to an EC2 Instance. This gives us direct access to container services, rather than the underlying hosts, in turn making Fargate possible.

Limitations

Of course, since Fargate is pretty new, there are some limitations on it’s usage. Amongst the most important are the following:

  • Currently, it’s available only in one region: us-east-1. This is because it has limited availability at the moment and us-east-1 is considered the experimental region.
  • It is not possible to attach persistent EBS volumes, directly to the container for now .
  • Limits of 20 Fargate services, per account, per region
  • Limits of 20 public IP addresses per Fargate services, per account, per region.

Reception

Despite the heavy snowstorm, we had a good turnout with many questions in between the presentation and after that led discussions regarding containers, service management and related problems. It was great to talk to passionate engineers and gain an insight into the tech environment of Edinburgh.

All in all, it was a fantastic experience and I feel that there will be a lot of potential in the Scottish capital for this new technology in the future.

Looking forward to the next DevOps Playground in Edinburgh.

Resources

In case you want to play with Fargate, We publicly released the walkthrough we executed at the meetup. You can find it here, in our public Github repository.

For more information about DevOps Playground in your city click below:

Interested in attending one of our DevOps Playground events? Follow up on Meetup to recieve a notification about the next event – Join us!

Enzo RivelloOur first DevOps Playground in Edinburgh – Hands on with AWS Fargate
read more
DevOps Playground #16 – Kubernetes on Azure

DevOps Playground #16 – Kubernetes on Azure

No comments

We ran our 16th DevOps Playground which took place in our offices at London Bridge. The focus of this month’s event was Kubernetes on Azure.

At the event, participants looked at how they can use Azure CLI to create an Azure Container Registry (ACR) and a managed Azure Container Service (AKS). They then deployed their flask app to their container registry and created a Kubernetes deployment and service to expose their flask app publically. This demonstrates how easy it can be to create an environment for getting started with Kubernetes.

We will publish a follow-up post soon so you are able to run through the Playground from the comfort of your home!

Our next meetup will soon be announced here, we look forward to seeing you there!

Interested in attending one of our DevOps Playground events? Follow up on Meetup to recieve a notification about the next event – Join us!

Billy MichaelDevOps Playground #16 – Kubernetes on Azure
read more