All Hands on DevOps #10

All Hands on DevOps #10

No comments

Yesterday our 10th All Hands On DevOps Meetup, co-organised with Third Republic, was hosted at Shazam’s office.

Our first talk was done by Dawn James, Portal Architect at Kobalt Music.

Dawn went through the issues he faced previously when Kobalt had a monolithic application. With monolithic application usually comes the same set of issues, regular downtime, difficulty to deliver changes quickly, etc.

Over the course of a year, he was able to transition from on premise to AWS, to go from using zip files to Docker, from PHP endpoints to JSON APIs. They now use micro services, Terraform and Fabric.

The change was progressive, one piece of the monolith at a time, but the result is no regular downtime (and hopefully no irregular one!), better performance (triple API speed), quicker deployments, and more independent developers who can self service.

Dawn mentioned he was a big fan of HashiCorp, but the biggest blocker to the adoption of Terraform being the difficulty for developers to adopt the tool.

ECS Digital offers a range of HashiCorp training sessions, from Consul, Terraform and Vault – register for our courses here.

Our second talk was split between Ben Belchak, Head of SRE and Jesús Roncero, Site Reliability Engineer at Shazam.

Ben talked about Shazam’s journey to containers over the past three years. When Shazam started, 20 odds years ago, it was a monolithic application, with unpatched OSes. Progressively, micro-services creeped in, in an ill-conceived way, while trying to match the business requirements and the deadlines, while putting out fires. A lack of good communication across offices spread around the globe also played a role in creating silos. At that point, there was a large amount of technical debt that needed to be addressed.

Enters Ben.

He started by defining targets: A happy team, a stable infrastructure, a good relationship between SREs and software engineers, and monitoring systems that could be trusted.

He took steps to move towards these targets. He demolished the silos that each SRE in the company had created, with the support of all levels of management, CTO and CEO comprised.

He started addressing each and every alert, deleting the useless ones, and properly annotating the useful ones.

He addressed recurring issues, which snowballed and freed up more time to fix more issues.

He then worked on automating deployments, going from taking more than an hour of an SRE’s time for a single deployment to no time at all after pressing a button.

He collected extensive metrics and incident tickets over the entire stack to understand exactly what was going on across the board, going as far as doing pre-mortems to try predicting future issues.

This lead to much more breathing room and a much more stable environment, with developers happy to focus on their work.

Later on, Ben walked us through how Shazam went from baremetal to running Kubernetes on Google Cloud.

Shazam’s servers were provisioned for large events, like the Superbowl or the Granny’s, which meant that most of the time, a lot of hardware was sitting unused.

At the beginning of 2017, they started migrating to Google Cloud. They now have almost all their clusters on Google Cloud, with only a few services left on premise.

The adoption of Kubernetes came after, and it emerged from these wants: Self healing services, auto scaling based on metrics, self sufficient developers, rolling deployments and rollbacks, dynamic monitoring based on SLOs, and the ability to create several environments from the same Docker image.

This has all been achieved, using Kubernetes on Google Cloud, with the help of helm (The Kubernetes Package Manager).

This gives Shazam the right amount of processing power at the right time, and the ability to deploy changes safely, quickly and reliably.


Thanks to everyone who came along. We love hearing about people’s knowledge, whilst consuming beer! We hope everyone had a great time and learned something new.


As always, we’d love to hear any ideas and suggestions you might have for our next event. 

Michel LebeauAll Hands on DevOps #10
read more
All Hands on DevOps #7 – Monitoring x3

All Hands on DevOps #7 – Monitoring x3

No comments

Metrics, monitoring and a generous helping of real-world experience set the stage for an intensely informative evening at the latest #AHODevOps.

For those that were unable to join us, and those whose memory needs refreshing, a quick summary of what we covered during the evening lies below, although we can’t possibly hope to capture the energy or passion of the nights two presenters in a blog post – to get a feel for that, you’ll need to attend a session in person!

Now on to our evening.…

Data driven Analytics for DevOps

Andi Mann, Chief Technology Advocate, Splunk@AndiMann.

Andi began the evening’s session with a thought provoking discussion on how decision making could be aided, or even automated, based on data collected from across the software cycle, with an end goal of improving velocity, quality and impact. However, the DevOps ecosystem is much more complicated than the sum of its parts, especially if you consider business impact.

Elaborating on this, Andi introduced the idea of using data as the single dependable constant across an entire business. He argued that if decisions were rationally based on good data that everyone can access, then everyone should be able to understand the individual reasons behind business each business decision and why it was made.

This approach inevitably leads to the following questions – what data is good data, and which metrics matter? It turns out that stakeholders could share more information than initially expected, as their concerns overlap.

Pasted image at 2017_03_21 11_41 AM.png

Finally, the focus turned to discussing three examples of how to put this process into practice:

  • Making data available to internal stakeholders to analyse velocity
  • Automatically analysing code quality to provide blameless feedback
  • Visualising business impact to give everyone a better idea of what’s working and what isn’t

Andi’s slides (including an impressive list of references) can be found here: Data-Driven Decisions for Better DevOps Outcomes

What we learned about Monitoring Production Container workloads in an Enterprise

Chris Jackson, Director of Cloud Platform Engineering, Pearson@chriswiggy.

During his session Chris shared some of his experiences in monitoring a custom, Kubernetes-based platform for digital education.

He began by describing his vision of “Development on rails” – the process of standardising and automating as much as possible, even offering self-service provisioning of databases etc. so developers can concentrate on actual development.

We then moved on to the idea of – “Monitoring driven development”- easily QOTD (after “the Spider-Man principle” with great power…). Specifically, this is the idea that you should be using APM to inform developers on what needs to change to get their apps containerised.

Throughout both topics, Chris’s passion and belief in proactive, User Experience-centric monitoring came through strongly. This can be a hard thing for enterprises with an established monitoring set-up to come around to, despite it being more important than ever since the advent of containerisation.

The key takeaway here is that monitoring and alerting should be geared towards what affects customers (internal or external!) and NOT simply on error rates or basic infrastructure metrics.

You can also find Pearson’s GitHub page for their custom Kubernetes/Jenkins Pipeline tool here: Custom Kubernetes/Jenkins Pipeline tool

As you can see, the evening covered some really insightful and engaging topics. And as such, we’d encourage you to book your place at our next Meetup before seats fill up!

Screen Shot 2017-03-20 at 15.42.04 copy.jpg

Thanks to everyone who came along. We always enjoy sharing our knowledge, pizza and beer! We hope everyone had a great time and learned something new.

As always, we’d love to hear any ideas and suggestions you might have for our next event.

Michael WarrAll Hands on DevOps #7 – Monitoring x3
read more
All Hands on DevOps #6

All Hands on DevOps #6

No comments

Last Thursday, we hosted our first All Hands on DevOps meetup of 2017 (our 6th so far!). It was great to see the successful turnouts of last year continue, with another fantastic group of attendees.

As it’s best to always start the year as you mean to go on, we aimed high (literally) with our kick-off venue – the impressive Puppet offices, located on the 13th floor of Aldgate tower.

Upon arrival at the office, we were greeted not only by our partner colleagues from Puppet, but also, by a welcome selection of drinks for the evening.

On to the real business of the meetup: We began with two engaging speakers from Elsevier, Iain Gray and James Rasell, discussing ‘Taming the chaos through Automation’, and finished with a Q&A session on the 2016 State of Devops report with Russ Parsloe and John Boero from Puppet.

For those who were unable to join us, and those who may want a refresher, below is quick summary of what was covered in the evening.


Taming the chaos though Automation:

This began with Iain Gray, Cloud hosting infrastructure and operations manager for Elsevier, giving a short intro on the challenges they, as a company, face with having 40 independent development teams, each with 2-3 AWS accounts, and each account with between 1 and 6 VPC’s.

In summary, they have 90 AWS accounts and 300 VPCS, all managed by a single person – our other speaker for the evening – James Rasell… something Iain hinted James deserved a pay rise for!

Their goal behind this way of working was to reduce the barrier of entry, have various development teams owning their infrastructure, and simply having infrastructure governed through technology. The approach, governed by a DevOps philosophy, places ops engineers within development teams and has them take responsibility for operational maturity and usage of platforms. They can then provide a consultation service to the users of their tools and give regular advice on best practice methods.They ended up with about a team of 20 people for ops.


Why Terraform?

James then picked up where Iain left off and described how they had initially gone through a proof of concept phase and rejected AWS CloudFormation for a number of reasons, but mainly as it did not meet their cloud provider agnostic requirement. They decided upon Terraform, which is a cloud provider agnostic, and also allowed them to store their code in version control and have a low barrier to entry.

Their highly modular design approach focused on creating core modules and having them tested by teams. By using semantic visioning and maturity ratings, each module was responsible for building all the necessary functions it needed to run itself. The modules would then have full README’s and change logs. Using DRY principles, (a software development principle, aimed at reducing repetition of information of all kinds) to achieve their modular approach, and using Jenkins Terraform plugin.



The evening then concluded with a Q&A session on the 2016 State of Devops, led by Russ Parsloe, Technical Solutions Engineer, and John Boero, Senior Technical Solutions Engineer from Puppet.

And while we can’t cover every question raised, one of the most interesting points was on the number of DevOps teams increasing from 19% in 2015 to 22% in 2016, something that is covered in greater detail in Puppet’s 2016 State of DevOps report.


As you can see, the evening covered some really insightful and engaging topics. And as such, we’d encourage you to book your place at our next Meetup before seats fill up!

Our next All Hands on DevOps Meetup will take place on Wednesday 15th March. Register here.


In the meantime, we’d like to offer you the opportunity to win one of a wide selection of prizes including $500 gift cards, an iPad Pro, a ticket to DevOps Enterprise Summits in London & San Fransisco, a Phillips Hue starter kit and more.

For a chance to win, and to view the full list of prizes on offer, simply complete the Puppet ‘2017 State of DevOps report’ survey before the 10th of February 2017.

Sunil TailorAll Hands on DevOps #6
read more
DevOps Playground Meetup #2: Hands on with the ELK Stack

DevOps Playground Meetup #2: Hands on with the ELK Stack

No comments

We’re back! Following on from the success of our first meetup, “Hands on with ChatOps”, we ran a meetup called “Hands on with ELK Stack”, presented by our very own Angus Williams. The course generated a lot of interest, with over 50 members still in the waitlist for the event. #devopsplayground was set up to to give the London #DevOps community an opportunity to go hands-on with some of the latest DevOps tools and provide an environment in which we can collectively share our experiences and knowledge.

To ensure the night ran smoothly, we provided everyone with @Docker compose as well as Vagrant files to get their stack up and running – as we all know, setting this up on the night would take longer than 45 minutes!

Our automated setup looked something like this:


Angus went on to explain some of his previous experiences with ELK stack and demonstrated how you would set up the log aggregation and display this in Kibana. Unfortunately, we didn’t have a large enough environment on hand to get some real data, so we made do by creating a script that spat out some random logging to allow us to display some data.

After some examples that everyone followed, we created a map showing the geolocations of where logging data was retrieved:


Dashboards were provided to all course attendees, and the enthusiasm in the room was almost pDevOps-Playground_2.jpgalpable. Nobody could wait to use this in their own environment – the creative juices (possibly helped by the beer) flowed, and everyone started discussing the different matrices and data they could gather!

Everything we ran through is on our GitHub page, and you should be able to re-run this in your own environment by clicking here.


We’d like to extend our thanks to everyone that came along and hope that you all enjoyed the experience.  And, of course, we ended in time honored meetup tradition!Dev-Ops_Playground.jpg

#devopsplayground members, please suggest topics for the next meetup.  Anyone can present and the location can be anywhere in London. Offers greatly appreciated!

Andy CuretonDevOps Playground Meetup #2: Hands on with the ELK Stack
read more