Solve your test data woes with GraphQL & schema introspection

Solve your test data woes with GraphQL & schema introspection

While highly technical in places, this article goes through some solutions ECS Digital has been able to provide for a client to improve testing strategies and reduce costs. Although, not for everyone, we hope that sharing our technical expertise in this area can benefit the community at large.

Tech stack: React | GraphQL | Apollo GraphQL | Javascript | Typescript | Cypress

One of our clients has been going through a change period where we are re-platforming their whole tech stack. As part of this process, we felt that now was a really great time to address an underlying issue that we have experienced with the old tech stack.

That problem is test data.

When I speak of test data, this applies to not only the unit and integration tests, but also our functional UI tests.

We had two fundamental problems we wanted to solve:

Problem one

It was the responsibility of the developer to test his component with any unstructured data they saw fit. If a developer creates a component that expects data of shape A and creates a test with data shape A, the test will pass. If, however, over time the real data that is passed to the component changes to shape B we have no idea if our component will still work until quite late in the development process, which introduces a long feedback loop.

Problem two

Our functional UI tests ran on a long living full stack. There was a known data set that we could reset to, which was all stored as json in its own repository – completely separated from the rest of stack and its tests. To update the fixture data on the full stack you would need to understand what test cases were already using the json, then manually change the json, create a PR, get it reviewed, merged and then run the mechanism to reset the data on the long living stack.

At the start of the project this fixture data was very useful. It allowed our functional UI tests to be robust and repeatable. As a result, when all our tests passed we had high confidence our site was releasable.

Unfortunately, over time and as software naturally adapts our fixture data started to become harder to update and maintain. Some parts were updated inconsistently, we had no clarity on what tests were tied to fixture data and shouldn’t be updated. Eventually our fixture data became unmaintainable or would break other tests as it was updated. 

We spent a lot of time thinking about how to solve both of these problems and after quite some time and several approaches we finally achieved something that we felt was clean, and maintainable.

Solution

Like a lot of the industry we are migrating to a GraphQL back end.

This opened an interesting opportunity as GraphQL uses types and fields to develop a `query language` for your API. You are only ever able to query for fields that exist on their corresponding types, otherwise GraphQL will complain.

GraphQL also supports something called schema introspection, which provides a mechanism to pull down a schema for any enabled GraphQL server. This can be useful to see what queries your API will support.

https://graphql.org/learn/introspection/

Another tool called GraphQL code generator can take a GraphQL schema as an input and will output all the type definitions of your GraphQL schema as a typescript file, along with any type descriptions present on your schema. (shown below)

https://github.com/dotansimha/graphql-code-generator

Problem One Solved

Now that we had the capability for translating our production GraphQL server types into typescript definitions, we were satisfied that we could start to build a fixture generator package matching our production GraphQL server. A key part of building this package was to provide a consistent API for all clients of the fixture generator package. We also ensured that whenever the logic for building fixture data started to become complicated unit and integration tests were baked in.

Once the generator package was in place, the workflow used was anytime a client of the fixture generator package is run, the schema introspection and generating our type files comes as a precursor. The whole process takes around a second and once completed the fixture generator typescript package will build. If the schema has changed, and the fixtures no longer adhere to the types, the build will fail and you are alerted straight away.

This provides huge benefit to our tests as it now means that our tests ask for the data they require. The complexity around managing test data is no longer the responsibility of the tests. We also know that the data will be correct as per the production schema even over time. Finally, if the types do change, we only need to fix it in one place for all our tests to be updated.

You can see an example of how you would use the fixture generator below

Problem Two solved

The fixture generator brought us closer to solving problem two, but we still had no way to run our functional UI tests and somehow pass our fixture generator data to our front end. The front end was still querying the long living GraphQL environment.

Another tool Apollo GraphQL provides some powerful tools around stubbing whereby you can pass your GraphQL Schema, as well as overrides for type definitions in your resolver map. Once you have defined what data you want to return when you query a type you can start a local GraphQL server.

https://www.apollographql.com/docs/apollo-server/features/mocking.html

Once running we could point our front end to our local stubbed GraphQL server.

The final piece was to have our tests once again define the data they required and then spin up the local GraphQL server.

We have also been using Cypress as our new functional UI test tool.

Cypress as a tool is groundbreaking and is revolutionising UI tests. It runs in the same run loop as your application in the browser and provides new features for UI testing such as playback mode. I’d really recommend looking if you haven’t already

In our tests we run a Cypress task to start up our short living mock GraphQL server and provide the fixture data that we want GraphQL to run with straight from the test.

Once again this means that our tests explicitly ask for data. Previously if we wanted this test to work with 4 related articles, we would have had to edit a separate repository. Try to understand if the data we want to edit is already being used by other tests. Create a pull request, get it approved, merged, then run the reset data mechanism.

Now it’s as simple as updating the variable inside the test and it is clear what data the test needs to run and practically removes the previous feedback loop.

If you want to understand at a deeper level how this works, it’s all open source.

Take a look at the repositories below:

https://github.com/newsuk/times-components/tree/master/packages/fixture-generator

https://github.com/newsuk/times-components/tree/master/packages/mock-tpa-server

Or alternatively, find out how ECS Digital can help improve your test strategy by contacting us.

Matt LowrySolve your test data woes with GraphQL & schema introspection
read more
DevOps Playground: Hands-on Visual Regression with AyeSpy

DevOps Playground: Hands-on Visual Regression with AyeSpy

The Speaker: Matt Lowry – https://www.linkedin.com/in/matthewlowry92/

Do you get frustrated by tools like Selenium where you are testing webpages in ways where it’s not intended?

Are you struggling to reduce the manual overhead of asserting that your site looks visually correct and checking that it has not regressed after implementing new changes? Visual regression testing is one of the lesser known tools in the SDET toolbox, but when implemented properly can be incredibly powerful.

AyeSpy is a new tool that we helped one of our clients News UK create to address some of the issues that existing open source visual regression tools provide. In this video, we will learn what AyeSpy is about and as usual, as this is a hands-on session, we’ll show you how to use AyeSpy to visually test your application on different viewports.

Thank you to everyone who attended. If you want to learn more about the tool, check out my recent blog on how AyeSpy is already delivering an incredible amount of business value to our client The Times.

Interested in attending our next DevOps Playground events. Follow us on Meetup to receive a notification about the next event

Matt LowryDevOps Playground: Hands-on Visual Regression with AyeSpy
read more
Is your master branch production ready?

Is your master branch production ready?

Delivering software in a continuous delivery capacity is something that nearly every project strives for. Problem is, not many projects are able to achieve continuous delivery because they don’t have the confidence in their applications quality, their build pipelines, their branching strategy or worst case, all of them.

A good indicator as to whether you fall into one of the above is to ask yourself: `can I confidently release master branch right now`.

If your answer is no, then how do we start to break down and resolve these problems.

Building confidence in quality

A recent project I have been working on fell into a few of the above categories. Nearly all their testing was done on a deployment to a long living environment, after a merge commit to master. Along with a lot of duplicated work throughout their pipeline.

The test strategy shown above was for a simple front-end application that reads data from an external API.

To start, we identified areas of our application that we knew were unloved, or treacherous to develop. Once identified, we put in place appropriate test automation. When writing test automation it is so important that your tests are robust, fast and deterministic.

We pushed as much of our UI automation down into the application. Ideally you want your application adhering to the testing pyramid principles. Testing elements that have particular classes with tools such as selenium are both time costly and of no value. There are better, more appropriate tools to do this.

Once our test scaffolding was in place, we started to feel more comfortable refactoring problem areas and reducing complexity.

We isolated our application by stubbing out external services or dependencies where necessary –  we didn’t want to be testing services outside our scope. Where possible, we recommend agreeing a contract with your external dependencies and using this to develop against.

We also recommend containerizing your app. Being able to deploy and run the same image of an application locally and on production is incredibly powerful. Long gone are the days of having long living application servers and the phrase of ‘well it works on my machine’.

Start failing fast 

Once we had confidence that when our tests all passed then the application could be deployed, we then looked to address where our tests were running.

Having tests run after a merge commit to master is too late in the process. Leaving it this long introduces a risk that someone pushes the release to production button before tests have been run.

We need to run tests earlier in the process.

In the past, to solve this problem you may have adopted complicated branching strategies dev, test, master which on paper seem reasonable, but in practice introduce horrendously slow unnecessary feedback loops and messy merges between multiple branches.

We decided to harness the power of pull request environments instead, to allow our tests to run on short living infrastructure before we merge to Master. With DevOps paradigms such as immutable infrastructure, infrastructure as code and containerisation, deploying a new environment becomes trivial.

This becomes even more powerful if you deploy your pull request environments in the same way as your production site, since you effectively test the deployment itself.

Having pull request environments spun up also caters for any testing requirements, such as exploratory testing or demos, and massively speeds up developer feedback loops.

The end result is a much higher confidence in your applications quality in master branch, which to any project is invaluable.

*******

This a two-part series, with the next article focusing on how we can start to deliver master branch to production. Watch this space.

Matt LowryIs your master branch production ready?
read more
Open source. Are you part of the community?

Open source. Are you part of the community?

Open source is a type of licensing agreement – not very exciting. The exciting bit is that it allows users to create and publish work that can be freely used, modified, integrated into larger projects or derived into new work based on the original by other users.

In an age of trade secrets and profit-driven professions, this is a unique platform that actively promotes a global exchange of innovation. It has been specifically designed to encourage contributions so that the software doesn’t stand still. The collective goal of this barrier-free community is the advancement of creative, scientific and technological tools and applications – which for many is more important than a price tag.

Who uses open source?

Although, it is most commonly used in the software industry, professionals adopt open source licenses in many industries including biotech, fashion, robotics and teaching. This article will focus solely on software applications.

What’s interesting is that more and more businesses are contributing their own source code to the community – Facebook, Airbnb, Cyprus are leading examples. According to a 2018 Tidelift Professional Open Source Survey, 92% of projects amongst European respondents contain open source libraries. Whilst on the surface this contradicts conventional commercial instinct, businesses gain a lot by giving away a little. Whilst the benefits are vast, we are going to focus on five:

  1. Competition:

Since the late 90’s and the advancement of the digital age, competition no longer resides simply between two rival companies. Businesses today also find themselves competing with open source software projects that are free, open to the public and constantly evolving.

Due to the current scale of open source contribution, even the giants in the tech industry are struggling to devote the resources or teams large enough to compete with their community counterparts.

Turning to the open source community enables businesses to outsource resource rich projects to a bottomless sea of innovative capabilities. This potentially reduces cost, pressure and speeds up the feedback loop considerably.

  1. Reputation:

In the same way the Big Bang Theory made traditional science nerds cool, the open source community can boost a business’ profile on the cool/not-cool spectrum.

Not only do businesses become more attractive to potential employees, by initiating an open source software project, or contributing to an existing one, they make their mark on an additional and power channel popular within IT circles. If done well, this has the potential to establish, maintain or improve a brand’s image, as well as attract new business.

  1. Advancement:

Helping to advance something as big as the technology industry isn’t something to turn your nose up at. In fact, businesses revel with the idea of having their name against a leading piece of software that has the potential to make history.

But history moves fast. And building software inhouse can be stifled by other business priorities, resource restrictions and other competitors beating you to the finish line.

Rather than building behind closed doors and waiting until your software it is perfect, opening your source code to the community in its earlier stages has two benefits:

  1. You can plant your flag earlier
  2. You invite an endless list of innovative capability to help advance your idea at a rate unlikely attainable behind closed doors

It also acts as an incentive for individuals to feel part of a project than extends far beyond the business they work for.

  1. Trust:

Fake news, data breaches, shady deals – all of these have encouraged people to lose trust in businesses. Including open source projects in company policy encourages business to be more transparent with its consumers. Whilst it is naive to believe a company will lay down all their cards, companies such as Facebook made 15,682 contributions in 2016. Automattic created WordPress as an open source project and currently powers 31% of the internet, and Netflix frequently open sources the tools they develop in-house.

Not only are they strengthening their brand, sharing is showing the world they have nothing to hide – which is a proven way to start winning back trust.

A great example of building this trust through transparency is the cryptocurrency space where many projects including Bitcoin allow you to browse through the project’s source. A very different approach to their corporate counterparts.

  1. Speed:

Many companies face the same problems. Sometimes companies are kind enough to share the solution. If a problem has been solved before and will provide business value in a fraction of the time and half the man power everybody wins.

Contributing to the community also gives you the capability to ask the projects contributors directly questions, ask for features or raise issues enabling you fast feedback which keeps your project moving

How does open source work? 

Contributors create a project and solve a problem. They realise that other people might benefit from this project to solve their problems. The project is shared on an open platform such as GitHub which can be downloaded and used by other users interested in the project.

If users wish to contribute, they can do this by downloading the project, creating a fork (which is an exact replica of a certain part of the pipeline) and editing the code until they are happy with the changes. Users can then request a pull request which notifies the authors that a suggested change is requested.

It is up to the author to approve the change, before deciding whether they want to include the changes. If they do, this usually becomes part of the next version, which is released at the author’s discretion.

The problem is, this could take some time. The author is under no obligation to release new versions or accept proposed changes. In fact, this is one of the limitations of the open source community. People will only give up as much information as they want to / their projects need. Authors are not there to solve specific problems, and often release software that focuses on their needs rather than trying to create something too generic.

This can be frustrating if an open source project only solves half your problem, however, the community can help bridge knowledge gaps. Users also have the option to download, build and run the project locally in the interim whilst waiting for the official new version – meaning they don’t need to wait for the software to be released with the changes they need.

How it is viable?

Whilst it doesn’t make economic sense on the surface, the community have found a way to make open source viable from a business and individual perspective. Some have capitalised on their projects, making basic versions available at no cost to the user, but adding a price tag to different versions or ‘add-ons’.

Other businesses or individuals actively contributing to the platform have benefited from angel investments, as well as new business after demonstrating successful projects.

It is also often a side project for businesses and individuals. Due to the legal freedom attributed to an open source platform, you’re able to modify the code of the product you’re using endlessly, for free, at no risk of breaching privacy policies or user agreements. This makes it the opportune ‘playground’ for those looking to get into the industry or develop new skills. According to LinkedIn:

“We believe that open sourcing projects makes our engineers better at what they do best. Engineers grow in their craft by having their work shared with the entire community.”

Risks:

With all open platforms, there is a risk of abuse. Open source communities are no different and have certainly experienced their fair share of malicious activity. However, it is the open source approach that significantly increases the reliability of the projects available to the public.

By establishing a community who believe in the future potential of the projects produced, you immediately have a security indicator in place. Many of them in fact. And with so many eyes looking at projects, malicious activity is quick to be spotted and remedied. This is because open source platforms embody an agile mentality, applied in a community wide approach. Rather than make one big change and focus on ensuring it is okay for the next six-months, contributors and authors are interested in making changes quickly, so things get fixed and evolved just as quick.

******

ECS Digital love to find value for our clients and give it back to the wider community, which is why we make tools available on open source platforms such as GitHub and NPM.

We will also be hosting a hands-on session and demonstration of AyeSpy– a visual regression testing tool – at an upcoming DevOps Playground on the 29th of November. Come along to learn more about what the AyeSpy has to offer!

Matt LowryOpen source. Are you part of the community?
read more
AyeSpy, a new way to test for visual regressions

AyeSpy, a new way to test for visual regressions

Bill Gates famously said, “I will always choose a lazy person to do a difficult job because a lazy person will find an easy way to do it.”

At The Times, there is an incredible amount of business value placed on the aesthetics of the site. There have also been past incidents where CSS bugs have caused rollbacks.

With this in mind, traditional `automated` functional testing with selenium is ineffective to find these defects – in addition to being slow and high maintenance. To add to the problem, The Times release far too often to make manual verification possible.

This is where visual regression tools shine through. Their sole purpose is to give confidence that the applications under test are visually correct.

So what is visual regression?

There are 3 main parts to understanding how visual regression works.

  1. Baseline

A set of images that define how the application should look, based on previous recordings.

  1. Latest

A set of images detailing how the application currently looks.

  1. The comparison

Once we have both the baseline and the latest, we are able to perform a comparison between how the application is supposed to look and how it looks now. If there are differences, the build will fail, and you will need to approve the changes to update the baseline images once more.

We have used a number of visual regression tools within the Times Tooling team at News UK and each proved to have limitations.

A core testing principle that we believe at ECS Digital is you need to be testing as close to production/end users as possible.

Headless browsers such as phantomJS may give you a small performance increase when executing tests, but these browsers are far from how your end users will be interacting with the application under test.

Our first visual regression tool only supported headless browsers. We had several instances where it allowed bugs through, but this only occurred on Firefox and not PhantomJS. This loophole was the reason we decided to move on.

The second tool we tried was what we believed to be the industry open source favourite. After battling with it for well over a week we could not get it running stable or under 30 minutes, which as a developer is an unacceptable feedback loop.

As you can imagine, these inefficiencies didn’t sit well with the Times Tooling team and we decided to address the problem head-on and create our own “hand-rolled” visual regression tool.

Based on our previous painful visual regression experience, we were determined to build a tool that was:

  • Super performant
  • Lightweight and,
  • Made it easy to interpret results

A proof of concept was put together before fully refining the capabilities of the tool. We then waited for priority to allow before creating ‘AyeSpy’ in one sprint.

Four months down the line and AyeSpy has been successfully implemented, gaining approval from our clients and users on GitHub. Whilst the Times Tooling Team engineered AyeSpy, The Sun and data teams within News UK have since adopted it and it’s not hard to see why – AyeSpy takes less than 90 seconds to run 44 webpage comparisons. Other benefits include:

  • Only requires a .json config file to run
  • Maintenance is low
  • Able to explicitly wait for elements before screenshot is taken
  • Can integrate the Dom before screenshot
  • Drop cookies into browser
  • Remove dynamic elements from the DOM
  • Tests are farmed out to a containerised selenium grid for distributed testing and consistent state

When deciding to use visual regression, we have found in our experience that the tool works best on reasonably static sites that do not require long user journey to be completed before the screenshot. For example, clicking through a checkout journey would introduce a high level of risk and take away value from the tool. Ideally, you want to load the page, remove all dynamic elements, and then snapshot.

Where you can find the tool?

ECS Digital love to find value for our clients and give it back to the wider community, which is why we make these tools available on open source platforms such as GitHub and NPM.

I will also be hosting a hands-on session and demonstration of AyeSpy at an upcoming DevOps Playground on the 29th of November. Come along to learn more about what the AyeSpy has to offer!

Matt LowryAyeSpy, a new way to test for visual regressions
read more