Why you should invest in AWS Big Data & 8 steps to becoming certified

Why you should invest in AWS Big Data & 8 steps to becoming certified

No comments

A decision that many engineers face at some point of their career is deciding what to focus their attention on next. One of the amazing advantages of working in a consultancy is being exposed to many different technologies, providing you the opportunity to explore any emerging trends you might be interested in. I’ve been lucky enough to work with a huge variety of clients ranging from industry leaders in the FTSE 100 to smaller start-ups disrupting the same technology space.

So why did I pick Big Data?

A common pattern I’ve noticed is that everyone has access to data – large amounts of raw, unstructured data. Business and technology leaders all recognise the importance of it, and the value and insight that it can deliver. Processes have been established to extract, transform and store this large amount of information, but the architecture is usually inefficient and incomplete.

Years ago these steps may have equated to the definition of an efficient data pipeline but now with emerging technologies such as Kinesis Streams, Redshift and even Server-less databases there is another way. We now have the possibility of having a real-time, cost efficient and low operational overhead solution.

Alongside this, companies set their sights on creating a data lake in the cloud. In doing so, they take advantage of a whole suite of technologies to store information in formats that they currently leverage and also in a configuration they possibly may harness in the future. These are all clear steps in the journey towards digital transformation, and with the current pace of development in AWS technologies it is the perfect time to become more acquainted with Big Data.

 

But why is the certification necessary?

The AWS Certified Big Data Speciality exam introduces and validates several key big data fundamentals. The exam itself is not just limited to AWS specific technologies but also explores the big data community. Taken straight from the exam guide we can see that the domains cover:

  1. Collection
  2. Storage
  3. Processing
  4. Analysis
  5. Visualization
  6. Data Security

These domains involve a broad range of technical roles ranging from data engineers and data scientists to individuals in SecOps. Personally, I’ve had some exposure to collection and storage of data but much less with regards to visualisation and security. You certainly have to be comfortable with wearing many different hats when tackling this exam as it tests not only your technical understanding of the solutions but also the business value created from the implementation. It’s equally important to consider the costs involved including any forecasts as the solution scales.

Having already completed several associate exams I found this certification much greater in difficulty because you are required to deep dive into Big Data concepts and the relevant technologies. One of the benefits of this certification is that the scope extends to these technologies’ application of Big Data so be prepared to dive into Machine Learning and popular frameworks like Spark & Presto.

 

Okay so how do I pass the exam?

1. A Cloud Guru’s certified big data specialty course provides an excellent introduction and overview.

2. Have some practical experience of Big data in AWS, theoretical knowledge is not enough to pass this exam…

  1. Practice architecting data pipelines, consider when Kinesis Streams vs Firehose would be appropriate.
  2. Think about how the solution would differ according to the size of the data transfer, sometimes even Snowmobile can become efficient.

3. Understand the different storage options on AWS – S3, DynamoDB, RDS, Redshift, HDFS vs EMRFS, HBase…

4. Understand the differences and use cases of popular Big Data frameworks e.g. Presto, Hive, Spark. 

5. Data Security contributes the most to your overall exam score at 20% and is involved in every single AWS service. There are always options for making the solution more secure and sometimes they’re enabled by default.

  1. Understand how to enable encryption at rest or in-transit, whether to use KMS or S3, or client side vs server side.
  2. How to grant privileged access to data e.g. IAM, Redshift Views.
  3. Authentication flows with Cognito and integrations with external identity providers.

6. Performance is a key trend

  1. Have a sound understanding of what GSI’s and LSI’s are in DynamoDB.
  2. Consider primary & sort keys, distribution styles in all of the database services
  3. Different compression types and speed of compressing/decompressing.

7.  Dive into Machine learning (ML)

  1. The Cloud Guru course mentioned above gives a good overview of the different ML models.
  2. If you have time I would recommend this machine learning course by Andrew Ng on Coursera. The technical depth is more lower level than you will need for the exam but it provides a very good introduction to a novice about the whole machine learning landscape.

8. Dive into Visualisation

  1. The A Cloud Guru course provides more than enough knowledge to tackle any questions here.
  2. Again if you have the time there’s an excellent data science course on Udemy which has a data visualisation chapter that would prove useful here.

 

Exam prep

It can’t be emphasised enough that AWS themselves provide amazing resources for learning. Definitely as preparation for the exam watch re:Invent videos and read AWS blogs & case studies.

 

Watch these videos:

  1. AWS re:Invent 2017: Big Data Architectural Patterns and Best Practices on AWS 
  2. AWS re:Invent 2017: Best Practices for Building a Data Lake in Amazon S3 and Amazon
  3. AWS re:Invent 2016: Deep Dive: Amazon EMR Best Practices & Design Patterns  
  4. AWS Summit Series 2016 | Chicago – Deep Dive + Best Practices for Real-Time Streaming Applications 

 

Read these AWS blogs:

  1. Secure Amazon EMR with Encryption 
  2. Building a Near Real-Time Discovery Platform with AWS 

 

Whitepapers

  1. Streaming Data Solutions on AWS with Amazon Kinesis
  2. Big Data Analytics Options on AWS 
  3. Lambda Architecture for Batch and Real-Time Processing on AWS with Spark Streaming and Spark SQL 

 

All of the Big Data services developer guides.

 

One last note….

This exam will expect you to consider the question from many different perspectives. You’ll need to think about not just the technical feasibility of the solution presented but also the business value that can be created. The majority of questions are scenario specific and often there is more than one valid answer, look for subtle clues to determine which solution is more ‘correct’ than the others, e.g. whether speed is a factor or if the question expects you to answer from a cost perspective.

Finally, this exam is very long (3 hours) and requires a lot of reading. I found that the time given was more than enough but remember to pace yourself otherwise you can get burned out quite easily.

Hopefully my experience and tips will have helped in preparation for the exam. Let us know if they helped you. 

Good Luck!!!

Visit our services to explore how we enable organisations to transform their internal cultures, to make it easier for teams to collaborate, and adopt practices such as Continuous Integration, Continuous Delivery, and Continuous Testing. 

ECS DigitalWhy you should invest in AWS Big Data & 8 steps to becoming certified
read more
Alexa: Building Skills for the World of Tomorrow

Alexa: Building Skills for the World of Tomorrow

No comments

We have all seen the TV Ads with someone asking Alexa (Amazons personal assistant AI) to dim the lights or start playing ‘The Grand Tour’ on Prime Video, and this technology is growing larger and faster every day.

Most commercial technologies like computers and internet started their lives in the hands of big businesses and large institutes that could afford the large initial RnD costs. In light of this, the Amazon team have taken a reverse approach and employed a small scale, iterative expansion of the product.

By providing developers access to the Alexa development kit and opening the voice service to the public, Amazon have made Alexa development a straightforward, painless and rewarding process.

Amazon incentivises its cult following of open source developers by rewarding those who create great skills that others want to use. Amazon announced:

“Publish a new skill this month and get an Alexa water bottle to help you stay hydrated during your coding sessions. If more than 75 customers use your skill in its first 30 days in the Alexa Skills Store, you can also qualify to receive an Echo Dot to help you make Alexa even smarter. The skill with the most unique users within its first 30 days after publishing in February will also earn an Echo Spot.”

Vocal Skills Revolution

We should all remember the mobile app revolution along with the tremendous increase in the number of smartphone users  experienced in global mobile app markets . A massive increase in the user base drove innovation, producing better mobile phones. An organised marketplace for app download, timely updates, advanced app development platforms became the norm. Most significantly, the development of some very useful and revolutionary apps have become part of our everyday lives. With the number of users almost doubling over the last 5 years, mobile app developers can reach more consumers than ever.

At ECS Digital, we believe Voice will experience the same type of growth as mobile applications did.

While consumers command more of their day to day life using voice-controlled technologies, from smart TVs to Alexa enabled electric cars, we can be safe in the knowledge that the voice revolution is coming and will change the way future generations interact with technology.

Alexa for Business

What is Alexa for Business?

Alexa for Business makes it easy for you to use Alexa in your organisation. Alexa for Business provides tools to manage Alexa devices, enrol users and configure skills across those devices. You can build your own context-aware voice skills using the Alexa Skills Kit (ASK) and conferencing device APIs, and you can make them available as private skills for your organisation.

What is an Alexa Skill?

Alexa is Amazon’s voice service and the brain behind tens of millions of devices like the Amazon Echo, Echo Dot, and Echo Show. It provides capabilities, or skills, that enable customers to create a more personalised experience. There are now tens of thousands of skills from companies like Starbucks, Uber, and Capital One as well as other innovative designers and developers.

Alexa Voice Service

The Alexa Voice Service (AVS) enables you to integrate Alexa directly into your products. We provide you with access to a suite of resources to quickly and easily build Alexa-enabled products, including APIs, hardware and software development tools, and documentation. With AVS, you can add a new intelligent interface to your products and offer your customers access to a growing number of Alexa features, smart home integrations, and skills.

What is the Alexa Skills Kit?

The Alexa Skills Kit (ASK) is a collection of self-service APIs, tools, documentation, and code samples that makes it fast and easy for you to add skills. ASK enables designers, developers, and brands to build engaging skills and reach customers through tens of millions of Alexa-enabled devices. With ASK, you can leverage Amazon’s knowledge and pioneering work in the field of voice design.

ECS Digital and Amazon Alexa

With Alexa for business being released in the US and coming to the rest of the world soon, we at ECS Digital have been using her to increase productivity and enable innovation within the office. We have been working on a few different initiatives coining the term OfficeOps.

Here are some of them:

Booking a meeting room

Working in a large consultancy,  it can be difficult to know if a meeting room is free. Moreover, booking said room can be a complicated and confusing process. The answer: create an internal/Dev skill to track the availability of a room, who has it and for how long. This skill also allows users to book a room on the spot, allowing our colleagues to interact with the booking process by literally asking the room for a booking slot .

Interactive Training

As a fast-moving DevOps consultancy, ECS Digital are always looking for innovative ways to improve our skills. For a long time now, we have been using Alexa to learn new skills and brush up on existing ones by using her as a pop quiz master. Colleagues located in our London Bridge office can ask Alexa to test their knowledge about a technology, helping them to maintain a high level of competency.

Summary

All evidence suggests that voice is here to stay, and will drive the next wave of technical innovation, both in business and at home, making those laborious, everyday tasks a little easier and futuristic. However, our assessment comes with a note: work still needs to be done in order make voice the standard, but we are confident that changes will be made swiftly.

Visit our services to explore how we enable organisations to transform their internal cultures, to make it easier for teams to collaborate, and adopt practices such as Continuous Integration, Continuous Delivery, and Continuous Testing. 

Morgan AtkinsAlexa: Building Skills for the World of Tomorrow
read more
The Psychology Behind Agile Practice and Better Communication

The Psychology Behind Agile Practice and Better Communication

No comments

As the number of unfilled technology based jobs increases, companies can no longer only rely on computer science graduates. Companies need to seek new sources of human capital – one being career changers. Employers of career changers, particularly in junior positions, often benefit from a number of transferable skills that might traditionally fall outside the average developer’s repertoire.

I am a career changer. The main transferable skill I bring along with me from my previous life in sales and teaching, is effective communication. This is a skill I have now honed and adapted to fit within the realms of the tech world – a world where the only common language is code. 

But when I entered the world of software, there was a notable difference to my previous jobs. Although people were willing to talk about technical concepts to a newbie developer like me, there was no willingness to ensure I could understand what had just been said, which was hardly supportive or empowering.

It’s strikingly obvious to anyone looking from the outside that there is a massive communication issue within technology today. Developers need to be able to explain technical concepts in a way that is accessible and comprehendible, not just for the sake of career changers, but also for key stakeholders and other departments. Using layman’s terms will lead to greater transparency, clearer communication and a better understanding of technical issues – both within project teams, as well as at a business level.

The most successful organisations I’ve observed and worked with tend to think a bit deeper about how they can address these issues beyond the obvious agile processes and ceremonies. Their leaders are willing to innovate and even change their own behaviour with employees. These organisations also have certain processes in place which facilitate better communication and agile practice.

Below are some of their tried and tested approaches that you might want to use or encourage at your own organisation:

        1. Questions can take your organisation a mile

Often, those in technical roles can be so absorbed in their own work that they unintentionally encourage their peers to not ask questions. There is an assumption that the rest of us in similar roles are well versed in all technology and technical concepts – an unrealistic assumption given the changing nature of the industry. Differences in skill set and experience levels make it probable that your colleagues don’t fully understand everything that you’re trying to communicate. This assumed knowledge can also become an issue, especially for new employees who are trying to prove their worth in a new workplace. 

What’s worse (and I’m sure you’ll all agree) is when people respond by ‘filling in the gaps’ or just nodding emptily through the jargon. This will always affect the end product/performance of a team, which makes it so important to encourage questions – realising that individuals learn in different ways. While some people like reading independently, others might need somebody to break it down and explain concepts to them. 

This culture of blagging is actually advocated in some companies. New employees are forced to “fake it till they make it.” But why? Sure, you should expect someone working in tech to be a quick learner and know how to research. However, instead of assuming they know everything or have the perfect understanding from their own research – why not assign someone to clarify and confirm if their current understanding is correct, whilst also explaining areas that they’re unsure of and generally just practice being helpful? The consequences of not encouraging questions is far more costly for the industry in the long run. 

TLDR: Don’t make it difficult to ask questions and don’t encourage a culture of blagging. 

        2. Leave unwanted attitudes at the door

In technology-focused roles it’s not uncommon for the staff to be highly opinionated. Having a unique perspective or understanding should be encouraged – as long as it doesn’t turn into a way for you to look down on others.

If a new developer is paired with someone who thinks they’re right all the time and doesn’t invest time bringing the newcomer up to speed, it can be detrimental. It can lead to frustration from both sides and hinder the learning process significantly. This attitude can also lead to poor productivity on a particular project – leaving both employees with low morale or blaming each other for the end result.

Ultimately, developers shouldn’t look down on each other or non-technical individuals for not knowing. Instead, they should be concerned when someone doesn’t understand a certain concept and try their best to explain. Technology affects all areas of business and developers need to practice explaining even the simplest of concepts. Although difficult and seemingly arbitrary, the knock-on effect for business growth is huge. The more others understand conceptually how things work, the smarter questions they’ll ask and the more empowered they’ll feel to tackle issues on their own – helping to foster a culture of innovation and agile practice.

TLDR: Don’t look down on people that don’t know and don’t encourage a culture of snobbery.

        3. Two heads are better than one

Employees should be encouraged to learn, teach and mentor one another. There are a number of ways to do this and facilitate communication. Many companies don’t understand the benefits of pairing and believe it can slow teams down. From experience at Makers Academy where we paired 100% of the time – I learnt much faster and the productivity levels of both parties involved increased dramatically.

I believe this is because it’s no longer a solo mission. To get to the other side successfully – communication becomes necessary. Discussing ideas and bringing your pairing partner on the journey can help cement your own understanding, as well as iron out any creases along the way. As we know, two pairs of eyes are better than one – errors are resolved faster and code is cleaner.

Another advantage to pairing is that distractions are minimised – because your very presence prompts your partner to keep working and vice versa.

Experienced developers who have been coding alone for a long time may not want to pair. The question they need to ask themselves is “why.”  Does it relate to looking down on people? Are they slowing you down? What these developers don’t realise is that they are also missing out on a great opportunity for their own learning and progress. Whether it be another technical perspective allowing you to see a blind spot or improving your soft skills. Everyone has something to learn from someone and pairing facilitates that.

TLDR: Do Mandatory pairing – helps bring people out of their comfort zones and creates basic rapport.

        4. We’re not all bots

During my first role, I noticed that developers preferred to communicate with each other over Slack. As great as having a communication tool like Slack is, it’s hardly a substantial replacement for face to face interaction. A number of studies have been conducted on the limitations of virtual text-based communication. These seem to indicate that no matter how many emoji’s or gifs used, text-based communication still cannot accurately convey the messages found in facial expressions, gestures, body language and eye contact. Text-based communication is a comfort space for many highly competent software developers but companies need to encourage other methods of communication as well.  

With reports coming out about mental health issues across the tech industry, stemmed from living isolated lives in high-pressure environments – this could be one of the most important things to consider and changes to make. A simple smile, a hello or even a ‘how was your weekend?’ would do the trick. Developers need to speak to each other in person and use the interaction muscle on a regular basis. Discussing work in person not only forces us out of isolation and from being in front of a screen all day but creates better working teams; teams comprised of actual people to connect with and not eating-sleeping-coding machines.

TLDR: Don’t communicate using only technology – talk to your colleagues face to face!

        5. Lunch ’n’ learn

Everyone loves to have a conversation over lunch. So why not take advantage of it? Lunch and learns are a great way to facilitate better communication and create a communal culture of learning. For example, internal and external individuals could speak regularly about a variety of topics. Facilitate this with a free lunch for those attending and you’re well on your way to communal culture of learning…

TLDR: Create a communal culture of learning – by providing lunchtime activities.

        6. Wellness matters

The wellbeing of your people, is a key success factor in business today. Activities such as meditation have been proven to lead to increased productivity. The de-stress effect leads to better work and can even motivate people. If the mind is in the right place, clean and organised, then code will follow. It also helps people communicate and talk to each other because they are relaxed, less anxious or stressed. Encourage this by promoting Calm or Headspace subscriptions. Another idea might be communal meditation sessions after lunch.

Wellbeing can also mean providing access to counselling, discounts at gyms and generally encouraging a healthy work/life balance through flexibility where possible.

If you’re not convinced why addressing mental health in the workplace is important, see this Deloitte article. 

TLDR: Do not neglect mental health, physical health, work/life balance and the important role they play in technology today. 

We know this kind of change is not going to happen overnight and will require some serious cultural and psychological shifts. There are gatekeepers, financial concerns and the general but ubiquitous ‘fear of change’. People can get very defensive about keeping things as they are – often to their own detriment. But even implementing one of these approaches within your place of work will help massively in truly being agile and improving communication.

Visit our services to explore how we enable organisations to transform their internal cultures, to make it easier for teams to collaborate, and adopt practices such as Continuous Integration, Continuous Delivery, and Continuous Testing. 

Asif HafeezThe Psychology Behind Agile Practice and Better Communication
read more