Closing the gap between business and technology

Closing the gap between business and technology

No comments

27% of final decisions regarding IT planning, spending and management are now made by someone other than the IT department. For a successful DevOps transformation, you need to have implementation come from both the top and the bottom. Here’s how…

For a successful DevOps transformation, you need to have implementation come from both the top and the bottom.

A recent survey by IT industry association CompTIA found that 27% of final decisions are now made by someone other than the IT department (i.e., the heads of other business functions such as finance, marketing, sales and logistics).

Within a bottom-up, grassroots approach, whereby engineering alone is trying to build a better continuous development pipeline, without the support from senior stakeholders, DevOps will only stay siloed in one area. DevOps can and does scale across whole organisations, the problem is that there will be disconnect between the business and development teams. Changes to the organisation and culture are needed to close the gap.

But how should you go about ‘bridging the gap’?

It sounds cliché, but it’s all about communication. Ensuring goals of both the organisation and the teams within it all have overarching and very strategic goals they can work towards collaboratively. Making sure every product is geared towards achieving that goal. Whether it’s to increase sales or click-throughs to a specific page of a website, it must be specific, clear and concise.

Organisations must bring the business and development teams together so they can build products that help achieve strategic goals, and there are a number of effective ways that can help achieve this.

  1. Impact Mapping

Impact mapping is a strategic planning technique that prevents organisations from getting lost while building products and delivering projects. It does this by clearly communicating assumptions, helping teams align their activities with overall business objectives and make better roadmap decisions.

2. Customer Dashboards

Another way to ensure your team is guaranteeing business and IT function buy-in is through custom dashboards. These show you a representation of where you are as well as the business value of the digital transformation. The best countermeasures to inaccurate communications are the mutually reinforcing pillars of automation and measurement.

Automated systems, like custom dashboards, enable better reporting of business metrics. Rather than relying on information that’s filtered upwards to executives, you have an objective measurement system to share across the business, helping everyone get onto the same page.

Meeting the strategic goals of the organisation is imperative. Dashboards are one of the ways we ensure that we are as transparent as possible when communicating our progress, inspection and adaption from the other two core pillars of Scrum Theory and should be adopted not just at the engineering and team level, but also the program and portfolio level. Mapping things at the start does not mean that the job is done, we must update the plan to take into consideration the competitive landscape outside the organisation.

3. Organisational Culture

Finally, organisational culture is extremely important when planning a Digital Transformation project. It often comes down to how your team communicates with one another that makes the biggest difference. Ensuring that your team plans workshops with business/IT functions to get the most value from the projects, and all stakeholders are kept up to date with new developments on the project will also help.

What have we learned?

With the two examples outlined, it’s clear that if you don’t get your business involved, the product team can easily go-off on a tangent. The business will be frustrated as the product won’t be servicing a business need, and objectives will not be fulfilled.

Communicating is key, without it both parties will become disengaged.

Cultural change has to come from the top, leadership must be bought into the transformation and motivated to make it a success. Those at the coal face, the development teams rarely need convincing, they understand the benefits of a DevOps culture and in most cases will always be your path to least resistance. The key to developing applications that release true business value is bridging the gap between the two, the development teams should be seen as part of the business, rather than a service set up to support it.

ECS Digital can help you close your business’ gap on your digital transformation journey, get in contact today to find out how. We held a webinar in November which explained how to get past the DevOps Deadlock within your company – watch now.

Sarndeep NijjarClosing the gap between business and technology
read more
How to set-up a simple web development environment (web & database server) with Puppet

How to set-up a simple web development environment (web & database server) with Puppet

No comments

In this step-by-step guide, we will see how to set-up a Puppet Master in Amazon Web Services, and how to use it to create two other AWS instances.

We will then use Puppet to configure these two instances, one will be a MySQL Database Server and the other an Apache Web Server.

Summary

  1. AWS Setup
  2. Set up the Master
    1. Create the AWS Master Instance
    2. Install Puppet Enterprise
  3. Configure the Agent Nodes
    1. Launch the agent nodes with Puppet
    2. Configure Apache and MySQL using Roles and Profiles
      1. Create The Database and and the Webserver Roles
      2. Create the Apache and MySQL Profiles
    3. Classify our Nodes
    4. Manually run Puppet on each Agent Nods
  4. Related Articles

AWS Setup

Note that in this Guide, we use the eu-central-1 / Frankfurt zone.

If you intend to use a different zone, you will have to change the ami-id in the appropriate places in the scripts.

  1. Login to your AWS EC2 Console.
  2. Select the zone mentioned above.
  3. Create a AWS Access Key and a Secret Key in Security Credentials > Users > Your User Name > Create Access Key, and keep them handy so you can refer to them later.
  4. Go to the EC2 console.
  5. Create a Key Pair in Network & Security > Key Pairs > Create Key Pairs called Webdev-forest,and save the pem file to an accessible locaiton. (You will need this in order to access the Master).

Set up the Master

Create the AWS Master Instance

  1. In order to create the Master Instance, select EC2 Console > Instances > Launch Instance, and configure it as follows:
    1. Choose the Ubuntu Server 14.04 LTS (HVM), SSD Volume Type AMI.
    2. Choose the t2.large type.
    3. Use the default instance details settings.
    4. Use the default storage, 20 GB SSD.
    5. Give it a recognizable name, e.g. master_of_puppets.
    6. Create a security group with ports 22 for SSH only from your IP, and 3000, 8140, 443, 61613, 8142 for puppet services from anywhere.
    7. Review and launch.
    8. Use the keypair that you just created.
    9. Launch!
    10. Wait until the instance has finished initializing.

Install Puppet Enterprise

  1. Now using the key created before and the public hostname of your instance, which you can find in the ec2 description of your instance at the Public DNS section
    1. chmod 400 Webdev-forest.pem
    2. ssh -i Webdev-forest.pem ubuntu@[public hostname]
    3. accept the connection 
  2. Become root
    1. sudo su 
  3. Edit your /etc/hosts and add the following line at the top:
    1. vim /etc/hosts
    2. "127.0.0.1   localhost master.puppet.vm master puppet" 
  4. Change the hostname to “master.puppet.vm”
    1. hostname master.puppet.vm 

  5. Download the pe master installer
    1. wget -O puppet-installer.tar.gz "https://pm.puppetlabs.com/cgi-bin/download.cgi?dist=ubuntu&rel=14.04&arch=amd64&ver=latest"
      
      
  6. Unpack the installer
    1. tar -xf puppet-installer.tar.gz
  7. Install puppet master
    1.  ./puppet-enterprise-<version>-ubuntu-14.04-amd64/puppet-enterprise-installer
    2. Select the [1] option to perform a guided installation
    3. Copy the public hostname of your ec2 instance, and go to https://<public-hostname>:3000Puppet_1.png
    4. There will be an error displayed by your browser, add an exception in firefox or click on advanced and then proceed in chrome. For a more elaborate guide go tohttps://docs.puppetlabs.com/pe/latest/console_accessing.html to access the console.
    5. Click on Let’s get started!
    6. Select a monolithic installation
    7. Type in the Puppet master FQDN: master.puppet.vm
    8. Type in the Puppet master DNS aliases: puppet
    9. Type in a Console Administrator password. Later on you will use it to login as the adminuser.
    10. Click on Submit and then Continue
    11. Now the Puppet Installer will do some checks before the installation, and will probably prompt some warnings which can be skipped.
    12. Click Deploy Now
    13. This step will take around 10 minutes, which is normal and you will then see this screen indicating that all went well:Puppet_2.png
  8. access the console at https://<public-hostname>
    1. The user is “admin” and the password is the one that you chose in the step before.
    2. You will then see the console:Puppet_3.png

The puppet master is now all set, so let’s take care of the agents.

Configure the Agent Nodes

Launch the agent nodes with Puppet

  1. On the master, create a new directory called create_instances in the /tmp/ directory.
    1. mkdir ~/create_instances 
  2. Create a new file create.pp that will create the instances
    1. vim ~/create_instances/create.pp
    2. Paste the following code:
      $pe_master_hostna$::fqdn # Get the master's fqdn
      me = $facts['ec2_metadata']['hostname'] # Get the hostname of the master $pe_master_ip = $facts['ec2_metadata']['local-ipv4'] # Get the ip of the master $pe_master_fqdn = 
      # Set the default for the security groups
      Ec2_securitygroup {
        region => 'eu-central-1', # Replace by the region in which your puppet master is
        ensure => present,
        vpc    => 'My VPC', # Replace by the name of your VPC
      }
      
      # Set the default for the instances
      Ec2_instance {
        region        => 'eu-central-1', # Replace by the region in which your puppet master is
        key_name      => 'Webdev-forest', # Replace by the name of your key if you chose something else
        ensure        => 'running',
        image_id      => 'ami-87564feb', # ubuntu-trusty-14.04-amd64-server-20160114.5 (ami-87564feb)
        instance_type => 't2.micro',
        tags          => {
          'OS'    => 'Ubuntu Server 14.04 LTS',
          'Owner' => 'Michel Lebeau' # Replace by your name
        },
        subnet        => 'My Subnet', # Replace by the name of your Subnet
      }
      
      # Set up the security group for the webserver
      ec2_securitygroup { 'web-sg':
        description => 'Security group for web servers',
        ingress     => [{ 
          # Open the port 22 to be able to SSH into, replace by your.ip/32 to secure it better 
          protocol => 'tcp',
          port     => 22,
          cidr     => '0.0.0.0/0'
        },{ 
          # Open the port 80 for HTTP
          protocol => 'tcp',
          port     => 80,
          cidr     => '0.0.0.0/0'
          },
        ],
      }
      
      # Set up the security group for the database server
      ec2_securitygroup { 'db-sg':
        description => 'Security group for database servers',
        ingress     => [{ 
          # Open the port 22 to be able to SSH into, replace by your.ip/32 to secure it better 
          protocol => 'tcp',
          port     => 22,
          cidr     => '0.0.0.0/0'
        },{ 
          # Open the port 3306 to be able to access mysql
          protocol => 'tcp',
          port     => 3306,
          cidr     => '0.0.0.0/0'
          },
        ],
      }
       
      # Set up the instances, assign the security groups and provide user data that will be executed at the end of the initialization 
      ec2_instance { 'webserver':
        security_groups => ['web-sg'],
        user_data       => template('/root/create_instances/templates/webserver.sh.erb'),
      }
      ec2_instance { 'dbserver':
        security_groups => ['db-sg'],
        user_data       => template('/root/create_instances/templates/dbserver.sh.erb'),
      }

      You can find the VPC and subnet in the VPC section of AWS, please note that Puppet expects the name of the VPc and subnet, the ID will not work.

    3. If you are using a different region than eu-central-1, change the region and the image_id accordingly.
  3. Create 2 templates
    1. Create a directory called “templates” inside the /tmp/create_instances directory
      1. mkdir ~/create_instances/templates
    2. Create the webserver template
      1. vim ~/create_instances/templates/webserver.sh.erb
        #!/bin/bash
        PE_MASTER='<%= @pe_master_hostname %>'
        echo "<%= @pe_master_ip %> <%= @pe_master_fqdn %>" >> /etc/hosts
        # Download the installation script from the master and execute it
        curl -sk https://$PE_MASTER:8140/packages/current/install.bash | /bin/bash -s agent:certname=webserver
    3. Create the dbserver template
      1. vim ~/create_instances/templates/dbserver.sh.erb 
        #!/bin/bash
        PE_MASTER='<%= @pe_master_hostname %>'
        echo "<%= @pe_master_ip %> <%= @pe_master_fqdn %>" >> /etc/hosts
        # Download the installation script from the master and execute it
        curl -sk https://$PE_MASTER:8140/packages/current/install.bash | /bin/bash -s agent:certname=dbserver
    4. Now let’s create the instances:
      1. Install the retries gem and the Amazon AWS Ruby SDK gem
        1. /opt/puppetlabs/puppet/bin/gem install aws-sdk-core retries
      2. export your aws access key, here is a very small guide on where to find it: http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSGettingStartedGuide/AWSCredentials.html
        1. mkdir ~/.aws/
        2. vim ~/.aws/credentials
          [default]
          aws_access_key_id =          # Paste here your Access Key ID
          aws_secret_access_key =   # Paste here your Secret Access Key ID
          region =                                # Specify your region, optional
      3. install puppet’s AWS module
        1. puppet module install puppetlabs-aws
      4. finally apply the create script
        1. puppet apply /root/create_instances/create.pp
          [root@master ~]# puppet apply /root/create_instances/create.pp
          Notice: Compiled catalog for master.puppet.vm in environment production in 0.11 seconds
          Notice: /Stage[main]/Main/Ec2_instance[webserver]/ensure: changed absent to running
          Notice: /Stage[main]/Main/Ec2_instance[dbserver]/ensure: changed absent to running
          Notice: Applied catalog in 25.15 seconds
      5. Wait for the instances to be started and initialized. Once this process is finished, puppet will run and you will have to accept their certificates before they can communicate with the master.Puppet_4.png
      6. In the Puppet Enterprise Console, go to Nodes > Unsigned certificatesPuppet_5.png

         

      7. Accept all so the nodes will be able to get their latest configuration from the master.

Configure Apache and MySQL using Roles and Profiles

Now, we have two running Puppet Agent nodes communicating with our Puppet Enterprise Master. Only a few steps more and we will enjoy our new website!

Create The Database and and the Webserver Roles

The Roles will define the business logic of our applications, and will be composed by one or more profiles.

  1. In the master, navigate to the production environment:
    1. cd /etc/puppetlabs/code/environments/production/ 
  2. create the modules/roles/manifests directory
    1. mkdir -p modules/roles/manifests 
  3. create the dbserver role
    1. vim modules/roles/manifests/dbserver.pp
      # Role for a Database Server
      class roles::dbserver {
        # Include the mysql profile
        include profiles::mysql
      }
      
  4. create the webserver role
    1. vim modules/roles/manifests/webserver.pp
      # Role for a Web Server
      class roles::webserver {
        # Include the apache profile
        include profiles::apache
      }

Create the Apache and MySQL Profiles

 

Now, we will create our Profiles which will define the application stack for Aache and MySQL

  1. create the modules/profiles/manifests directory
    1. mkdir -p modules/profiles/manifests 
  2. create the apache profile
    1. vim modules/profiles/manifests/apache.pp
      # Install and configure an Apache server
      class profiles::apache {
        # Install Apache and configure it
        class { 'apache':
          mpm_module => 'prefork',
          docroot    => '/var/www',
        }
        # Install the PHP mod
        include apache::mod::php
      
        # Install php5-mysql for PDO mysql in PHP
        package { 'php5-mysql':
          ensure => installed,
        }
      
        # Get the index.php file from the master and place it in the document root
        file { '/var/www/index.php':
          ensure => file,
          source => 'puppet:///modules/profiles/index.php',
          owner  => 'root',
          group  => 'root',
          mode   => '0755',
        }
      
        # Declare the exported resource
        @@host { 'webserver':
          ip           => $::ipaddress,
          host_aliases => [$::hostname, $::fqdn] ,pin
        }
      
        # Collect the exported resources
        Host <<||>>
      }
  3. create the mysql profile
    1. vim modules/profiles/manifests/mysql.pp
      # Install and configure a MySQL server
      class profiles::mysql {
      
        # Install MySQL Server and configure it
        class {'mysql::server':
          root_password           => 'p4ssw0rd',
          remove_default_accounts => true,
          restart                 => true,
          override_options        => {
            mysqld => {
              bind_address            => '0.0.0.0',
              'lower_case_table_name' => 1,
            }
          }
        }
        # Copy the sql script from the puppet master to the /tmp directory
        file { 'mysql_populate':
          ensure => file,
          path   => '/tmp/populate.sql',
          source => 'puppet:///modules/profiles/populate.sql',
        } ->
        # Only once the file has been copied, use it to populate a new database
        mysql::db { 'cats':
          user     => 'forest',
          password => 'p4ssw0rd2',
          grant    => ['SELECT', 'UPDATE', 'INSERT', 'DELETE'],
          host     => '%', # You can replace by 'webserver' to make it more secure,
          # but you might have to flush your hosts in mysql for it
          # to be taken into account
          sql      => '/tmp/populate.sql',
        }
      
        # Declare the exported resources
        @@host { $::hostname:
          ip           => $::ipaddress,
          host_aliases => [$::fqdn, 'database'] ,
        }
      
        # Collect the exported resources
        Host <<||>>
      }
  4. Create the files that will be used to pre populate the MySQL database with some sample data and the webpage that will consume that information
    1. mkdir modules/profiles/files
    2. vim modules/profiles/files/populate.sql

      1. USE `cats`;
        
        CREATE TABLE `family` (
          `id` mediumint(8) unsigned NOT NULL auto_increment,
          `Name` varchar(255) default NULL,
          `Age` mediumint default NULL,
          PRIMARY KEY (`id`)
        ) AUTO_INCREMENT=1;
        
        INSERT INTO `family` (`Name`,`Age`) VALUES ("Hasad",6),("Uma",5),("Breanna",17),("Macaulay",14),("Colton",11),("Serina",16),("Emery",13),("Christian",7),("Vladimir",16),("Wang",13);
        INSERT INTO `family` (`Name`,`Age`) VALUES ("Hermione",12),("Yoshio",9),("Hilel",10),("Autumn",6),("Solomon",7),("Briar",6),("Armand",9),("Alyssa",1),("Shelby",1),("Yasir",15);
        INSERT INTO `family` (`Name`,`Age`) VALUES ("Wallace",1),("Yoshio",5),("Pascale",6),("Dalton",17),("Trevor",9),("Joan",10),("Zephr",14),("Neville",3),("Nicole",4),("Halee",14);
        INSERT INTO `family` (`Name`,`Age`) VALUES ("Wayne",15),("Maile",8),("Alfonso",9),("Neve",6),("Heidi",16),("Mona",11),("Mollie",16),("Audra",16),("Karyn",12),("Acton",17);
        INSERT INTO `family` (`Name`,`Age`) VALUES ("Xyla",1),("Cole",6),("Blossom",9),("Sybill",4),("Lavinia",4),("Keely",14),("Gwendolyn",15),("Trevor",10),("Acton",12),("Christine",10);
        INSERT INTO `family` (`Name`,`Age`) VALUES ("Stone",17),("Erich",12),("Elijah",10),("Emerson",14),("Rafael",8),("Scott",17),("Olympia",13),("Nehru",14),("Casey",8),("Michael",3);
        INSERT INTO `family` (`Name`,`Age`) VALUES ("Montana",8),("Heidi",11),("Edward",13),("Xenos",1),("Venus",9),("Malik",5),("Madeline",2),("Sacha",8),("Whitney",13),("Eagan",8);
        INSERT INTO `family` (`Name`,`Age`) VALUES ("Lewis",2),("Guinevere",17),("Oliver",6),("Jana",7),("Rachel",2),("Ariel",7),("Pamela",6),("Medge",11),("Clare",10),("Meghan",8);
        INSERT INTO `family` (`Name`,`Age`) VALUES ("Stone",10),("Chase",4),("Vladimir",17),("Grace",11),("Damon",15),("Ferdinand",11),("Veronica",14),("Wesley",13),("Zelda",15),("Eugenia",6);
        INSERT INTO `family` (`Name`,`Age`) VALUES ("Carlos",9),("Cherokee",14),("Theodore",3),("Tanisha",11),("Grant",7),("Xyla",6),("Austin",11),("Madison",4),("Kasper",7),("Andrew",10);
        
    3. vim modules/profiles/files/index.php
      <?php
      echo "<h1>Our small cat family</h1>";
      echo "<table style='border: solid 1px black;'>";
      echo "<tr><th>Id</th><th>Name</th><th>Age</th></tr>";
      
      class TableRows extends RecursiveIteratorIterator {
          function __construct($it) {
              parent::__construct($it, self::LEAVES_ONLY);
          }
      
          function current() {
              return "<td style='width:150px;border:1px solid black;'>" . parent::current(). "</td>";
          }
      
          function beginChildren() {
              echo "<tr>";
          }
      
          function endChildren() {
              echo "</tr>" . "\n";
          }
      }
      
      $host = "database";
      $port = "3306";
      $username = "forest";
      $password = "p4ssw0rd2";
      $dbname = "cats";
      
      try {
          $conn = new PDO("mysql:host=$host;port=$port;dbname=$dbname", $username, $password);
          $conn->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
          $stmt = $conn->prepare("SELECT id, Name, Age FROM family");
          $stmt->execute();
      
          // set the resulting array to associative
          $result = $stmt->setFetchMode(PDO::FETCH_ASSOC);
          foreach(new TableRows(new RecursiveArrayIterator($stmt->fetchAll())) as $k=>$v) {
              echo $v;
          }
      }
      catch(PDOException $e) {
          echo "Error: " . $e->getMessage();
      }
      $conn = null;
      echo "</table>";
      ?>
  5. install the apache and mysql modules
    1. puppet module install puppetlabs-apache
    2. puppet module install puppetlabs-mysql

 

Classify our Nodes

  1. Edit the manifest/site.pp
    1. vim manifests/site.pp
      node 'dbserver'{
        include roles::dbserver
      }
      
      node 'webserver'{
        include roles::webserver
      }
      
      node default {
      }

 

Manually run Puppet on each Agent Node

Puppet can be run using various methods, with the CLI, using MCollective or by using the Web Console for example. In this case we are going to use MCollective:

root@master:~# su - peadmin

peadmin@master:~$ mco puppet runonce -v -I webserver -I dbserver 

 * [ ============================================================> ] 2 / 2


webserver                               : OK
    {:summary=>      "Started a Puppet run using the '/opt/puppetlabs/bin/puppet agent --onetime --no-daemonize --color=false --show_diff --verbose --splay --splaylimit 120' command",     :initiated_at=>1471353250}

dbserver                                : OK
    {:summary=>      "Started a Puppet run using the '/opt/puppetlabs/bin/puppet agent --onetime --no-daemonize --color=false --show_diff --verbose --splay --splaylimit 120' command",     :initiated_at=>1471353250}



---- rpc stats ----
           Nodes: 2 / 2
     Pass / Fail: 2 / 0
      Start Time: 2016-08-16 13:14:11 +0000
  Discovery Time: 0.00ms
      Agent Time: 142.88ms
      Total Time: 142.88ms
peadmin@master:~$

To check if Puppet run successfully in the nodes and the changes that were applied to them, login to the Web Console and go to Configuration > Overview

 

Puppet_7.png

Now paste the public address of your webserver in your favourite browser and voilà, you are done! Note that if you get a Error: SQLSTATE[HY000] [2005] Unknown MySQL server host ‘database’ (2), you should try to run puppet using mco another time, as the exported resources haven’t been collected. This happens if the ip from the dbserver is not exported before when the webserver collects its resources. Running puppet another time will collect it.

Please note that if you terminate an AWS instance and start another with the create.pp script, it will have the same certname as the one that has been terminated, however the IP will differ. In order for Puppet to run correcty in this case, on the master execute:

puppet cert clean <certname>
With <certname> being either dbserver or webserver,

Related articles

  1. https://forge.puppetlabs.com/puppetlabs/aws
  2. https://forge.puppetlabs.com/puppetlabs/mysql
  3. https://forge.puppetlabs.com/puppetlabs/apache
  4. https://docs.puppetlabs.com/puppet/latest/reference/modules_fundamentals.html

Puppet Enterprise is one of the leading continuous delivery technologies, building on its heritage in infrastructure automation with the addition of Puppet Application Orchestration. Forest Technologies are proud partners of Puppet Labs and experts in delivering rapid value to our customers’ digital transformation initiatives using Puppet Enterprise.

Michel LebeauHow to set-up a simple web development environment (web & database server) with Puppet
read more