In this step-by-step guide, we will see how to set-up a Puppet Master in Amazon Web Services, and how to use it to create two other AWS instances.
We will then use Puppet to configure these two instances, one will be a MySQL Database Server and the other an Apache Web Server.
Summary
- AWS Setup
- Set up the Master
- Create the AWS Master Instance
- Install Puppet Enterprise
- Configure the Agent Nodes
- Launch the agent nodes with Puppet
- Configure Apache and MySQL using Roles and Profiles
- Create The Database and and the Webserver Roles
- Create the Apache and MySQL Profiles
- Classify our Nodes
- Manually run Puppet on each Agent Nods
- Related Articles
AWS Setup
Note that in this Guide, we use the eu-central-1 / Frankfurt zone.
If you intend to use a different zone, you will have to change the ami-id in the appropriate places in the scripts.
- Login to your AWS EC2 Console.
- Select the zone mentioned above.
- Create a AWS Access Key and a Secret Key in Security Credentials > Users > Your User Name > Create Access Key, and keep them handy so you can refer to them later.
- Go to the EC2 console.
- Create a Key Pair in Network & Security > Key Pairs > Create Key Pairs called Webdev-forest,and save the pem file to an accessible locaiton. (You will need this in order to access the Master).
Set up the Master
Create the AWS Master Instance
- In order to create the Master Instance, select EC2 Console > Instances > Launch Instance, and configure it as follows:
- Choose the Ubuntu Server 14.04 LTS (HVM), SSD Volume Type AMI.
- Choose the t2.large type.
- Use the default instance details settings.
- Use the default storage, 20 GB SSD.
- Give it a recognizable name, e.g. master_of_puppets.
- Create a security group with ports 22 for SSH only from your IP, and 3000, 8140, 443, 61613, 8142 for puppet services from anywhere.
- Review and launch.
- Use the keypair that you just created.
- Launch!
- Wait until the instance has finished initializing.
Install Puppet Enterprise
- Now using the key created before and the public hostname of your instance, which you can find in the ec2 description of your instance at the Public DNS section
chmod 400 Webdev-forest.pem
ssh -i Webdev-forest.pem ubuntu@[public hostname]
- accept the connection
- Become root
sudo su
- Edit your /etc/hosts and add the following line at the top:
vim /etc/hosts
"127.0.0.1 localhost master.puppet.vm master puppet"
- Change the hostname to “master.puppet.vm”
hostname master.puppet.vm
- Download the pe master installer
-
wget -O puppet-installer.tar.gz "https://pm.puppetlabs.com/cgi-bin/download.cgi?dist=ubuntu&rel=14.04&arch=amd64&ver=latest"
-
- Unpack the installer
tar -xf puppet-installer.tar.gz
- Install puppet master
-
./puppet-enterprise-<version>-ubuntu-14.04-amd64/puppet-enterprise-installer
- Select the [1] option to perform a guided installation
- Copy the public hostname of your ec2 instance, and go to https://<public-hostname>:3000
- There will be an error displayed by your browser, add an exception in firefox or click on advanced and then proceed in chrome. For a more elaborate guide go tohttps://docs.puppetlabs.com/pe/latest/console_accessing.html to access the console.
- Click on Let’s get started!
- Select a monolithic installation
- Type in the Puppet master FQDN: master.puppet.vm
- Type in the Puppet master DNS aliases: puppet
- Type in a Console Administrator password. Later on you will use it to login as the adminuser.
- Click on Submit and then Continue
- Now the Puppet Installer will do some checks before the installation, and will probably prompt some warnings which can be skipped.
- Click Deploy Now
- This step will take around 10 minutes, which is normal and you will then see this screen indicating that all went well:
-
- access the console at https://<public-hostname>
- The user is “admin” and the password is the one that you chose in the step before.
- You will then see the console:
The puppet master is now all set, so let’s take care of the agents.
Configure the Agent Nodes
Launch the agent nodes with Puppet
- On the master, create a new directory called create_instances in the /tmp/ directory.
mkdir ~/create_instances
- Create a new file create.pp that will create the instances
vim ~/create_instances/create.pp
- Paste the following code:
$pe_master_hostna$::fqdn # Get the master's fqdn me = $facts['ec2_metadata']['hostname'] # Get the hostname of the master $pe_master_ip = $facts['ec2_metadata']['local-ipv4'] # Get the ip of the master $pe_master_fqdn = # Set the default for the security groups Ec2_securitygroup { region => 'eu-central-1', # Replace by the region in which your puppet master is ensure => present, vpc => 'My VPC', # Replace by the name of your VPC } # Set the default for the instances Ec2_instance { region => 'eu-central-1', # Replace by the region in which your puppet master is key_name => 'Webdev-forest', # Replace by the name of your key if you chose something else ensure => 'running', image_id => 'ami-87564feb', # ubuntu-trusty-14.04-amd64-server-20160114.5 (ami-87564feb) instance_type => 't2.micro', tags => { 'OS' => 'Ubuntu Server 14.04 LTS', 'Owner' => 'Michel Lebeau' # Replace by your name }, subnet => 'My Subnet', # Replace by the name of your Subnet } # Set up the security group for the webserver ec2_securitygroup { 'web-sg': description => 'Security group for web servers', ingress => [{ # Open the port 22 to be able to SSH into, replace by your.ip/32 to secure it better protocol => 'tcp', port => 22, cidr => '0.0.0.0/0' },{ # Open the port 80 for HTTP protocol => 'tcp', port => 80, cidr => '0.0.0.0/0' }, ], } # Set up the security group for the database server ec2_securitygroup { 'db-sg': description => 'Security group for database servers', ingress => [{ # Open the port 22 to be able to SSH into, replace by your.ip/32 to secure it better protocol => 'tcp', port => 22, cidr => '0.0.0.0/0' },{ # Open the port 3306 to be able to access mysql protocol => 'tcp', port => 3306, cidr => '0.0.0.0/0' }, ], } # Set up the instances, assign the security groups and provide user data that will be executed at the end of the initialization ec2_instance { 'webserver': security_groups => ['web-sg'], user_data => template('/root/create_instances/templates/webserver.sh.erb'), } ec2_instance { 'dbserver': security_groups => ['db-sg'], user_data => template('/root/create_instances/templates/dbserver.sh.erb'), }
You can find the VPC and subnet in the VPC section of AWS, please note that Puppet expects the name of the VPc and subnet, the ID will not work.
- If you are using a different region than eu-central-1, change the region and the image_id accordingly.
- Create 2 templates
- Create a directory called “templates” inside the /tmp/create_instances directory
mkdir ~/create_instances/templates
- Create the webserver template
vim ~/create_instances/templates/webserver.sh.erb
#!/bin/bash PE_MASTER='<%= @pe_master_hostname %>' echo "<%= @pe_master_ip %> <%= @pe_master_fqdn %>" >> /etc/hosts # Download the installation script from the master and execute it curl -sk https://$PE_MASTER:8140/packages/current/install.bash | /bin/bash -s agent:certname=webserver
- Create the dbserver template
vim ~/create_instances/templates/dbserver.sh.erb
#!/bin/bash PE_MASTER='<%= @pe_master_hostname %>' echo "<%= @pe_master_ip %> <%= @pe_master_fqdn %>" >> /etc/hosts # Download the installation script from the master and execute it curl -sk https://$PE_MASTER:8140/packages/current/install.bash | /bin/bash -s agent:certname=dbserver
- Now let’s create the instances:
- Install the retries gem and the Amazon AWS Ruby SDK gem
/opt/puppetlabs/puppet/bin/gem install aws-sdk-core retries
- export your aws access key, here is a very small guide on where to find it: http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSGettingStartedGuide/AWSCredentials.html
mkdir ~/.aws/
vim ~/.aws/credentials
[default] aws_access_key_id = # Paste here your Access Key ID aws_secret_access_key = # Paste here your Secret Access Key ID region = # Specify your region, optional
- install puppet’s AWS module
puppet module install puppetlabs-aws
- finally apply the create script
puppet apply /root/create_instances/create.pp
[root@master ~]# puppet apply /root/create_instances/create.pp Notice: Compiled catalog for master.puppet.vm in environment production in 0.11 seconds Notice: /Stage[main]/Main/Ec2_instance[webserver]/ensure: changed absent to running Notice: /Stage[main]/Main/Ec2_instance[dbserver]/ensure: changed absent to running Notice: Applied catalog in 25.15 seconds
- Wait for the instances to be started and initialized. Once this process is finished, puppet will run and you will have to accept their certificates before they can communicate with the master.
- In the Puppet Enterprise Console, go to Nodes > Unsigned certificates
- Accept all so the nodes will be able to get their latest configuration from the master.
- Install the retries gem and the Amazon AWS Ruby SDK gem
- Create a directory called “templates” inside the /tmp/create_instances directory
Configure Apache and MySQL using Roles and Profiles
Now, we have two running Puppet Agent nodes communicating with our Puppet Enterprise Master. Only a few steps more and we will enjoy our new website!
Create The Database and and the Webserver Roles
The Roles will define the business logic of our applications, and will be composed by one or more profiles.
- In the master, navigate to the production environment:
cd /etc/puppetlabs/code/environments/production/
- create the modules/roles/manifests directory
mkdir -p modules/roles/manifests
- create the dbserver role
vim modules/roles/manifests/dbserver.pp
# Role for a Database Server class roles::dbserver { # Include the mysql profile include profiles::mysql }
- create the webserver role
vim modules/roles/manifests/webserver.pp
# Role for a Web Server class roles::webserver { # Include the apache profile include profiles::apache }
Create the Apache and MySQL Profiles
Now, we will create our Profiles which will define the application stack for Aache and MySQL
- create the modules/profiles/manifests directory
mkdir -p modules/profiles/manifests
- create the apache profile
vim modules/profiles/manifests/apache.pp
# Install and configure an Apache server class profiles::apache { # Install Apache and configure it class { 'apache': mpm_module => 'prefork', docroot => '/var/www', } # Install the PHP mod include apache::mod::php # Install php5-mysql for PDO mysql in PHP package { 'php5-mysql': ensure => installed, } # Get the index.php file from the master and place it in the document root file { '/var/www/index.php': ensure => file, source => 'puppet:///modules/profiles/index.php', owner => 'root', group => 'root', mode => '0755', } # Declare the exported resource @@host { 'webserver': ip => $::ipaddress, host_aliases => [$::hostname, $::fqdn] ,pin } # Collect the exported resources Host <<||>> }
- create the mysql profile
vim modules/profiles/manifests/mysql.pp
# Install and configure a MySQL server class profiles::mysql { # Install MySQL Server and configure it class {'mysql::server': root_password => 'p4ssw0rd', remove_default_accounts => true, restart => true, override_options => { mysqld => { bind_address => '0.0.0.0', 'lower_case_table_name' => 1, } } } # Copy the sql script from the puppet master to the /tmp directory file { 'mysql_populate': ensure => file, path => '/tmp/populate.sql', source => 'puppet:///modules/profiles/populate.sql', } -> # Only once the file has been copied, use it to populate a new database mysql::db { 'cats': user => 'forest', password => 'p4ssw0rd2', grant => ['SELECT', 'UPDATE', 'INSERT', 'DELETE'], host => '%', # You can replace by 'webserver' to make it more secure, # but you might have to flush your hosts in mysql for it # to be taken into account sql => '/tmp/populate.sql', } # Declare the exported resources @@host { $::hostname: ip => $::ipaddress, host_aliases => [$::fqdn, 'database'] , } # Collect the exported resources Host <<||>> }
- Create the files that will be used to pre populate the MySQL database with some sample data and the webpage that will consume that information
mkdir modules/profiles/files
vim modules/profiles/files/populate.sql
-
USE `cats`; CREATE TABLE `family` ( `id` mediumint(8) unsigned NOT NULL auto_increment, `Name` varchar(255) default NULL, `Age` mediumint default NULL, PRIMARY KEY (`id`) ) AUTO_INCREMENT=1; INSERT INTO `family` (`Name`,`Age`) VALUES ("Hasad",6),("Uma",5),("Breanna",17),("Macaulay",14),("Colton",11),("Serina",16),("Emery",13),("Christian",7),("Vladimir",16),("Wang",13); INSERT INTO `family` (`Name`,`Age`) VALUES ("Hermione",12),("Yoshio",9),("Hilel",10),("Autumn",6),("Solomon",7),("Briar",6),("Armand",9),("Alyssa",1),("Shelby",1),("Yasir",15); INSERT INTO `family` (`Name`,`Age`) VALUES ("Wallace",1),("Yoshio",5),("Pascale",6),("Dalton",17),("Trevor",9),("Joan",10),("Zephr",14),("Neville",3),("Nicole",4),("Halee",14); INSERT INTO `family` (`Name`,`Age`) VALUES ("Wayne",15),("Maile",8),("Alfonso",9),("Neve",6),("Heidi",16),("Mona",11),("Mollie",16),("Audra",16),("Karyn",12),("Acton",17); INSERT INTO `family` (`Name`,`Age`) VALUES ("Xyla",1),("Cole",6),("Blossom",9),("Sybill",4),("Lavinia",4),("Keely",14),("Gwendolyn",15),("Trevor",10),("Acton",12),("Christine",10); INSERT INTO `family` (`Name`,`Age`) VALUES ("Stone",17),("Erich",12),("Elijah",10),("Emerson",14),("Rafael",8),("Scott",17),("Olympia",13),("Nehru",14),("Casey",8),("Michael",3); INSERT INTO `family` (`Name`,`Age`) VALUES ("Montana",8),("Heidi",11),("Edward",13),("Xenos",1),("Venus",9),("Malik",5),("Madeline",2),("Sacha",8),("Whitney",13),("Eagan",8); INSERT INTO `family` (`Name`,`Age`) VALUES ("Lewis",2),("Guinevere",17),("Oliver",6),("Jana",7),("Rachel",2),("Ariel",7),("Pamela",6),("Medge",11),("Clare",10),("Meghan",8); INSERT INTO `family` (`Name`,`Age`) VALUES ("Stone",10),("Chase",4),("Vladimir",17),("Grace",11),("Damon",15),("Ferdinand",11),("Veronica",14),("Wesley",13),("Zelda",15),("Eugenia",6); INSERT INTO `family` (`Name`,`Age`) VALUES ("Carlos",9),("Cherokee",14),("Theodore",3),("Tanisha",11),("Grant",7),("Xyla",6),("Austin",11),("Madison",4),("Kasper",7),("Andrew",10);
-
vim modules/profiles/files/index.php
<?php echo "<h1>Our small cat family</h1>"; echo "<table style='border: solid 1px black;'>"; echo "<tr><th>Id</th><th>Name</th><th>Age</th></tr>"; class TableRows extends RecursiveIteratorIterator { function __construct($it) { parent::__construct($it, self::LEAVES_ONLY); } function current() { return "<td style='width:150px;border:1px solid black;'>" . parent::current(). "</td>"; } function beginChildren() { echo "<tr>"; } function endChildren() { echo "</tr>" . "\n"; } } $host = "database"; $port = "3306"; $username = "forest"; $password = "p4ssw0rd2"; $dbname = "cats"; try { $conn = new PDO("mysql:host=$host;port=$port;dbname=$dbname", $username, $password); $conn->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION); $stmt = $conn->prepare("SELECT id, Name, Age FROM family"); $stmt->execute(); // set the resulting array to associative $result = $stmt->setFetchMode(PDO::FETCH_ASSOC); foreach(new TableRows(new RecursiveArrayIterator($stmt->fetchAll())) as $k=>$v) { echo $v; } } catch(PDOException $e) { echo "Error: " . $e->getMessage(); } $conn = null; echo "</table>"; ?>
- install the apache and mysql modules
puppet module install puppetlabs-apache
puppet module install puppetlabs-mysql
Classify our Nodes
- Edit the manifest/site.pp
vim manifests/site.pp
node 'dbserver'{ include roles::dbserver } node 'webserver'{ include roles::webserver } node default { }
Manually run Puppet on each Agent Node
Puppet can be run using various methods, with the CLI, using MCollective or by using the Web Console for example. In this case we are going to use MCollective:
root@master:~# su - peadmin
peadmin@master:~$ mco puppet runonce -v -I webserver -I dbserver
* [ ============================================================> ] 2 / 2
webserver : OK
{:summary=> "Started a Puppet run using the '/opt/puppetlabs/bin/puppet agent --onetime --no-daemonize --color=false --show_diff --verbose --splay --splaylimit 120' command", :initiated_at=>1471353250}
dbserver : OK
{:summary=> "Started a Puppet run using the '/opt/puppetlabs/bin/puppet agent --onetime --no-daemonize --color=false --show_diff --verbose --splay --splaylimit 120' command", :initiated_at=>1471353250}
---- rpc stats ----
Nodes: 2 / 2
Pass / Fail: 2 / 0
Start Time: 2016-08-16 13:14:11 +0000
Discovery Time: 0.00ms
Agent Time: 142.88ms
Total Time: 142.88ms
peadmin@master:~$
To check if Puppet run successfully in the nodes and the changes that were applied to them, login to the Web Console and go to Configuration > Overview
Now paste the public address of your webserver in your favourite browser and voilà, you are done! Note that if you get a Error: SQLSTATE[HY000] [2005] Unknown MySQL server host ‘database’ (2), you should try to run puppet using mco another time, as the exported resources haven’t been collected. This happens if the ip from the dbserver is not exported before when the webserver collects its resources. Running puppet another time will collect it.
Please note that if you terminate an AWS instance and start another with the create.pp script, it will have the same certname as the one that has been terminated, however the IP will differ. In order for Puppet to run correcty in this case, on the master execute:
puppet cert clean <certname>
With <certname> being either dbserver or webserver,
- https://forge.puppetlabs.com/puppetlabs/aws
- https://forge.puppetlabs.com/puppetlabs/mysql
- https://forge.puppetlabs.com/puppetlabs/apache
- https://docs.puppetlabs.com/puppet/latest/reference/modules_fundamentals.html
Puppet Enterprise is one of the leading continuous delivery technologies, building on its heritage in infrastructure automation with the addition of Puppet Application Orchestration. Forest Technologies are proud partners of Puppet Labs and experts in delivering rapid value to our customers’ digital transformation initiatives using Puppet Enterprise.