Practical OneOps

Chapter 1. Getting Started with OneOps

Since you are reading this book, you are probably keen to get up-and-running with OneOps. Deploying OneOps in an enterprise environment is a complex task. It is always prudent to start with a test installation. A test installation allows you to test all the features in isolation without affecting your current network and applications. You can also use a test installation as your developer sandbox.

A fully fledged OneOps installation consists of lot of moving parts and complex configurations. Fortunately, almost all of these are automated and for the test install they come in nice little bundles. This chapter will show you two ways to configure a test installation that can also be used as developer sandbox. We will then see how to configure our first installation of OneOps. The two ways you can install OneOps are:

Amazon AMI
Vagrant

Amazon AMI

There are multiple reasons why you may want to run OneOps on AWS rather than your own infrastructure. Running on AWS offloads all your infrastructure management needs to Amazon so you can focus on delivery. This is crucial for you if you want to keep your DevOps team small. If the rest of your infrastructure, services, and application are also on AWS, it makes strategic sense to have your management software on AWS too. Finally, AWS provides easy options for redundancy and scalability. The downside of this option is that you cannot add your own custom cloud that runs on your own premises without considerable configuration, for example, a VPN tunnel.

OneOps provides a basic AMI in the public marketplace on Amazon AWS. This AMI comes with some basic configuration already in place to get you up-and-running. Instantiating OneOps on AWS is the quickest way you can get up and running with OneOps. There are two ways you can do this. They are as follows:

Using the AWS command line tool
Using the AWS web-based console

Knowing how to spin up the OneOps AMI from the command line comes in handy as you can make it a part of your automation or DevOps delivery chain later.

Note

You can find more details on installing the AWS command line tool here:http://docs.aws.amazon.com/cli/latest/userguide/installing.html.

Once you have the command line tool installed, the first step is to generate a keypair, which you will use to connect to the new instance you will spin up.

Run the command following to generate the keypair.

$ aws ec2 create-key-pair --key-name OneOpsAuth         --query 'KeyMaterial' --output 'text' > OneOpsAuth.pem

Once the keypair is generated, create a security group to control the traffic to the instance.
```
$ aws ec2 create-security-group --group-name OneOps-sg         --description "Oneops Security Group"
        {   
          "GroupId": "sg-05e2847d"
        }
```
Note the GroupId of the group that you just created as we will need it when we open up ports. We will now open up access to ports 22 and 3000, port 22 so you can SSH to it remotely and 3000 because the OneOps frontend is a Ruby on Rails application and is accessible on port 3000.
```
$ aws ec2 authorize-security-group-ingress --group-id         sg-903004f8 --protocol tcp --port 22 --cidr 0.0.0.0/0
        $ aws ec2 authorize-security-group-ingress --group-id         sg-903004f8 --protocol tcp --port 3000 --cidr 0.0.0.0/0
```
Note that we opened up access to the whole world on port 22 and 3000 by giving access to 0.0.0.0/0. If you always login from a fixed set of IP addresses, such as a corporate network, you might want to secure your installation by giving access to that set of IP addresses only.

Finally verify that your security group was created successfully and the permissions were applied correctly.

$ aws ec2 describe-security-groups --group-id sg-05e2847d
        {
        "SecurityGroups": [
            {
                "IpPermissionsEgress": [
                    {
                        "IpProtocol": "-1", 
                        "IpRanges": [
                            {
                                "CidrIp": "0.0.0.0/0"
                            }
                        ], 
                        "UserIdGroupPairs": [], 
                        "PrefixListIds": []
                    }
                ], 
                "Description": "OneOps Security Group", 
                "IpPermissions": [
                    {
                        "PrefixListIds": [], 
                        "FromPort": 22, 
                        "IpRanges": [
                            {
                                "CidrIp": "0.0.0.0/0"
                            }
                        ], 
                        "ToPort": 22, 
                        "IpProtocol": "tcp", 
                        "UserIdGroupPairs": []
                    }, 
                    {
                        "PrefixListIds": [], 
                        "FromPort": 3000, 
                        "IpRanges": [
                            {
                                "CidrIp": "0.0.0.0/0"
                            }
                        ], 
                        "ToPort": 3000, 
                        "IpProtocol": "tcp", 
                        "UserIdGroupPairs": []
                    }
                ], 
                "GroupName": "OneOps-sg", 
                "VpcId": "vpc-45e3af21", 
                "OwnerId": "377640004228", 
                "GroupId": "sg-05e2847d"
            }
        ]
        }

Finally, we are almost ready to launch the AMI. However, for that we need the instance ID for the correct AMI. OneOps AMI names are structured as OneOps-basic-preconf-v* where * is 1.1, 1.2.

To find the latest version of image available run the command following:

$ aws ec2 describe-images --filter "Name=name,Values=OneOps*"         --query 'Images[*].{Name:Name,ID:ImageId}'
        [
            {
                "Name": "OneOps-basic-preconf-v1.3", 
                "ID": "ami-076c416d"
            }, 
        {
                "Name": "OneOps-basic-preconf-v1.2", 
                "ID": "ami-fc1b3996"
        }
        ]

Once you have the AMI ID, you are now ready to launch the OneOps instance.

Run the following command to launch the AMI instance.

$ aws ec2 run-instances --image-id ami-076c416d         --count 1 --instance-type m4.large  
        --key-name OneOpsAuth --security-group-ids sg-05e2847d
        {
        "OwnerId": "377640004228",
        "ReservationId": "r-e8a1e943",
        "Instances": [
            {
                "Hypervisor": "xen",
                "PublicDnsName": "",
                "VpcId": "vpc-45e3af21",
                "RootDeviceName": "/dev/sda1",
                "StateReason": {
                    "Message": "pending",
                    "Code": "pending"
                },
                "ProductCodes": [],
                "ClientToken": "",
                "Architecture": "x86_64",
                "SubnetId": "subnet-2311d009",
                "State": {
                    "Name": "pending",
                    "Code": 0
                },
                "PrivateIpAddress": "172.31.56.32",
                "InstanceId": "i-4af1d7c9",
                "SecurityGroups": [
                    {
                        "GroupId": "sg-05e2847d",
                        "GroupName": "OneOps-sg"
                    }
                ],
                "EbsOptimized": false,
                "VirtualizationType": "hvm",
                "KeyName": "OneOpsAuth",
                "Placement": {
                    "AvailabilityZone": "us-east-1b",
                    "Tenancy": "default",
                    "GroupName": ""
                },
                "StateTransitionReason": "",
                "ImageId": "ami-076c416d",
                "Monitoring": {
                    "State": "disabled"
                },
                "SourceDestCheck": true,
                "InstanceType": "m4.large",
                "BlockDeviceMappings": [],
                "RootDeviceType": "ebs",
                "PrivateDnsName":                     "ip-172-31-56-32.ec2.internal",
                "NetworkInterfaces": [
                    {
                        "PrivateIpAddress": "172.31.56.32",
                        "PrivateIpAddresses": [
                            {
                                "PrivateIpAddress": "172.31.56.32",
                                "Primary": true,
                                "PrivateDnsName":                                     "ip-172-31-56- 32.ec2.internal"
                            }
                         ],
                        "MacAddress": "12:a5:58:7e:f5:f1",
                        "VpcId": "vpc-45e3af21",
                        "Groups": [
                            {
                                "GroupId": "sg-05e2847d",
                                "GroupName": "OneOps-sg"
                            }
                        ],
                        "OwnerId": "377649884228",
                        "SourceDestCheck": true,
                        "Attachment": {
                            "DeleteOnTermination": true,
                            "AttachTime": "2016-03-24T19:53:13.000Z",
                            "Status": "attaching",
                            "DeviceIndex": 0,
                            "AttachmentId": "eni-attach-9a96416b"
                        },
                        "PrivateDnsName":                             "ip-172-31-56-32.ec2.internal",
                        "Status": "in-use",
                        "Description": "",
                        "SubnetId": "subnet-2311d009",
                        "NetworkInterfaceId": "eni-2aeed90f"
                    }
                ],
                "LaunchTime": "2016-03-24T19:53:13.000Z",
                "AmiLaunchIndex": 0
            }
        ],
        "Groups": []
}

image-id is the ID you obtained from the previous query for the latest image version (in this case v1.3). The key name is the new key you created and the security group is the group you created. Replace the values in the preceding command as needed. Make sure the InstanceType is m4.large as OneOps needs at least 8 GB of memory to run. The image will also attach two block devices to the instance for storage. Give the instance a few minutes to start. Lastly tag the instance with a tag called OneOps so you can easily find it. For this you will need the InstanceId. For example, in the previous output, you can see the InstanceId for the instance created above is i-4af1d7c9.

That's it! Now look up the public IP of your instance and connect to it at port 3000. You should see the OneOps login screen.

$ aws ec2 describe-instances --filter "Name=tag:Name,Values=OneOps"
{
"Reservations": [
    {
        "Groups": [],
        "OwnerId": "377649884228",
        "ReservationId": "r-e8a1e943",
        "Instances": [
            {
                 "PublicIpAddress": "54.86.6.251",
                "State": {
                    "Code": 16,
                    "Name": "running"
                },
                "Architecture": "x86_64",
                "LaunchTime": "2016-03-24T19:53:13.000Z",
                "SubnetId": "subnet-2311d009",
                "ProductCodes": [],
                "PrivateIpAddress": "172.31.56.32",
                "Monitoring": {
                    "State": "disabled"
                },
                "PrivateDnsName": "ip-172-31-56-32.ec2.internal",
                "NetworkInterfaces": [
                    {
                        "Groups": [
                            {
                                "GroupId": "sg-05e2847d",
                                "GroupName": "OneOps-sg"
                            }
                        ],
                        "OwnerId": "377649884228",
                        "NetworkInterfaceId": "eni-2aeed90f",
                        "SourceDestCheck": true,
                        "SubnetId": "subnet-2311d009",
                        "PrivateIpAddress": "172.31.56.32",
                        "Status": "in-use",
                        "MacAddress": "12:a5:58:7e:f5:f1",
                            "VpcId": "vpc-45e3af21",
                        "Description": "",
                        "PrivateDnsName":                             "ip-172-31-56-32.ec2.internal",
                        "PrivateIpAddresses": [
                            {
                                "Association": {
                                    "PublicIp": "54.86.6.251",
                                    "IpOwnerId": "amazon",
                                    "PublicDnsName":                                         "ec2-54-86-6-251.compute-                                        1.amazonaws.com"
                                },
                                "PrivateDnsName":                                     "ip-172-31-56-32.ec2.internal",
                                "PrivateIpAddress": "172.31.56.32",
                                "Primary": true
                            }
                        ],
                        "Attachment": {
                            "DeviceIndex": 0,
                            "AttachTime": "2016-03-24T19:53:13.000Z",
                            "AttachmentId": "eni-attach-9a96416b",
                            "DeleteOnTermination": true,
                                "Status": "attached"
                        },
                        "Association": {
                            "PublicIp": "54.86.6.251",
                            "IpOwnerId": "amazon",
                            "PublicDnsName":                                 "ec2-54-86-6-251.compute-                                 1.amazonaws.com"
                        }
                     }
                ],
                "ImageId": "ami-076c416d",
                "RootDeviceType": "ebs",
                "KeyName": "OneOpsAuth",
                "Placement": {
                    "AvailabilityZone": "us-east-1b",
                    "Tenancy": "default",
                    "GroupName": ""
                },
                "Hypervisor": "xen",
                "RootDeviceName": "/dev/sda1",
                "Tags": [
                    {
                        "Key": "Name",
                        "Value": "OneOps"
                    }
                ],
                "VirtualizationType": "hvm",
                "SourceDestCheck": true,
                "VpcId": "vpc-45e3af21",
                "StateTransitionReason": "",
                "ClientToken": "",
                "AmiLaunchIndex": 0,
                "InstanceType": "m4.large",
                "EbsOptimized": false,
                "InstanceId": "i-4af1d7c9",
                "BlockDeviceMappings": [
                    {
                        "DeviceName": "/dev/sda1",
                        "Ebs": {
                            "AttachTime": "2016-03-24T19:53:14.000Z",
                            "DeleteOnTermination": false,
                            "VolumeId": "vol-a0c6837e",
                            "Status": "attached"
                        }
                    },
                    {
                        "DeviceName": "/dev/sdb",
                        "Ebs": {
                            "AttachTime": "2016-03-24T19:53:14.000Z",
                            "DeleteOnTermination": false,
                            "VolumeId": "vol-23c782fd",
                            "Status": "attached"
                        }
                    }
                ],
                "SecurityGroups": [
                    {
                        "GroupId": "sg-05e2847d",
                        "GroupName": "OneOps-sg"
                    }
                ],
                "PublicDnsName":                     "ec2-54-86-6-251.compute-1.amazonaws.com"
            }
        ]
    }
]
}

Although it's nice to know how to create a OneOps instance from the command line and it does come in handy if you want to make it part of your automation, the same results can be obtained in a much quicker way through the Amazon AWS console.

On the AWS console:

Select EC2.
Then select Instances from the left navigation bar.
Click the big blue Launch Instance button on the top.
On the left navigation bar click on Community AMIs.
Search for OneOps.

You may see several results for OneOps AMIs as shown in the following screenshot:

Select the AMI with the latest revision number; in the screenshot preceding it is v1.3. Click Select. On the next screen you will be asked what kind of instance you want to launch. If you have the one-year free membership from Amazon, it is very tempting to launch the t2.micro instance, which is eligible for the free tier. However OneOps will give very poor performance on micro instances and may even fail to launch fully due to a lack of resources. To perform well OneOps needs at least 8 GB of memory. It is recommended you use at least the m4.large instance, which comes with 2 vCPUs and 8 GB of memory. Select m4.large and click on Next which will allow you to configure and review instance details.

On this screen a few options are of interest. Keep the number of instances to 1. Don't request a spot instance as they are best used for batch jobs and processes that are fault-tolerant. Select the defaults for the rest. Make sure you assign a public IP if you want your OneOps instance to be accessible from the outside world. You can also add protection against accidental termination and CloudWatch monitoring from here. Amazon charges extra for CloudWatch monitoring; however, if you will be using OneOps for a long period and ultimately carrying it over into production, you should select CloudWatch. Lastly the Tenancy of the instance will also affect its performance. For most purposes a shared instance should be fine.

Click on Next to review the storage. The AMI already comes with two volumes attached to it: a root volume of 10 GB and an additional volume of 40 GB. You can increase the size of the additional volume as most of OneOps is stored in /opt/oneops. You can also select the Delete on Termination options if you are just trying out OneOps since the storage volumes will not be terminated on instance termination and Amazon will charge you for volume storage. After making your changes, click Next to tag your instance. Here you can create tags as name/value pairs. Create a tag with the name Name and the value as OneOps to easily identify your instance. Click Next to configure the security group. Create a new group called OneOps-sg. Make sure port 22 is enabled so you can SSH to your instance. Also enable access to port 3000; OneOps is a Ruby on Rails application and binds to port 3000.

It is not really prudent to open access to all IP addresses and Amazon will warn you against that. If you know the range of IP addresses you will be logging in from, restrict access to that range.

You are now ready to review and launch your instance. You may get a couple of warnings. If you are on the free tier, you will get a warning that this instance is not eligible for the free tier since you selected m4.large. You will also get a warning that access to your instance is open to the world. Review the warnings and click Launch. Once you click Launch, you will be asked to either create a new keypair or use an existing keypair to attach to the instance. If you have an existing keypair that you have access to, select it here or else generate a new keypair and download the key. Be warned if you lose this key, you will be unable to SSH to the instance to keep this key in a safe place. Click on Launch instance. The instance will take some time to initialize. OneOps will also take some time to initialize. Once the initialization is done, you can connect to port 3000 via a browser and you should see a login screen as shown in the image above. You can also use the key you generated to login to the instance to have a look around. Make sure to use centos as the username. You can SSH to your instance directly from a compatible browser as shown in the following screenshot.

Vagrant

You can also launch a copy of OneOps using Vagrant. Vagrant is a orchestration tool from HashiCorp used to create lightweight and portable development environments. This is particularly useful if you want to create a OneOps installation on a laptop that you will carry with you. You can also create your own OneOps development environment using Vagrant.

Note

Download the latest version of VirtualBox from the Oracle VirtualBox website (https://www.virtualbox.org/wiki/Downloads) and install it. Also download and install the latest version of Vagrant from the Vagrant website (https://www.vagrantup.com/downloads.html).

Also install the extension packages for VirtualBox. These install instructions will work on the Mac or Linux. Currently the Vagrant installation does not work on Windows. Go ahead and clone the OneOps setup repository from GitHub:

git clone https://github.com/oneops/setup cd setup/vagrant-centos7

Tip

Before you start the Vagrant install, you can tune your installation to allocate more resources to your OneOps install. By default, your OneOps install will have only 1 CPU and 8 GB memory. If the host machine has more resources available, it is recommended you allocate more resources to your VM too. This will increase the performance of your OneOps installation.

Open Vagrantfile and find the vb.memory line. It will be set to 8 GB. Increase it to about half the size of your host operating system. Below that, add the line vb.cpus and give it a value of about half the number of CPUs available to your host machine. Make sure you tune it this way only if nothing else is running on the host machine. Reduce the allocation if you have other things running too. You are now ready to run vagrant up. Vagrant up starts to build OneOps for you. It does so by executing the following steps.

Vagrant imports CentOS box and creates a centos virtual machine to build a OneOps installation on top of.
It runs oneops-jreqs.sh, which installs java, postgres, vim and git among other things.
It runs install-es.sh, which installs elasticsearch.
It runs install-rvm.sh, which installs the current stable version of Ruby.
It runs install-ruby.sh, which installs Ruby on Rails and other modules required to run the OneOps frontend
It runs install-logstash.sh, which installs and configures logstash.
Finally it runs oo-setup.sh, which clones the dev-tools repository and runs all the scripts in the setup-scripts directory to actually set up OneOps.

After the installation is done you will see a message like the following.

However, in a vagrant install, the port actually gets mapped on port 9090 of the host OS. So, if you want to try your new OneOps installation, you should connect to http://localhost:9090.

OneOps application key concepts

OneOps is a complex application that facilitates DevOps. As such it uses its own terminology and introduces its own concepts. Once you familiarize yourself with these concepts you are one step closer to understanding OneOps.

Organization

An organization in OneOps is a group under which various teams, clouds, assemblies, and services can be logically grouped. An organization can be your whole organization, a sub-unit of the organization, a single group inside the organization or any combination of those. You can create organizations to logically organize your projects, sub-projects, applications of all sizes.. In that respect organizations can also be various projects in your organizations. It is entirely up to you how you define an organization.

User

A user is a single user with a single login to whom you can assign permissions and responsibilities inside of OneOps. You can group various users into a team.

Team

You can group various users into teams. You can logically group several users into cohesive teams to make management of your users simple. This also simplifies permissions. You can assign permissions to the whole team instead of individual users, for example, the development team, QA team, operations team, DevOps team, and so on.

Clouds

Clouds are groups of common services that are offered under a common tag. By themselves clouds are nothing but a label and are not functional at all. They just act as an aggregator under which you can group various public (AWS compute, Azure compute, AWS DNS, and so on) or private (OpenStack) services. It is a good idea to give a descriptive name to the cloud. For example, if you are creating a cloud to group together some AWS services such as AWS EC2, Route 53 DNS, and S3 DNS, then it's a good idea to call it AmazonCloud instead of MyCloud.

Services

As mentioned above, a cloud is just a tag and is of little use until you add a service to it. You can add a public service to it, provided you have a key, such as AWS EC2, or you can add a private service to it such as Nova Compute from OpenStack (if you have your own OpenStack installation).

Assembly

An assembly, as defined by OneOps, is the definition of an application architecture and all its components including its infrastructure. OneOps provides a very visual way of defining an assembly. Assemblies are highly customizable and portable; once defined, they can be deployed to any target cloud without any more changes. Assemblies are defined by adding different platforms to them and defining relationships between them.

Platforms

Platforms as prepackaged software templates that can be installed from packs. Some examples of platforms are MySQL, HAProxy, and so on. OneOps comes with a robust pack preinstalled. You can also define your own pack or add to the existing pack by following the template.