How to Make A LOT of Devstack with Vagrant

Sometimes you need to make A LOT of devstack instances, and on different providers. That is, some in the cloud, others locally, and then some. At least, I keep finding myself in situations where this happens, so, I made a little Vagrantfile to handle the details for me. Here is the Vagrantfile I use to make that happen. After you look the file over, I’ll explain a few key spots and show you how to use it:

Now let’s talk about the interesting spots:

  • Line 5: This defines the host name and number of devstacks you need (2 in this case). It also defines the starting IP if you are using this locally.
  • Lines 8 – 71: These are the bit that gets you ready to do the devstacks.
  • Lines 88 – 93: To make this work on the Rackspace cloud, modify these lines as needed.
  • Lines 97 – 118: These are some VMware Fusion performance hacks. Note 116 sets it to 6gb. Change that if doing this locally.

If you intend to use the Rackspace provider, you’ll need it installed, you can get that from here
That’s about it for the interesting parts. Now to use this, you can bring up all the machines at once using:

vagrant up --provider=vmware_fusion or vagrant up --provider=rackspace

To bring them up one at a time:

vagrant up devstack-01 --provider=rackspace

Once things are online, you still need to access and run stack.sh. To do that:

vagrant ssh devstack-01
su stack -
cd ~/devstack
./stack.sh

OpenStack Heat Concepts, Architecture, and Workflow

It’s not often (at least in the OpenStack world) that you get 70% done with a blog post explaining a thing and then find a resource that does it better than you could have. In this case, while it’s a bit dated, is a presentation from Mirantis on Heat, it’s Concepts, Architecture, and Workflow:

Autoscaling with Heat on Devstack

Autoscaling is one of the more interesting (and outstanding) features of OpenStack Heat. In this post, you’ll build a Devstack environment in a VM with Heat, Neutron, and Ceilometer. This will enable you to run the Autoscaling examples.

 

Getting Started

To get started, copy the following into a Vagrantfile and vagrant-up. This generally takes 3-5 minutes. Now that you have the VM powered on we need to get Devstack running. To do that, run the following commands:

vagrant ssh
sudo su stack
cd ~/devstack/
./stack.sh

Note: Now is the time to top off your coffee and batch process your emails. It’s going to be a while. On my box: stack.sh completed in 2679 seconds.

Basic WordPress

Now that Devstack is up and running, let’s dive right into the examples. First we’ll build a single WordPress install just to get a feel for things. To do that, run the following commands:

source ~/devstack/openrc
cd ~/heat-templates/hot/
ssh-keygen -t rsa -N ""
nova keypair-add --pub-key ~/.ssh/id_rsa.pub testkey
heat stack-create WordPress -f F20/WordPress_Native.yaml -P key_name=testkey

What we did, was set some environment variables for our credentials (as who wants to type those out EVERY time?). Next we moved into where we cloned the heat templates, generated an SSH key, and added it to Nova as ‘testkey’. Finally we launched our heat command.

Let’s break that command down some:

  • heat – The CLI tool we’re calling stack-create – This follows the noun-verb nomenclature of OpenStack. For heat, the ‘thing’ on which you operate is a stack. In this case we’re instructing heat to create a stack.
  • WordPress – This provides the name of the stack. These should be unique within a tenant.
  • -f F20/WordPress_Native.yaml – This specifies the template file to use. In this case, the Fedora 20 WordPress install.
  • -P key_name=testkey – -P tells the heat client that we are providing parameters that are requested within the template. In this case, key_name is a required parameter. So we tell it to use the testkey we created.

Ok, it should be done building now, let’s use another heat command to check:

heat stack-list
+--------------------------------------+------------+-----------------+----------------------+
| id                                   | stack_name | stack_status    | creation_time        |
+--------------------------------------+------------+-----------------+----------------------+
| 5fad5da6-75e9-40b2-a32f-86345b0263f3 | WordPress  | CREATE_COMPLETE | 2014-04-04T17:33:08Z |
+--------------------------------------+------------+-----------------+----------------------+

There we go! Let’s find out a bit more about it, we do that with heat stack-show:

 heat stack-show WordPress
+----------------------+----------------------------------------------------------
| Property             | Value
+----------------------+----------------------------------------------------------
| capabilities         | []
| creation_time        | 2014-04-04T17:33:08Z
| description          | Heat WordPress template to support F20, using only Heat
|                      | OpenStack-native resource types, and without the
|                      | requirement for heat-cfntools in the image. WordPress

Note, this can be enough of the name to be unique, or the UUID. In most cases, you’ll want to use the UUID. Finally, let’s clean up our environment to make room for the autoscale example:

heat stack-delete 5fad5da6-75e9-40b2-a32f-86345b0263f3
+--------------------------------------+------------+--------------------+----------------------+
| id                                   | stack_name | stack_status       | creation_time        |
+--------------------------------------+------------+--------------------+----------------------+
| 5fad5da6-75e9-40b2-a32f-86345b0263f3 | WordPress  | DELETE_IN_PROGRESS | 2014-04-04T17:33:08Z |
+--------------------------------------+------------+--------------------+----------------------+
heat stack-list
+----+------------+--------------+---------------+
| id | stack_name | stack_status | creation_time |
+----+------------+--------------+---------------+
+----+------------+--------------+---------------+

Autoscaling WordPress with Heat

Sorry for the delay in getting here, however, covering the basics will let us skip a lot of that now. The Autoscale environment is a bit more complex. That is, we have the following:

Load Balancer (LBaaS) –> Web Tier (Autoscaling Group) –> Database Tier

To build the stack, we need to find out some things about our environment. Specifically a subnet ID to place the load balancer and nodes on and an Image ID for the base of our webserver & db servers. Let’s do that:

neutron subnet-list
+--------------------------------------+----------------+-------------+--------------------------------------------+
| id                                   | name           | cidr        | allocation_pools                           |
+--------------------------------------+----------------+-------------+--------------------------------------------+
| 82596713-60ff-42ee-b766-6c501991cf55 | private-subnet | 10.0.0.0/24 | {"start": "10.0.0.2", "end": "10.0.0.254"} |
+--------------------------------------+----------------+-------------+--------------------------------------------+
$ glance image-list
+--------------------------------------+---------------------------------+-------------+------------------+-----------+--------+
| ID                                   | Name                            | Disk Format | Container Format | Size      | Status |
+--------------------------------------+---------------------------------+-------------+------------------+-----------+--------+
| 7b3bf53c-3875-4f7c-bc6d-eff11290078c | F19-i386-cfntools               | qcow2       | bare             | 469383680 | active |
| 134c4d81-5c00-45d5-b089-9fbd2d5b9afc | F19-x86_64-cfntools             | qcow2       | bare             | 528297984 | active |
| 64ade0c6-9e10-4257-9884-7406d68d49de | Fedora-x86_64-20-20131211.1-sda | qcow2       | bare             | 214106112 | active |
+--------------------------------------+---------------------------------+-------------+------------------+-----------+--------+

Now that we have those, let’s fire up the build:

heat stack-create WP-Autoscale -f autoscaling.yaml -P database_flavor=m1.small -P subnet_id=82596713-60ff-42ee-b766-6c501991cf55 -P key="testkey" -P flavor=m1.small -P image=7b3bf53c-3875-4f7c-bc6d-eff11290078c
+--------------------------------------+--------------+--------------------+----------------------+
| id                                   | stack_name   | stack_status       | creation_time        |
+--------------------------------------+--------------+--------------------+----------------------+
| f736fd6f-0348-4381-b4e5-7a4da4c3282e | WP-Autoscale | CREATE_IN_PROGRESS | 2014-04-04T17:45:25Z |
+--------------------------------------+--------------+--------------------+----------------------+

On my box, this takes A WHILE to complete. Once it’s done, you can use the same commands to view the environment, or, you can log into the dashboard and check it out:

Stack Detail - OpenStack Dashboard 2014-04-04 12-52-30 2014-04-04 12-52-33

OpenStack Heat Link-Dump

Link dump! That is, I’ve been looking into OpenStack Heat quite a bit lately. That generally means I’ve got way too many browser tabs open at the moment and need to shut some down. Thus, here comes a huge list of OpenStack links and resources.

Heat Websites

Heat related videos

#vBrownBag at OpenStack Summit, first Sponsor

I’m delighted to welcome HP to the vBrownBag sponsor list. HP are the first sponsor to signup to support the vBrownBag TechTalks at the OpenStack Summit in Atlanta.

The TechTalks are taking the place of the lightning talks that happened at previous summits. We are mainly adding recording/streaming as well as the structure. The aim is still to get as many people as possible to present the core of a topic for the OpenStack community.

We already have 25 sessions proposed, and plan on accepting every session we can fit into our schedule. If you would like to present simply fill in this form. I’m really impressed with the depth of the topics being proposed, it should be a great set of TechTalks.

We need a few more sponsors, if your organization would like to be aligned with this community effort then please drop me an email, alastair@vbrownbag.com

With only six weeks to go it’s getting exciting, looking forward to meeting some new people.

My First OpenStack Heat Template

If you have followed some of my past work, you know I have a thing for automation and orchestration. Needless, I was super excited when I discovered OpenStack Orchestration (aka Heat).

Don’t know what Heat is? The OpenStack Wiki page has a great explanation:

Heat is the main project in the OpenStack Orchestration program. It implements an orchestration engine to launch multiple composite cloud applications based on templates in the form of text files that can be treated like code. A native Heat template format is evolving, but Heat also endeavours to provide compatibility with the AWS CloudFormation template format, so that many existing CloudFormation templates can be launched on OpenStack. Heat provides both an OpenStack-native ReST API and a CloudFormation-compatible Query API.

In fact the Heat documentation is ok in most areas: http://docs.openstack.org/developer/heat/getting_started/index.html

Note: The example below uses Rackspace Cloud. In order to adjust this to work with Devstack some changes will need to be made (changing the flavors, etc).

Building a Simple OpenStack Heat Template

Heat Templates use a format conveniently called the Heat Orchestration Template or HOT. It’s important to note that at this time, HOT is still under active development. For the template, I decided to draw on my earlier work with SpiderFoot here. To get started, open your favorite text editor, and paste this YAML in. Once you have it in, I’ll describe it a bit:

 

  • 1: specifies the HOT template version. When reviewing third party templates, you can use this to see if said template is compatible with the version of Heat you have deployed.
  • 3 – 4: The description. Not required but a really good practice. the | character allows it to be split onto multiple lines
  • 6 – 39: Parameters. These lines specify various bits of input that are required when spinning a thing up from a HOT template.
  • 6: Opens the parameters clause
  • 8 – 15: Opens the “server_name” parameter. The name here is arbitrary, but should be self descriptive and consistent throughout
    • 9: Provides the variable type for ‘server_name’, in this case, a string
    • 10: A description
    • 11: default – allows us to specify a default value. In this case, we require this value to come from the CLI (get_param) (Edited per comments)
    • 12 – 15: This is where we set up a string replacement. The specifies a string in which we perform replacements. The “params” specifies what to replace with.
  • 16 – 25: Image – Specifies which image to use. A lot of this follows the same setup as lines 8-15. That said, note 22 – 25. This provides constraints on inputs. In this case defining images that can be used.
  • 26-39 – Flavor – Specifies the flavors to use. Note: These flavors are rather Rackspace Cloud centric. You can modify to fit your OpenStack deployment.
  • 41 – 65: Resources – Resources is a required section. It specifies what resources will be deployed and how they will be configured when you launch the Heat template. There are a number of resource types available, a relatively complete list can be found here: http://docs.openstack.org/developer/heat/template_guide/openstack.html
    • 42 – 65: “spiderfoot” This specifies the one resource we will be deploying as such:

Note: Don’t actually use these, especially 48 – 65. I do some terrible things like setting the root password to vagrant and running an InfoSec service that has no auth control (basically anyone, anywhere, can use the box to scan /anything/). I’ve only provided this as an example of how to assemble a Heat template

  • 66 – End of the file: These specify outputs available. These are what becomes available using the heat stack-list and heat stack-show commands.

Summary

Ok, in what turned out to be a much longer post than I’d anticipated, we built a very very simple heat template. Hope this has been helpful. In some future posts, I’ll show how to deploy Heat itself and from there, how to use this template to deploy against it.

#vBrownBags At the Atlanta Summit!

Great news, the vBrownBag crew will be at the OpenStack Summit in Atlanta this May. We will be making TechTalks and generally helping people get the most out of the summit. If you’d like to present a TechTalk you can signup using this form here.

Now that we are committed to being there we need some help to make it happen. If your organization is going to be at the Summit and would like to get a little closer to our corner of the community please get in touch with, alastair@vbrownbag.com It is the combined efforts of the community and our awesome sponsors who make vBrownBag such a great resource.

Getting Started with SpiderFoot

At some point in my past, I may or may not have done some InfoSec. As part of that role, “foot printing” a target was a tedious task. Basically, you would try to find anything and everything you could about said target, without actually engaging. When you consider the number of datasources available (Google, gmane.org, DNS, etc, etc), you can begin to see where this becomes cumbersome.

Enter SpiderFoot

The folks at SpiderFoot (http://www.spiderfoot.net/) have done A LOT of work in to automate the data gathering and assimilation process. First, take a look at this preso to more fully understand the issue and what they’ve undertaken:

Installing SpiderFoot

The install process is relatively straight forward, but has a number of little libraries you’ll need to pull down and install. The following commands can be used on an Ubuntu 12.04 install to get you up and running.

# Update the box, install some tools
sudo apt-get update
sudo apt-get install git iftop iptraf vim curl wget lighttpd python-pip python-dev swig -y
sudo apt-get install -y build-essential linux-headers-`uname -r`
# Install the needed libraries
sudo pip install netaddr
sudo pip install M2Crypto
sudo pip install cherrypy
sudo pip install mako
sudo pip install dnspython
# Clone SpiderFoot
git clone https://github.com/smicallef/spiderfoot.git
# Run it
cd ./spiderfoot
python ./sf.py 0.0.0.0:5001

Starting Your First Scan

Once installed, you should be able to point your browser to the IP of the box on port 5001 (http://<my_ip>:5001) and you’ll be presented with the following:

SpiderFoot_v2.1_20140319_135810_20140319_135841[1]

Click “New Scan” provide it a scan name, domain name, and tick or un-tick options as is relevant (some of them are more ‘intrusive’ than others, like open TCP port):

SpiderFoot_v2.1_20140319_140016_20140319_140019[1]

Finally, click Run Scan, and well, get some coffee, it’ll take a while.

Disclaimer

Something something y’all are responsible for what you do with this information, only scan things you are authorized to scan, yadda yadda…

Restart VMware Fusion Networking from the Command Line

Every now and again I run into issues using Vagrant and VMware Fusion togther. Specifically around starting VMNET-DHCP servers. The following commands have done wonders to fix that in a pinch:

sudo /Applications/VMware\ Fusion.app/Contents/Library/vmnet-cli --configure
sudo /Applications/VMware\ Fusion.app/Contents/Library/vmnet-cli --stop
sudo /Applications/VMware\ Fusion.app/Contents/Library/vmnet-cli --start